Imaging apparatus and imaging method

Abstract

The present disclosure relates to an imaging apparatus and an imaging method. A merge processor performs a (k x k) merge process, where k is an integer. The merge processor is further capable of performing a uniform merge process. In the uniform merge process, signals from at least one pixel of each color that is disposed relatively closer to a center side of a pixel group are excluded from the processing target.

Description

[0001] Cross-reference of related applications

[0002] This application claims priority to Japanese Patent Application No. 2024-208407, filed on November 29, 2024, the entire contents of which (including the description, claims, drawings and abstract) are incorporated herein by reference. Technical Field

[0003] This disclosure relates to an imaging device and an imaging method. Background Technology

[0004] For example, Simon Grosche, Andy Regenski, Jürgen Seiler, and André Kaup's "Image Super-Resolution Using T-Tetromino Pixels" (IEEE / CVF Conference on Computer Vision and Pattern Recognition Proceedings, 2023, pp. 9989-9998) and Jürgen Seiler, Markus Jonscher, Thomas Ussmueller, and André Kaup's "Increasing Imaging Resolution by Non-Regular Sampling and Joint Sparse Deconvolution and Extrapolation" As described in IEEE Transactions on Circuits and Systems on Video Technology, February 2019, Vol. 29, No. 2, pp. 308-322, merging is a well-known method for image processing. In merging, the values ​​of multiple pixels from an image sensor are summed or averaged. For example, the values ​​of pixels of the same color are summed or averaged.

[0005] When multiple pixels are collected into a single unit, the image resolution decreases. On the other hand, by summing the pixel values, brightness increases. Furthermore, by averaging, noise decreases. In other words, by treating multiple pixels as a single large pixel, the light-receiving area can be expanded and sensitivity improved. Based on these advantages, for example, the merging process can be performed when the imaging environment is dark.

[0006] Figure 23 and 24Provide an example to illustrate the process of merging related technologies. Figure 23 An example is given to illustrate the process of merging Gb pixels and B pixels. Figure 24 The following example illustrates the merging process for R pixels and Gr pixels. "Gb pixel" refers to a pixel placed under the Gb color filter. Similarly, B pixel, R pixel, and Gr pixel refer to pixels under the B filter, R filter, and Gr filter, respectively.

[0007] Figure 23 and 24 Here's an example illustrating the (2×2) merging process. The values ​​of four adjacent pixels of the same color are summed or averaged. The resulting pixel value is the value at the center of gravity (geometric center of gravity) of the four pixels on the light-receiving surface, as shown by the white-filled circle.

[0008] Figure 25 This provides a conceptual example illustrating the placement of pixels after merging. Figure 23 and 24 Compared to the pixel size in the original text, the corresponding pixel size is theoretically doubled both vertically and horizontally thanks to the (2×2) merging process. Figure 25 Display the center of gravity (geometric center) with an expanded pixel marked by a white-filled circle.

[0009] For example, in a pixel group consisting of R, Gb, Gr, and B pixels, the distances d1 and d3 between the centers of gravity within the group are shorter than the distances d2 and d4 between neighboring pixel groups. Due to this uneven placement of the centers of gravity, artifacts may appear in the image after the merging process.

[0010] This disclosure discloses an imaging device and imaging method, which, compared with related technologies, can suppress the non-uniformity of the pixel gravity center after the merging process. Summary of the Invention

[0011] According to one aspect of this disclosure, an imaging apparatus is provided, comprising a pixel array, a color filter array, and a merging processor. In the pixel array, a plurality of pixels are arranged two-dimensionally. The color filter array is disposed on the pixel array. In the color filter array, R, G, and B color filters are arranged in a Bayer arrangement. The merging processor performs a (k×k) merging process, where k is an integer. That is, the merging processor performs the merging process that treats k pixels in the vertical direction and k pixels in the horizontal direction of the pixel array as a pixel group. The merging processor is capable of performing a uniform merging process. In the uniform merging process, signals from at least one pixel of each color are excluded from the processing target, the at least one pixel being disposed relatively close to the center side of the pixel group.

[0012] Based on the structure described above, and with the aid of the uniform merging process, as will be described later... Figure 6 As shown in the figure, compared with related technologies, the distance between the gravity centers of pixels becomes more uniform after the merging process.

[0013] In the structure described above, the merging processor is able to switch between a normal merging process that uses signals from all the pixels in the pixel group and the uniform merging process.

[0014] Based on the structure described above, the normal merging process is selected when the brightness of the pixels and the reduction of noise after the merging process are important. Furthermore, the uniform merging process is selected when the reduction of artifacts is important. As described, based on the structure described above, a merging process tailored to the desired advantages of the image can be performed.

[0015] In the structure described above, the merging processor can perform a (2×2) merging process. In this case, two row selection lines are provided for one row of the pixel array. Furthermore, the connection destinations of the plurality of pixels in the pixel array can be switched every two pixels along the row direction between one row selection line and another.

[0016] Based on the structure described above, the uniform merging process can be performed on an analog circuit.

[0017] In the structure described above, four column signal lines can be provided for one column of the pixel array. In this case, in the color filter array, color filters of two colors can be arranged alternately along the column direction. Furthermore, the column signal lines can be assigned to each color of the two color filters. Additionally, for a pair of pixel groups along the column direction, the column signal line connected to one of the pixel groups can be different from the column signal line connected to the other of the pixel group.

[0018] According to the structure described above, pixel signals can be captured simultaneously and in parallel from a pair of pixel groups along the column direction.

[0019] In the structure described above, during video image capture, the merging processor can cause the plurality of pixels to be grouped to be shifted two-dimensionally between the pixel group in a predetermined frame and the pixel group in a subsequent frame.

[0020] As described above, the positions of the artifacts appearing in the predetermined frames are offset from the positions of the artifacts appearing in the subsequent frames. Using this configuration, the artifacts can be visually reduced when the video image is displayed.

[0021] In the structure described above, the imaging device may include a display and a tracking mechanism. An image can be displayed on the display via the pixel array. The eye-tracking mechanism can measure the gaze directed towards the display. The merging processor can perform the merging process for areas outside the gaze point of the gaze.

[0022] Based on the structure described above, an image process based on so-called concave rendering is performed, and the computational load of the image process can be reduced.

[0023] According to another aspect of this disclosure, an imaging method is provided. The imaging method is performed in an imaging apparatus. The imaging apparatus includes a pixel array and a color filter array. In the pixel array, a plurality of pixels are arranged two-dimensionally. The color filter array is disposed on the pixel array. In the color filter array, R, G, and B color filters are arranged in a Bayer arrangement. In the imaging apparatus, a (k×k) merging process is performed, where k is an integer. That is, a merging process is performed that treats k pixels in the vertical direction and k pixels in the horizontal direction of the pixel array as a pixel group. As a merging process, a uniform merging process is performed, wherein signals from at least one pixel of each color are excluded from the processing target, the at least one pixel being disposed relatively close to the center side of the pixel group.

[0024] In the structure described above, the merging process can switch between a normal merging process that uses signals from all the pixels in the pixel group and the uniform merging process.

[0025] In the structure described above, during the capture of video images, the plurality of pixels to be grouped can be shifted in two dimensions between the pixel group in a predetermined frame and the pixel group in a subsequent frame.

[0026] In the structure described above, the imaging device may include a display and a tracking mechanism. An image is displayed on the display via the pixel array. The eye-tracking mechanism measures the gaze directed towards the display. The merging process can be performed on the area outside the gaze point.

[0027] According to the imaging device and imaging method disclosed herein, compared with related technologies, the non-uniformity of the gravity center of the pixel after the merging process can be suppressed. Attached Figure Description

[0028] Embodiments of this disclosure will be described based on the following figures, wherein:

[0029] Figure 1 These are diagrams illustrating an imaging system according to an embodiment of the present disclosure;

[0030] Figure 2 This is a diagram illustrating a pixel arrangement based on the Bayer arrangement;

[0031] Figure 3 This is a diagram used to explain the uniform merging process (1 / 4);

[0032] Figure 4 This is a diagram used to explain the uniform merging process (2 / 4);

[0033] Figure 5 This is a diagram used to explain the uniform merging process (3 / 4);

[0034] Figure 6 This is a diagram used to explain the uniform merging process (4 / 4);

[0035] Figure 7 It is a graph comparing images obtained through a uniform merging process (horizontal line image) and images obtained through a normal merging process using related techniques;

[0036] Figure 8 It is a graph comparing images obtained through a uniform merging process (vertical line image) and images obtained through a normal merging process using related techniques;

[0037] Figure 9 This is a diagram illustrating the S / N ratio based on the brightness (light intensity) of the imaging target during the uniform merging process and other image processes.

[0038] Figure 10 These are diagrams illustrating the structure of a pixel array according to an embodiment of the present disclosure;

[0039] Figure 11 This is a diagram used to explain the uniform merging process (1 / 3) of a pixel array using an embodiment of the present disclosure;

[0040] Figure 12 This is a diagram used to explain the uniform merging process (2 / 3) of a pixel array using an embodiment of the present disclosure;

[0041] Figure 13 This is a diagram used to explain the uniform merging process (3 / 3) of a pixel array using an embodiment of the present disclosure;

[0042] Figure 14 It is a diagram used to explain the group shifting in uniform merging;

[0043] Figure 15 This is a diagram used to explain the visual effects when group shifting is performed;

[0044] Figure 16 This is a diagram illustrating the extent to which a uniform merging process is applied during image rendering based on concave points;

[0045] Figure 17 This is a diagram used to explain the normal merging of (3×3) elements;

[0046] Figure 18 This is a diagram used to explain (3×3) uniform merging;

[0047] Figure 19 It is used to explain when from Figure 18 Modify the image by uniformly merging (3×3) unused pixels;

[0048] Figure 20 This is a diagram used to explain the normal merging of (4×4) elements;

[0049] Figure 21 This is a diagram used to explain (4×4) uniform merging;

[0050] Figure 22 It is used to explain when from Figure 21 Modify the image when unused pixels are uniformly merged in a (4×4) pattern;

[0051] Figure 23 This is a diagram used to explain the merging process (1 / 3) based on related technologies;

[0052] Figure 24 This is a diagram used to explain the merging process (2 / 3) based on related technologies; and

[0053] Figure 25 This is a diagram used to explain the merging process (3 / 3) based on relevant technologies. Detailed Implementation

[0054] Imaging apparatus and imaging method according to embodiments of the present disclosure will now be described with reference to the accompanying drawings. The shapes, materials, numbers, and values ​​described below are merely illustrative for descriptive purposes. These shapes, etc., may be appropriately varied depending on the specifications of the imaging apparatus. Furthermore, in the following drawings, similar components are given the same reference numerals throughout.

[0055] 1. Structure of the imaging system

[0056] Figure 1 An imaging system according to an embodiment of the present invention is shown. The system includes an imaging unit 10, an image processor 30, a display device 40, an eye-tracking mechanism 45, and an input device 47. An imaging device according to an embodiment of the present disclosure includes an imaging unit 10 and an image processor 30.

[0057] The imaging unit 10 includes a pixel array 12, a color filter array 14, a horizontal scanning circuit 16, a CDS-ADC circuit 18, and a vertical scanning circuit 20. For example, the imaging unit 10 is a CMOS image sensor.

[0058] In pixel array 12, multiple pixels are arranged in a two-dimensional manner. For example, multiple pixels are arranged in both row and column directions, as will be described later. Figure 10 As shown in the image.

[0059] A color filter array 14 is disposed on a pixel array 12. In the color filter array 14, R filters, G filters, and B filters are arranged in a two-dimensional manner. For example, in the color filter array 14, the R filters, G filters, and B filters are arranged in a two-dimensional manner according to a Bayer arrangement.

[0060] Figure 2 Explain the arrangement of the color filter array 14 according to the Bayer arrangement. For ease of illustration, green (Gb) in the blue (B) row and green (Gr) in the red (R) row are shown using different reference numerals. Strictly speaking, Figure 2 Instead of showing the color filter array 14 itself, the pixel arrangement below the color filter array 14, which is arranged in a Bayer configuration, is shown. For example, the pixel at coordinate (0, 0) outputs a signal of Gb (pixel value).

[0061] The horizontal scanning circuit 16 is used to select the readout rows of the pixel array 12. (As described later...) Figure 10 As shown, the row selection lines RS (RS1a to RS4a, RS1b to RS4b) extend from the horizontal scan circuit 16 into the pixel array 12.

[0062] The CDS-ADC circuit 18 performs the holding of the signal (voltage value) of each pixel in the pixel array 12 and the analog-to-digital conversion (A / D conversion) of the pixels. Since the mechanisms used for holding and A / D conversion signals are known, these mechanisms will not be described herein.

[0063] The vertical scanning circuit 20 commands the CDS-ADC circuit 18 to read out which column of the pixel array 12.

[0064] The image processor 30 includes an image capture signal acquisition unit 32, a merging processor 34, and a display image generator 36. For example, the image processor 30 is formed by a computer. That is, these functional units are formed through the cooperation of the computer's CPU and memory.

[0065] The image capture signal acquisition unit 32 acquires digitally converted pixel values ​​from the CDS-ADC circuit 18. The pixel values ​​acquired by the image capture signal acquisition unit 32 are sent to the merging processor 34 and the display image generator 36. For example, as will be described below, values ​​of regions with pixel values ​​below a predetermined threshold are sent to the merging processor 34. Values ​​of regions with pixel values ​​greater than or equal to the predetermined threshold are sent to the display image generator 36. Details of the merging process will be described later.

[0066] The pixel values ​​after the merging process, as well as the pixel values ​​of each pixel in pixel array 12, are sent to display image generator 36. Display image generator 36 generates image data from these pixel values. The generated image data is sent to display device 40. The image is then displayed on display device 40.

[0067] An eye-tracking mechanism 45 is configured in the imaging system to perform concave rendering, which will be described later. For example, the eye-tracking mechanism 45 detects the gaze of a user viewing an image displayed on the display device 40.

[0068] Input device 47 is a user interface such as a touchpad, keyboard, and mouse. As described later, the user can select the uniform merging process by operating input device 47. Figures 3 to 6 ) or normal merging process ( Figures 23 to 25 In other words, the merging processor 34 can switch the merging process between a normal merging process and a uniform merging process by manipulating the input device 47.

[0069] 2. Principle of Uniform Merging

[0070] Figure 2 Let's take a portion of pixel array 12 as an example. To define pixels, numbers are assigned in the row and horizontal directions. For example, in XY coordinates, the pixel position at the top left corner is represented as (0, 0); that is, row 0, column 0. Furthermore, Figure 2 The arrangement of pixels that display the corresponding color signals of output R, G, and B.

[0071] Generally, merging refers to the image processing procedure of reducing resolution to obtain a sharper image when the imaging environment is dark. For example, in the merging process, a (k×k) merging process is performed (where k is an integer). That is, k pixels in the vertical direction and k pixels in the horizontal direction are collected into a pixel group. The merging process is executed by the merging processor 34 of the image processor 30 (see reference). Figure 1 ).

[0072] For example, in Figures 2 to 6 The example demonstrates a (2×2) merging process (k=2). For instance, in a (2×2) merging process, a Gb pixel is four pixels; that is, the pixels (0,0), (2,0), (0,2), and (2,2) are grouped into a single pixel group.

[0073] Within the collected pixel group, the pixel values ​​are averaged or summed. Averaging reduces noise (smooths out changes). Summing increases the pixel value (brightness) within a pixel.

[0074] The merging processor 34 in this embodiment can perform a uniform merging process. During the uniform merging process, such as... Figure 6 As shown in the figure, compared to the normal merging process in the prior art (see reference). Figure 25 After merging, the gravity centers of the pixels (G_Gb1 to G_Gb4, G_B1 to G_B4, G_Gr1 to G_Gr4, G_R1 to G_R4) are geometrically more evenly distributed.

[0075] Figures 3 to 6 An example illustrates a (2×2) uniform merging process. The uniform merging process is executed by merge processor 34 (see [link]). Figure 1 During the (2×2) merging process, for colors Gb, B, Gr, and R, the four pixel types are treated as processing targets. In the following text, for ease of explanation, Gb and Gr are interpreted as different colors (or different types).

[0076] refer to Figure 2 Using a (2×2) merging process, the 64 pixels are divided into four pixel groups. These pixel groups are represented by quadrilaterals with thick lines. (Reference) Figure 3 During the uniform merging process, the merging processor 34 performs the merging process in the state of excluding signals from at least one pixel of each color from the processing target, the at least one pixel being relatively close to the center side of the pixel group.

[0077] In other words, Figure 3 Pixel signals shown with diagonal shading are not used in the merging process and are discarded. In this uniform merging process, Figure 4 Show the geometric gravity centers G_Gb1 to G_Gb4 of Gb pixels and G_Gr1 to G_Gr4 of Gr pixels. Figure 5 Show the geometric gravity centers G_B1 to G_B4 of pixel B and the geometric gravity centers G_R1 to G_R4 of pixel R.

[0078] like Figure 4 and 5 As shown, during the (2×2) uniform merging process, three pixels on the outer edge are selected for each color within the pixel group. The geometric center of gravity is equivalent to the center of gravity of the triangle connecting the centers of the three outer pixels.

[0079] Figure 6 This schematically illustrates the pixel arrangement after uniform merging. For example... Figure 25 As shown, compared with the existing merging process (normal merging process), the distances between the gravity centers G_B1 to G_B4, G_R1 to G_R4, G_Gb1 to G_Gb4, and G_Gr1 to G_Gr4 are more uniform through the uniform merging process, which theoretically expands the distances.

[0080] For example, in comparison Figure 6 and Figure 25 It is then immediately apparent that, compared to the distances d1 and d3 between the centers of gravity within a pixel group after a normal merging process, the distances d11 and d13 between the centers of gravity within the pixel group will be wider after a uniform merging process. Similarly, compared to the distances d2 and d4 between the centers of gravity within pixel groups after a normal merging process, the distances d12 and d14 between the centers of gravity within pixel groups will be shorter after a uniform merging process.

[0081] By utilizing the uniform placement of the pixel-wise gravity center after the expansion as described, artifacts on the image can be suppressed. Figure 7 An example is shown showing an image of a bundle of lines that mainly extends in the horizontal direction. Figure 8 Examples are shown of images of a bundle of lines that extend primarily in the vertical direction. In these images, the drift of the line image can be illustrated as a form of artifact (image interference).

[0082] exist Figure 7 and 8 Of the two, the image on the left is the one processed through the normal merging process. The image at the center of the image is the corrected image after the image sensor performs the normal merging process. The image on the right is the one processed through the uniform merging process. Comparing these three images immediately reveals that the uniform merging process reduces artifacts in the image. In the image at the center, weights are applied to the target pixel after the normal merging process. For example, for pixel B, correction is performed during the (2×2) digital filtering process, resulting in a corrected value of h. B = (49×a + 7×b + 7×c + 1×d) / 64. Here, the top-left pixel of the (2×2) pixel group is pixel a, the top-right pixel is pixel b, the bottom-left pixel is pixel c, and the bottom-right pixel is pixel d. Similarly, for Gb pixels, a correction is performed so that the corrected value is h. Gb = (35×a + 21×b + 5×c + 3×d) / 64. For Gr pixels, perform correction so that the corrected value is h. Gr = (35×a + 5×b + 21×c + 3×d) / 64, and for pixel R, perform correction so that the corrected value is h. R = (25×a+15×b+15×c+9×d) / 64.

[0083] For the pixels targeted in the uniform merging process, the merging processor 34 (reference) Figure 1 This calculates the average or sum of the corresponding pixel values. For example, refer to... Figure 4The merging processor 34 sums the corresponding pixel values ​​of Gb pixels (0,0), Gb pixels (2,0), and Gb pixels (0,2). By summing the pixel values, the brightness value at the expanded pixel increases. Alternatively, the merging processor 34 calculates the average of the corresponding pixel values ​​of Gb pixels (0,0), Gb pixels (2,0), and Gb pixels (0,2). By calculating the average, the noise in the expanded pixel is reduced.

[0084] like Figure 3 The example illustrates that during uniform merging, the signal of at least one pixel is intentionally excluded and discarded from the processing target. Therefore, the accuracy of averaging may decrease compared to using the signals of all pixels; that is, the ability to improve the S / N ratio is limited. Figure 9 Examples illustrating the S / N ratio for each merging process. The horizontal axis represents the light intensity of the imaging environment [cd / m²]. 2 On the horizontal axis, light intensity increases to the right. The vertical axis shows the signal-to-noise ratio (S / N). On the vertical axis, the S / N ratio increases upwards.

[0085] exist Figure 9 The image above displays the characteristic curves for four different image types. In other words, Figure 9 Examples illustrate the S / N ratio of four types of images, including images with normal merging, images with uniform merging, images with digitally weighted merging, and images without a merging process (4C). In digitally weighted merging, reference... Figure 2 For Gb pixels, a weight of 2.25 is multiplied by the pixel value at coordinate (0, 0), and a weight of 0.75 is multiplied by the pixel values ​​at coordinates (0, 2) and (2, 0). Additionally, a weight of 0.25 is multiplied by the pixel value at coordinate (2, 2).

[0086] refer to Figure 9 The characteristic curves show that there is no significant difference in the S / N ratio between normal merging and uniform merging, especially in dark environments.

[0087] As described, in the uniform merging process according to this embodiment, signals from at least one pixel of each color are excluded from the processing target, said at least one pixel being positioned relatively close to the center side of the pixel group. However, for processes other than the uniform merging process, such as phase difference detection, the excluded pixels can be utilized.

[0088] 3. Circuit structure for simulating uniform merging

[0089] The uniform merging process according to this embodiment can be performed by calculating the digital value after A / D conversion. Alternatively, a portion of the uniform merging process according to this embodiment can be performed before the A / D conversion; that is, during the simulation phase.

[0090] Figure 10 Provide an example of a circuit structure for implementing simulated uniform merging. Figure 10 In, corresponding to Figure 2 The coordinates are expressed as a number, with numbers assigned to pixels. For example, the pixel Gb00 at the top left corner of the image corresponds to coordinates (0, 0); that is, the Gb pixel in row 0 and column 0. The pixel Gr77 at the bottom right corner of the image corresponds to the Gr pixel at coordinates (7, 7).

[0091] Also in this figure, the pixels in pixel array 12 are arranged based on a Bayer arrangement. That is, pixels of the two colors are arranged alternately along the row direction. In addition, pixels of the two colors are arranged alternately along the column direction. For ease of explanation, Gb and Gr are considered as different colors.

[0092] The row selection line extends from the horizontal scan circuit 16 into the pixel array 12. Figure 10 In the pixel array 12, two row selection lines are provided for each row of pixels. For example, row selection lines RS1a and RS1b are provided for the first pixel row. In this way, row selection lines RSna and RSnb are provided for the nth pixel row (where n is an integer).

[0093] The connection destinations of multiple pixels in pixel array 12 are switched every two pixels along the row direction between one row selection line RSna and another row selection line RSnb. For example, referring to the first row, pixels B10 and Gb20 are connected to row selection line RS1b, and pixels B30 and Gb40 are connected to row selection line RS1a.

[0094] For a column of pixels, the row selection line used as the connection destination is unified as either row selection line RSna or row selection line RSnb. For example, all pixels in the first column are connected to row selection line RSna (RS1a, RS2a, RS3a, RS4a, RS5a, RS6a, RS7a, RS8a).

[0095] 4. Parallel Process Structure

[0096] Figure 10 The pixel array 12 illustrated in the example can simultaneously read out signals from groups of pixels along the column direction. (See reference...) Figure 10For one column of pixel array 12, four column signal lines are provided. That is, for the m-th column of pixel array 12 (where m is an integer), four column signal lines Cma, Cmb, Cmc, and Cmd are provided. CDS-ADC circuit 18 (reference) Figure 1 The CDS and ADC elements are connected to each of these column signal lines Cma, Cmb, Cmc and Cmd.

[0097] In a Bayer arrangement, color filters of two colors are arranged alternately along the column direction. For each color of the two color filters, a column signal line is assigned. For example, for... Figure 10 In the image, pixel group A1, pixel Gb00, and pixel Gb02 are both connected to column signal line C1d. Similarly, pixel R01 and pixel R03 are both connected to column signal line C1b.

[0098] For pixel group A2, which forms a pair with pixel group A1 along the column direction, pixels Gb04 and Gb06 are both connected to column signal line C1c, and pixels R05 and R07 are both connected to column signal line C1a.

[0099] In this way, Figure 10 In the pixel array illustrated in the example, column signal lines are assigned for each color of the two color filters. Furthermore, for a pair of pixel groups along the column direction, the column signal lines connected to one pixel group are different from those connected to the other. That is, the column signal lines do not overlap for each color within a pixel group. Moreover, the column signal lines do not overlap for pixel groups arranged along the column direction.

[0100] Using this type of column signal line and connection format, as described later, the signals of pixel groups along the column direction can be read simultaneously.

[0101] 5. Simulate the uniform merging process

[0102] Figure 11 This provides an overview of the uniform merging process. (Reference) Figure 11 The dotted quadrilateral frame is similar to Figure 3 The pixel groups are divided into four. Furthermore, pixels positioned relatively close to the center of the pixel group are shown with diagonal shading. During the merging process, the signals of pixels included in the shading are excluded (discarded).

[0103] Figure 12 Provide examples to illustrate specific circuit operations. Figure 12 The operation of pixel groups A1 and A2 is illustrated below. Due to the symmetry of the circuit, the remaining pixel groups are processed in a similar manner to pixel groups A1 and A2.

[0104] During the uniform merging process, merge processor 34 (see...) Figure 1 The command sets the row selection line of the horizontal scan circuit to the "on" state. Figure 12 In the middle, the row selection line that needs to be "connected" is displayed with an underline.

[0105] In other words, for the row above (first row) and the row below (fourth row) of pixel group A1, a set of row selection lines RS1a, RS1b, RS4a, and RS4b are set to the "on" state. Furthermore, for the middle rows (second and third rows) of pixel group A1, only row selection lines RS2a and RS3a are set to the "on" state. Using this configuration, the signals of pixels Gr11, R21, B12, and Gb22 located at the center of pixel group A1 are not read.

[0106] Similarly, for pixel group A2 in the previous row (fifth row) and the next row (eighth row), a set of row selection lines RS5a, RS5b, RS8a, and RS8b are set to the "on" state, while for pixel group A2 in the middle rows (sixth and seventh rows), only row selection lines RS6a and RS7a are set to the "on" state. Using this configuration, the signals of pixels Gr15, R25, B16, and Gb26 located at the center of pixel group A2 are not read.

[0107] When a normal merge process is performed instead of a uniform merge process, during row readout, a pair of row selection lines RSna and RSnb (where n is an integer) are set to the "on" state for all rows.

[0108] During the uniform merging process, as described above, the column signal lines C1a, C1b, C1c, and C1d, which serve as the connection destinations for pixel groups A1 and A2, do not overlap. Therefore, the readout of pixel groups A1 and A2 is performed simultaneously.

[0109] Looking at the first column, the signals of pixels Gb00 and Gb02 are simultaneously sent to column signal line C1d. Since the pixel signal is a voltage signal, and pixels Gb00 and Gb02 are arranged in parallel with each other in the circuit, the average voltage of pixels Gb00 and Gb02 is output from column signal line C1d. In other words, the pixel value averaging process is performed on the analog circuit. Similarly, when the signals of pixels of the same color are simultaneously sent to a column signal line, the average voltage of these pixels is output from the column signal line.

[0110] Looking at the second and third columns, the signal from one pixel is sent to each column of signal lines.

[0111] The signal sent to the column signal line is transmitted to the CDS-ADC circuit 18 (reference). Figure 1 The signal (pixel value) converted into a digital value by the CDS-ADC circuit 18 undergoes a uniform merging process by the merging processor 34.

[0112] For example, because the signals (pixel values) output from the first and fourth columns of pixel array 12 each contain signals from two pixels, the signals are multiplied by 2. The signals (pixel values) from the second and third columns of pixel array 12 are processed with a multiplication factor of 1. For example, refer to... Figure 13 For pixel group A1, the signal output from column signal line C1d is doubled, while the signal output from column signal line C3d is output with a multiplication factor of 1. These signals are summed to form the value of pixel A1_Gb extended in pixel group A1.

[0113] In this way, during the summation process, the pixel values ​​of pixels of the same color within a pixel group are summed. On the other hand, during the averaging process, the value after the summation process is divided by the number of pixels that are the target of the merging process (e.g., 3 pixels including Gb00, Gb20, and Gb02), so that the average value is determined.

[0114] 6. Uniform merging process during video image capture

[0115] The imaging device according to this embodiment can switch blocks of pixel groups for each frame during video image capture. Figure 3 Give an example of a group of pixels in a predefined frame. Figure 14 Give an example of a pixel group in a subsequent frame.

[0116] Compare Figure 3 and Figure 14 Combined processor 34 (reference) Figure 1 Pixel groups are configured in such a way that unused pixels (pixels whose signals are discarded) are complementary. For example, in Figure 14 In the middle, the pixel group is set to start from Figure 3 The pixel group is shifted two rows in the row direction and two columns in the column direction. By alternating... Figure 3 and 14 Grouping can visually reduce artifacts.

[0117] Figure 15 For example, in a predetermined frame (that is, based on) Figure 3 The image in which the uniform merging process is applied (on the left side of the figure) and in subsequent frames (that is, based on) Figure 14The image shows an image where a uniform merging process is applied (at the center of the figure). It is particularly clear from the lower image that the locations of artifacts become complementary due to the different pixel groupings between frames. For example, the advantage is that even if an artifact appears at a specific location in one image, no artifact will appear at the same location in another image. Therefore, in a video image displaying these images sequentially, artifacts are visually reduced, as shown on the right side of the figure.

[0118] 7. Uniform merging process based on concave rendering

[0119] Generally, the decision to perform a merging process is based on the brightness / darkness of pixel values. In addition, uniform merging and normal merging processes can be performed based on so-called concave rendering.

[0120] refer to Figure 1 The imaging system includes an eye-tracking mechanism 45. The eye-tracking mechanism 45 tracks the gaze of a user viewing multiple images displayed on the display device 40. For example, the eye-tracking mechanism 45 includes a near-infrared light source and a camera. In other words, the eye-tracking mechanism 45 is used to perform so-called non-contact eye tracking.

[0121] An eye-tracking mechanism 45 is used to define the user's gaze point. A merging processor 34 performs a uniform merging process on areas outside the gaze point. For example, as described above, the merging processor 34 performs a uniform merging process even for pixels whose brightness exceeds a threshold. In other words, by intentionally reducing the resolution of the image in areas other than the gaze point, the computational load associated with image display can be reduced.

[0122] refer to Figure 16 For example, when the center portion of the image displayed on the display device 40 in the height direction is the viewing area 42, the areas above and below the viewing area 42 are the non-viewing areas 44. The merging processor 34 performs uniform merging processing on the pixels corresponding to the non-viewing areas 44.

[0123] 8. (3×3) merge

[0124] In the embodiments described above, the (2×2) merging process is illustrated as a (k×k) merging process. However, the merging processor 34 according to this embodiment may also execute other merging processes.

[0125] Figure 17 This demonstrates a normal (3×3) merging process. In this exemplary configuration, three pixels are selected vertically and horizontally for each color. The size of the pixel group is (6×6). Figure 17 The image shows the gravity centers G_Gb, G_B, G_R, and G_Gr of the expanded pixels after the normal merging process. For example... Figure 17 As shown, the gravity centers G_Gb, G_B, G_R, and G_Gr are not uniformly placed near the center of the pixel group.

[0126] Figure 18 An example of a (3×3) uniform merging process is shown. Also in this exemplary configuration, signals from at least one pixel of each color are excluded (not used) from the processing target; said at least one pixel is positioned relatively closer to the center side of the pixel group. For example, for a Gb pixel, the pixel at coordinates (4,0), (0,4), (4,2), (2,4), and (4,4) is an unused pixel. Similarly, for a B pixel, the pixel at coordinates (1,0), (1,2), (1,4), (3,4), and (5,4) is an unused pixel; for an R pixel, the pixel at coordinates (0,1), (2,1), (4,1), (4,3), and (4,5) is an unused pixel; and for a Gr pixel, the pixel at coordinates (1,1), (3,1), (5,1), (1,3), and (1,5) is an unused pixel. Through this uniform merging process, as shown by gravity centers G_Gb, G_B, G_R, and G_Gr, the gravity centers of the corresponding colors are more evenly distributed compared to the gravity centers in a normal merging process (see [link]). Figure 17 ).

[0127] Figure 19 The presentation is different Figure 18 The selection is an example configuration that does not use pixels. In this example configuration, in Figure 18 In the exemplary configuration, the pixels at coordinates (1,0), (4,0), (0,1), (5,1), (0,4), (5,4), (1,5), and (4,5) that are set as unused pixels are the targets of the merging process (those pixels are the used pixels).

[0128] Using this configuration, in Figure 19 In the exemplary configuration, with Figure 18 In comparison, the center of gravity of the corresponding color shifts towards the center, but compared to... Figure 17 The normal merging process shown in the diagram is more evenly distributed compared to the previous one. Additionally, because... Figure 19 The number of signals to be discarded in the exemplary configuration and Figure 18 Since there are fewer of them, it is expected that summing and averaging through the merging process will have a greater advantage.

[0129] 9. (4×4) merge

[0130] Figure 20 This demonstrates a normal merging process (4×4). In this exemplary configuration, four pixels are selected vertically and horizontally for each color. The size of the pixel group is (8×8). Figure 20 Examples are given to illustrate the gravity centers G_Gb, G_B, G_R, and G_Gr of the expanded pixels after a normal merging process. For example... Figure 20 As shown, the gravity centers G_Gb, G_B, G_R, and G_Gr are not uniformly placed near the center of the pixel group.

[0131] Figure 21 An exemplary configuration for a (4×4) uniform merging process is shown. Also in this exemplary configuration, signals from at least one pixel of each color are excluded from the processing target (these pixels are not used), and this at least one pixel is positioned relatively closer to the center side of the pixel group. For example, for Gb pixels, the pixels at coordinates (6,0), (6,2), (4,4), (6,4), (0,6), (2,6), (4,6), and (6,6) are unused pixels. Similarly, for pixel B, the pixels at coordinates (1,0), (1,2), (1,4), (3,4), (1,6), (3,6), (5,6), and (7,6) are unused pixels; for pixel R, the pixels at coordinates (0,1), (2,1), (4,1), (6,1), (4,3), (6,3), (6,5), and (6,7) are unused pixels; and for pixel Gr, the pixels at coordinates (1,1), (3,1), (5,1), (7,1), (1,3), (3,3), (1,5), and (1,7) are unused pixels. Using this uniform merging process, as shown by the gravity centers G_Gb, G_B, G_R, and G_Gr, the gravity centers of the corresponding colors are more evenly distributed compared to the gravity centers of a normal merging process (see reference). Figure 20 ).

[0132] Figure 22 The presentation is different Figure 21 An exemplary configuration for selecting unused pixels. In this exemplary configuration, the pixels at coordinates (1,0), (6,0), (0,1), (7,1), (0,6), (7,6), (1,7), and (6,7) are the targets of the merging process (these pixels are the used pixels), and the pixels are... Figure 21 Unused pixels in the exemplary configuration. On the other hand, pixels at coordinates (2,3), (5,3), (2,4), and (5,4) (which are in Figure 21 In the example configuration, the pixels that are used are pixels that are not used.

[0133] Using this configuration, in Figure 22 In the exemplary configuration, the gravity centers of the corresponding colors are distributed in a manner similar to... Figure 21 To what extent. Also, because of... Figure 21 compared to Figure 22Since fewer signals are discarded during the merging process, it is expected that summing and averaging through the merging process will have greater advantages.

[0134] This disclosure is not limited to the embodiments described above, and includes all changes and modifications that do not depart from the technical scope or essence of this disclosure as defined by the claims.

Claims

1. An imaging device comprising: A pixel array, in which multiple pixels are arranged in a two-dimensional manner; A color filter array is disposed on the pixel array, and the R color filter, G color filter and B color filter are arranged in the color filter array in a Bayer arrangement; and A merging processor is configured to perform a (k×k) merging process that treats k pixels vertically and k pixels horizontally in the pixel array as a pixel group, where k is an integer. The merging processor is capable of performing a uniform merging process, wherein signals from at least one pixel of each color are excluded from the processing target, the at least one pixel being positioned relatively closer to the center side of the pixel group.

2. The imaging device according to claim 1, wherein The merging processor can switch between a normal merging process that uses signals from all the pixels in the pixel group and a uniform merging process.

3. The imaging device according to claim 1, wherein... The merge processor is configured to perform a (2×2) merge process. Two row selection lines are provided for one row of the pixel array, and The connection destination of the plurality of pixels in the pixel array is switched between one and the other of the row selection lines for every two pixels along the row direction.

4. The imaging device according to claim 3, wherein Four column signal lines are provided for one column of the pixel array. In the color filter array, color filters of two colors are arranged alternately along the column direction. The column signal lines are assigned to each color of the two color filters, and For a pair of pixel groups along the column direction, the column signal line connected to one of the pixel groups is different from the column signal line connected to the other of the pixel groups.

5. The imaging device according to claim 1, wherein... During video image capture, the merging processor is configured to shift the plurality of pixels to be grouped in two dimensions between the pixel group in a predetermined frame and the pixel group in a subsequent frame.

6. The imaging device according to claim 1, further comprising: A display on which an image of the pixel array is displayed; and An eye-tracking mechanism that measures the line of sight directed toward the display, wherein The merging processor is configured to perform the merging process for areas outside the gaze point of the line of sight.

7. An imaging method performed in an imaging device, the imaging device comprising: A pixel array, in which multiple pixels are arranged in a two-dimensional manner; as well as A color filter array is disposed on the pixel array, and the R, G, and B color filters are arranged in the color filter array in a Bayer arrangement. The method includes: Perform a (k×k) merging process that treats k pixels in the vertical direction and k pixels in the horizontal direction of the pixel array as a pixel group, where k is an integer; and A uniform merging process is performed as the merging process, wherein signals from at least one pixel of each color are excluded from the processing target, the at least one pixel being positioned relatively closer to the center side of the pixel group.

8. The method according to claim 7, wherein The merging process can switch between a normal merging process that uses signals from all the pixels in the pixel group and a uniform merging process.

9. The method of claim 7, wherein During video image capture, the plurality of pixels to be grouped are shifted in two dimensions between the pixel group in a predetermined frame and the pixel group in a subsequent frame.

10. The method of claim 7, wherein The imaging device further includes: A display showing an image of the pixel array; and An eye-tracking mechanism that measures the line of sight directed toward the display, and The merging process is performed on the area outside the gaze point of the line of sight.

Images