Imaging device

Abstract

To provide an imaging element and an imaging device for performing local high frame rate imaging or high resolution imaging for each imaging area, reducing artifacts that may occur at boundary portions between areas of different imaging modes, and improving the image quality of an entire screen.SOLUTION: An imaging device includes an imaging element having a function of grouping adjacent 2×2 pixels each having a photoelectric conversion portion into one pixel block, and performing switching between an operation of binning and reading all charges accumulated in the photoelectric conversion portions of 4 pixels of the pixel block every frame or an operation of sequentially reading out charges corresponding to a plurality of frames accumulated in the photoelectric conversion portion of one pixel of the pixel block pixel by pixel for each frame, and a signal processing unit that performs calculation matched with the mode selection signal for each output signal corresponding to each pixel block and generates a video signal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] This invention acquires video signals. To take a photo Regarding imaging devices. [Background technology]

[0002] In recent years, signal processing in imaging devices has been digitized, and the output signals of image sensors are also digital signals. To accommodate higher resolution (more pixels), column-parallel ADCs, which provide an ADC (analog-to-digital converter) for each column of the pixel array, are widely used in the high-speed pixel signal readout circuits of CMOS (Complementary Metal Oxide Semiconductor) image sensors (for example, Patent Document 1).

[0003] Signal readout period T per pixel row of an image sensor using a column-parallel ADC H V represents the number of pixels in the vertical direction (signal readout direction) of the image sensor, and f represents the frame rate. f It is inversely proportional to . Therefore, speeding up the readout circuit is essential for realizing high-resolution, high-frame-rate image sensors. For example, the signal readout period T per row of an 8K / 240fps image sensor using a column-parallel ADC H This is less than 1 μs. Furthermore, when performing fixed pattern noise reduction using digital CDS (correlated double sampling), it is necessary to perform AD conversion on both the pixel reset level and the signal level for each readout operation, thus requiring high-speed AD conversion of 0.5 μs or less.

[0004] Generally, the operating speed of an ADC is in a trade-off relationship with other performance aspects such as noise performance, bit depth, power consumption, and ADC area. Furthermore, in high-resolution, high-speed image sensors, as mentioned above, the allowable time for AD conversion becomes shorter, leading to challenges such as noise performance degradation, insufficient bit depth, increased power consumption, and increased chip area, all stemming from the limitations of the ADC's operating speed.

[0005] To address this challenge and achieve substantial image quality improvement within a limited AD conversion speed, a conventional method has been proposed in which the image sensor is equipped with multiple operating modes specialized for specific imaging performance, such as frame rate and resolution, and the optimal operating mode is selected according to the characteristics of the shooting scene (Patent Document 2). For example, by selecting a mode that is appropriate for the shooting scene, such as using a high frame rate / low resolution mode when imaging fast-moving subjects, or a low frame rate / high resolution mode when imaging high-definition subjects with little movement, substantial image quality improvement can be achieved. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2020-188360 [Patent Document 2] Japanese Patent Publication No. 2020-141405 [Overview of the project] [Problems that the invention aims to solve]

[0007] However, conventional mode selection specialized the entire image sensor (imaging area) into an operating mode focused on a specific imaging performance. Even if a mode selection function is provided on the image sensor, in typical shooting scenes, it is expected that multiple subjects with different characteristics will be captured simultaneously within the same image frame. This effect is particularly pronounced when performing wide-field imaging such as 360° video, where multiple shooting scenes with different characteristics are expected to coexist simultaneously. In this case, for example, if high-frame-rate mode is selected to match a fast-moving subject in the frame, the resolution of stationary, high-definition subjects simultaneously captured in the same frame will decrease. Conversely, if high-resolution mode is selected to match a high-definition subject in the frame, fast-moving subjects will become blurred, making it difficult to capture their movement.

[0008] Therefore, in view of the above problems, an object of the present invention is to achieve imaging by local high frame rate imaging or high resolution imaging for each imaging region, and to reduce artifacts that may occur at the boundary portions of regions with different imaging modes, thereby improving the image quality of the entire screen. Na To provide an imaging device.

Means for Solving the Problems

[0009] In order to solve the above problems, an image sensor according to the present invention (1) Pixels each having a photoelectric conversion section are grouped into one pixel block for every adjacent 2×2 pixels, and includes a pixel array constituted by the pixel blocks, and a column readout circuit provided corresponding to the columns of the pixel blocks for reading out the output signals of each pixel block. According to a mode selection signal set for each pixel block, for each frame, all the charges accumulated in the photoelectric conversion sections of the 4 pixels of the pixel block are binned and read out, or the charges for a plurality of frames accumulated in the photoelectric conversion section of one pixel of the pixel block are read out one pixel at a time in order for each frame. It is characterized by having a function of switching.

[0010] In order to solve the above problems, an imaging device according to the present invention (2) includes the image sensor, and a signal processing section that performs an operation corresponding to the mode selection signal on the output signal of the image sensor for each output signal corresponding to each pixel block to generate a video signal.

[0011] (3) In the imaging device of (2) above, it is desirable that the signal processing section generates a video signal in a high-speed imaging mode that outputs a signal for each frame based on the output signal obtained by binning and reading out all the charges accumulated in the photoelectric conversion sections of the 4 pixels of the pixel block for each frame, and generates a video signal in a high-resolution imaging mode with a resolution corresponding to the pixels of the pixel block based on the output signal obtained by reading out the charges for a plurality of frames accumulated in the photoelectric conversion section of one pixel of the pixel block one pixel at a time in order for each frame.

[0012] (4) The imaging device according to (2) or (3) above includes a prism that decomposes incident light by color, a plurality of the image sensors arranged corresponding to the colors decomposed by the prism, and a drive signal generation unit that outputs a drive signal to the image sensors according to the mode selection signal. It is desirable that the signal processing unit performs an operation according to the mode selection signal on the output signals of the plurality of image sensors for each output signal corresponding to each pixel block to generate a video signal.

[0013] (5) In the imaging device according to (4) above, the prism is a four-plate prism, four image sensors are arranged corresponding to Rch, G1ch, G2ch, and Bch, and it is desirable that the drive signal generation unit controls, for pixel blocks in the high-resolution shooting mode, the corresponding pixel blocks of the plurality of image sensors to output the signals of the pixel positions shifted from each other for each image sensor.

[0014] (6) In the imaging device according to (4) or (5) above, it is desirable that the drive signal generation unit controls, for pixel blocks in the high-resolution shooting mode, the corresponding pixel blocks of the plurality of image sensors to output signals of pixel positions that together form a Bayer array.

[0015] (7) In any of the imaging devices according to (4) to (6) above, it is desirable that the signal processing unit selects either a method of generating a video signal by performing demosaicing processing on the output signals read out one pixel at a time for each frame of the plurality of image sensors in the high-resolution shooting mode for each frame, or a method of generating a video signal by collecting the output signals corresponding to each pixel of the pixel blocks of the image sensors as signals of each pixel position every four frames.

[0016] (8) In any of the imaging devices according to (4) to (7) above, it is desirable that the signal processing unit adopts a method of generating a video signal by performing demosaicing processing on at least the pixel blocks at the boundary portions with the pixel blocks in the high-speed shooting mode among the pixel blocks in which the high-resolution shooting mode is selected for each frame.

Advantages of the Invention

[0017] Imaging in the present invention Device According to this invention, even in imaging scenes where fast-moving subjects and high-resolution subjects are simultaneously captured within the same frame, localized high-frame-rate shooting or high-resolution shooting can be achieved for each imaging area. Furthermore, the imaging device according to this invention can reduce artifacts that may occur at the boundaries of areas with different shooting modes, thereby improving the overall image quality of the screen. [Brief explanation of the drawing]

[0018] [Figure 1] This is an example of the configuration of an imaging device according to one embodiment of the present invention. [Figure 2] This is an example of the configuration of an image sensor according to one embodiment of the present invention. [Figure 3] This is an example of the circuit configuration of a pixel block in an image sensor. [Figure 4] This is an example of a timing chart for drive signals during each readout operation of a pixel block. [Figure 5] This figure shows an example of the operating timing of a row of image sensors. [Figure 6] This is an example of a feedback signal F and the output signal of the corresponding pixel block. [Figure 7] This figure shows an example of the output operation of each channel for each frame period. [Figure 8] This diagram illustrates an example of signal processing using an artifact reduction method. [Figure 9] This diagram illustrates an example of signal processing using a full-resolution method. [Modes for carrying out the invention]

[0019] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0020] (Embodiment) Figure 1 shows an example of the configuration of an imaging device according to one embodiment of the present invention. The imaging device comprises a lens 10, a four-plate prism 20, a feedback signal (mode selection signal) generation unit 30, a drive signal generation unit 40, image sensors 100 for each channel, and a signal processing unit 50. The feedback signal generation unit 30 may be configured as a device independent of the imaging device.

[0021] The lens 10 focuses the incident light from the subject and guides it to the four-plate prism 20. The four-plate prism 20 separates the incident light into four channels (ch) of optical signals: R, G1, G2, and B. G1 and G2 are G signals with the same wavelength band obtained by splitting the G signal into two using a beam splitter or the like. The feedback signal (mode selection signal) generation unit 30 generates a feedback signal (mode selection signal) F according to the image. The drive signal generation unit 40 receives the feedback signal F and outputs drive signals (R, G1, G2, B) corresponding to the feedback signal F to the image sensor 100 of each channel.

[0022] Four image sensors 100 are arranged, corresponding to the optical signals of each channel, R, G1, G2, and B. Each channel image sensor 100 is configured as a 2x2 pixel unit, as described later, and converts the subject image into an image sensor output signal based on the drive signal received from the drive signal generation unit 40, and transmits it to the signal processing unit 50. The signal processing unit 50 receives the image sensor output signals (R, G1, G2, B) and the feedback signal F, generates a video signal, and outputs it.

[0023] In this embodiment, a four-plate prism 20 is used, and four image sensors 100 corresponding to four channels R, G1, G2, and B are used. However, the imaging device of the present invention can employ any optical system for acquiring color images. In this embodiment, by processing the four video signals of R, G1, G2, and B, existing signal processing related to Bayer arrays can be utilized, as described later.

[0024] The following describes in detail the operating principles of each component and the imaging device.

[0025] (1) Image sensor Figure 2 shows an example of the configuration of an image sensor 100 according to one embodiment of the present invention. The image sensor 100 includes a pixel array 110, a row selection circuit 120, a read column control circuit 130, and a column read circuit 140.

[0026] The pixel array 110 is composed of H horizontal x V vertical pixel blocks 111 (2H horizontal x 2V vertical pixels), where adjacent horizontal 2 pixels x vertical 2 pixels (abbreviated as 2x2 pixels) are the unit (hereinafter referred to as "pixel block"), and each pixel block 111 can be controlled individually. In this embodiment, one signal readout line PIXOUT and four readout column control lines (TXA~TXD) are wired vertically for each row of pixel blocks. The readout column control circuit 130 controls the readout column control lines (TXA~TXD), and the signal readout line PIXOUT is connected to a column readout circuit 140 (141, 142...) corresponding to each row. The column readout circuit 140 reads the signals from each pixel block and performs signal processing such as ADC and digital CDS. Furthermore, three types of signal lines—row selection SL, transfer gate control TX, and reset RT—are horizontally wired from the row selection circuit 120 to each row of pixel blocks. The row selection circuit 120 scans these signal lines vertically to drive the pixel blocks 111 one row at a time.

[0027] Figure 3 shows an example of the circuit configuration of the pixel block 111 of the image sensor 100. The pixel block 111 comprises four photoelectric conversion units (photodiodes in this embodiment) PD(A) to PD(D) corresponding to four pixels. The four photodiodes PD(A) to PD(D) share a floating diffusion (FD) node. The gate electrode of the charge transfer transistor TGT of each photodiode PD receives the transfer gate control signal (TX signal) from the row selection circuit 120 via the source and drain terminals of the transfer operation control transistor TGC. Read row control signal lines (TXA to TXD) are connected to the gate electrode of the transfer operation control transistor TGC, allowing the transfer gate control signal (TX signal) to be enabled or disabled for each PD(A) to PD(D). The potential of the floating diffusion (FD) node is input to the gate electrode of the source follower transistor SFT connected to the power supply PIXVDD. When the selection transistor SLT, controlled by the row selection signal (SL signal), conducts, the output signal of the source follower transistor SFT is output to the signal readout line PIXOUT. The reset transistor RTT is controlled by the reset signal (RT signal), and when it conducts, it discharges the charge accumulated in the floating diffusion transistor FD.

[0028] Figure 4 shows an example of a timing chart for the drive signals during each readout operation of the pixel block 111. The pixel block 111 has five types of readout operations. Each timing chart in Figure 4 corresponding to each readout operation (TXF, TXA, TXB, TXC, TXD) shows the time (1 / f) during which one pixel block row is selected in one frame. f This shows each signal within (V seconds).

[0029] The row selection signal (SL signal) is common to each readout operation, and the period during which a pixel block row is selected (1 / f fThe voltage (V seconds) is always at a level that conducts the selection transistor SLT (in this embodiment, the signal that conducts each transistor is set to High). The reset signal (RT signal) is also common to each read operation and goes High at some point before the photodiode PD is read out (the charge transfer transistor TGT becomes conductive), discharging the charge from the floating diffusion FD.

[0030] The readout operation TXF is a readout operation used during high-speed imaging (high frame rate imaging). At a predetermined timing, the transfer gate control signal (TX signal) and the control signals (TXA to TXD) of the transistor TGC that controls the transfer operation of the four photodiodes PD(A) to PD(D) all become high. Therefore, in the readout operation TXF, the charge of all photodiodes PD(A) to PD(D) (in this case, the charge accumulated during one frame period) is read out to the floating diffusion FD. In this way, all the charge accumulated in the photoelectric conversion section of the four pixels of the pixel block 111 can be binned and read out for each frame.

[0031] The readout operation TXA is a readout operation used during high-resolution imaging. At a predetermined timing, the transfer gate control signal (TX signal) and the control signal (TXA) of the transistor TGC that controls the transfer operation of one photodiode PD(A) become high. Therefore, in readout operation TXA, the charge of only photodiode PD(A) is read out to the floating diffusion FD. During this period, the other photodiodes PD(B) to PD(D) continue to accumulate photocharge. As described later, readout operations TXA to TXD are performed sequentially, and readout operation TXA is performed every four frames. Therefore, in this readout operation, the charge of photodiode PD(A) accumulated during the four-frame period is read out.

[0032] The readout operation TXB is a readout operation used during high-resolution imaging. At a predetermined timing, the transfer gate control signal (TX signal) and the control signal (TXB) of the transistor TGC that controls the transfer operation of one photodiode PD(B) become high. Therefore, in the readout operation TXB, only the charge of photodiode PD(B) (in this case, the charge accumulated over a 4-frame period) is read out to the floating diffusion FD. During this period, the other photodiodes PD(A), PD(C), and PD(D) continue to accumulate photocharge.

[0033] The readout operation TXC is a readout operation used during high-resolution imaging. At a predetermined timing, the transfer gate control signal (TX signal) and the control signal (TXC) of the transistor TGC that controls the transfer operation of one photodiode PD(C) become high. Therefore, in the readout operation TXC, only the charge of photodiode PD(C) (in this case, the charge accumulated over a 4-frame period) is read out to the floating diffusion FD. During this period, the other photodiodes PD(A), PD(B), and PD(D) continue to accumulate photocharge.

[0034] The readout operation TXD is a readout operation used during high-resolution imaging. At a predetermined timing, the transfer gate control signal (TX signal) and the control signal (TXD) of the transistor TGC that controls the transfer operation of one photodiode PD(D) become high. Therefore, in the readout operation TXD, only the charge of photodiode PD(D) (in this case, the charge accumulated over a 4-frame period) is read out to the floating diffusion FD. During this period, the other photodiodes PD(A) to PD(C) continue to accumulate photocharge.

[0035] Immediately after switching the feedback signal from high-speed shooting mode to high-resolution shooting mode, the charge accumulated in each pixel may be less than 4 frames. However, in the steady state of high-resolution shooting mode, read operations TXA to TXD are repeated in sequence, and 4 frames' worth of charge is read from each pixel.

[0036] FIG. 5 is an example of the operation timing of a certain column of the image sensor 100. This figure shows, as an example, the operation timing when high-resolution shooting is performed only for a specific pixel block column (k-th column) of the image sensor 100 in one channel (for example, Rch) from the m1-th row to the m2-th row, and high-speed shooting is performed for the other rows. In FIG. 5, the horizontal axis is the time axis, the vertical axis is the position of the row, and the diagonal lines indicate which row is operating in what read operation at each time point.

[0037] The row selection circuit 120 scans from the first row to the V-th row of the pixel block within a period T f = 1 / f f (seconds) of one frame. The control signals TXA to TXD from the readout column control circuit 130 are different for each frame at a 4-frame period (f1 to f4) as shown in FIG. 4. That is, in the readout operation from the m1-th row to the m2-th row, in the first frame (f1), the readout operation TXA is performed, in the second frame (f2), the readout operation TXB is performed, in the third frame (f3), the readout operation TXC is performed, and in the fourth frame (f4), the readout operation TXD is performed. In the readout operation of the other rows, the readout operation TXF is performed regardless of the frame.

[0038] By performing the above operation, for the charge accumulated in the photoelectric conversion section of one pixel of the pixel block 111 from the m1-th row to the m2-th row, the charges for a plurality of frames are read out one pixel at a time for each frame. That is, for each of the charges accumulated in the photodiodes PD(A) to PD(D), an image signal with an exposure time of 4 / f f is obtained, so that a high-resolution image with a resolution corresponding to the pixel is realized. <

[0039] In the other rows, the charges accumulated in the photodiodes PD(A) to PD(D) are binned, and an image signal with a frame rate of f f is obtained for each pixel block 111, so that high-speed (high frame rate) shooting is realized.

[0040] Figure 5 illustrates the case where rows m1 to m2 are captured in high-resolution mode. However, the operating mode can be arbitrarily specified for each row, and the operating mode can be set independently for each column. Therefore, the operating modes for high-resolution and high-speed shooting can be freely set for each 2x2 pixel block 111. In other words, the image sensor 100 has the function of switching between high-speed shooting and high-resolution shooting for each pixel block 111 according to the feedback signal (mode selection signal) F.

[0041] (2) Feedback signal (mode selection signal) generation unit The feedback signal (mode selection signal) generation unit 30 outputs a feedback signal F. The feedback signal F is a signal that selects (specifies) the operating mode of each pixel block, and is sometimes called a mode selection signal. The feedback signal F is represented, for example, by a V row and H column array corresponding to the array of pixel blocks 111, and the elements of the array F mn The value of determines whether to perform high-speed or high-resolution shooting for each pixel block 111. That is, element F mn This allows specifying the operating mode of the pixel block 111 in the mth row and nth column. In this embodiment, as an example, F mn When =0, high-speed photography, F mn When = 1, it is defined as high-resolution shooting, but the definition of mode specification is not limited to this.

[0042] The feedback signal is an element F of F mn The method for generating the element is not limited in this invention. However, in order to improve effective image quality, it is desirable to capture stationary objects at high resolution and moving objects at high speed. For example, it is desirable to acquire the features (motion) of each region of the image using motion estimation of the subject, and to feed back an appropriate operating mode based on these features to the image sensor 100 as a feedback signal F. Also, in this embodiment, as shown in Figure 5, in high-resolution shooting, the video signal is acquired as a set of four readout operations TXA to TXD, so element F mn It is desirable that this be updated once every four frames.

[0043] As shown in Figure 1, the feedback signal (mode selection signal) generation unit 30 supplies the same feedback signal F to the drive signal generation unit 40 and the signal processing unit 50 that control the image sensors 100 of each channel. The feedback signal generation unit 30 may be provided outside the imaging device. Furthermore, instead of providing the generation unit 30, the feedback signal may be input as an external signal, or it may be freely set manually or by other means.

[0044] Further methods for generating the feedback signal F include those shown in (a) to (c) below.

[0045] (a) The user can input a value they have arbitrarily set, either as a fixed value or a value that is updated at a certain frequency. For example, if the position of a stationary object or a moving object is fixed on the screen, the mode of that area can be set as a fixed value for the feedback signal.

[0046] (b) The system determines the area of ​​interest on the screen, the area in focus, the speed of the moving subject, etc., based on the position and positional information of the subject acquired by the distance sensor and lens parameters (focus position, focal length, f-number), and generates a feedback signal in real time based on the results.

[0047] (c) Using image analysis techniques such as object recognition, divide the region for each subject and select an appropriate F according to the characteristics of each subject (estimated spatial frequency components and movement speed). mn The system automatically provides the necessary settings. For example, it recognizes what the subject is through image analysis and automatically sets the shooting mode to high-speed mode if the subject is moving, and the shooting mode to high-resolution mode if the subject is stationary.

[0048] Ideally, the feedback signal F should be updated every four frames, but an appropriate update cycle may be selected depending on the subject's movement speed and the rate of change in the shooting scene over time.

[0049] Figure 6 shows an example of the feedback signal F and the corresponding output signal of the pixel block 111. Here, we show the control of one image sensor (e.g., Rch) by the feedback signal F. Figure 6(a) is an example of the feedback signal (mode selection signal) F. Element F of the feedback signal F mn By taking a value of 0 or 1, the pixel block 111 of the pixel array 110 is divided into two types of imaging areas with different operating modes.

[0050] Figure 6(b) shows the output signal of each pixel block in each frame (f1 to f4). Feedback signal F (element F) mn In the region where ) is 0, the exposure time for each frame is 1 / f f Since a signal is obtained by binning the charge accumulated in PD(A) to PD(D), localized high-frame-rate imaging (high-speed imaging) can be realized. Element F mn In the region where the value is 1, the signal of the pixel position shown in the diagram is output for each frame. That is, the exposure time is 4 / f f Since the charge signals accumulated in each photodiode PD(A) to PD(D) can be acquired independently for each frame, localized high-resolution imaging can be achieved.

[0051] As described above, with the image sensor 100 according to this embodiment, it is possible to specify for each pixel block 111 which performance, resolution or frame rate, to prioritize using a feedback signal (mode selection signal) F. Therefore, even in imaging scenes where fast-moving subjects and high-resolution subjects are simultaneously captured in the same frame, it is possible to improve the overall image quality of the screen by achieving localized high-frame-rate shooting or high-resolution shooting.

[0052] (3) Drive signal generation unit As shown in Figure 1, the drive signal generation unit 40 receives a feedback signal (mode selection signal) F and outputs drive signals (R, G1, G2, B) corresponding to the feedback signal F to the image sensor 100 of each channel. The drive signals (R, G1, G2, B) control the operating mode (readout operation) of each pixel block 111 of the image sensor 100 of each channel.

[0053] The following describes how to control each channel to improve the overall image quality of the color image.

[0054] Figure 7 shows an example of the output operation of each channel for each frame period f1 to f4. Here, the relationship between each channel (R, G1, G2, B) and which pixel (photoelectric conversion unit) outputs for a specific pixel block 111 in each frame is shown. Since the feedback signal (mode selection signal) F is common to all channels, the corresponding pixel blocks 111 in each channel have the same shooting mode (high-speed shooting or high-resolution shooting). However, in high-resolution shooting mode, the drive signal generation unit 40 controls the corresponding pixel blocks 111 of the image sensor 100 in each channel using drive signals (R, G1, G2, B) so that they operate with different readout operations (TXA to TXD).

[0055] In the high-speed shooting mode, as shown in Figure 7(a), the pixel block 111 is read out by the readout operation TXF for all channels. That is, for every frame period f1 to f4, the exposure time 1 / f is read out from the pixel block 111 of all channels. f The signal obtained by binning the charge accumulated in PD(A) to PD(D) is output.

[0056] In the pixel block 111 of the high-resolution shooting mode, as shown in Figure 7(b), the readout operations TXA to TXD for each channel (R, G1, G2, B) are different for each frame period f1 to f4. Therefore, for each frame period f1 to f4, the charge signals accumulated in the photodiodes PD(A) to PD(D), which are located at different positions in each channel (exposure time is 4 / f fThe charge signal is read out. Under the control of the drive signal generation unit 40, each channel performs readouts in frame periods f1 to f4 with the phases of read operations TXA to TXD shifted relative to each other.

[0057] In high-resolution shooting mode, during each frame period f1 to f4, the signals output from each channel have shifted pixel positions, but the exposure period is the same. When the output signals from all channels (R, G1, G2, B) are combined, an image equivalent to a Bayer array image is obtained.

[0058] (4) Signal Processing Unit The signal processing unit 50 receives the image sensor output signal and the feedback signal (mode selection signal) F from the image sensor 100, and processes the elements of the signal F. mn Based on the operating mode of the m-th row, n-th column pixel block specified by the signal, calculations corresponding to signal F are performed on the output signal corresponding to each pixel block of the image sensor 100 to generate a video signal. A specific example of signal processing is described below.

[0059] (i)F mn =0 (high-speed photography) The signal processing unit 50 outputs the output signal of the corresponding pixel block (a pixel block of m rows and n columns) output from the image sensor 100 of each channel as a high-speed captured video signal. That is, the exposure time 1 / f is obtained by binning the charge accumulated in the photoelectric conversion units PD(A) to PD(D). f The signal is transmitted with a resolution of V x H pixels and a frame rate of f f The signal is output as such. The signal processing unit 50 can further convert the signal into a video signal compatible with the display device as appropriate. For example, the signal processing unit 50 can apply any upconversion process to the output signal of the pixel block in order to display it on a 2V x 2H pixel display device.

[0060] (ii)F mn =1 (high-resolution shooting) Each channel's image sensor 100 outputs a signal corresponding to the pixel shown in Figure 7(b). Therefore, from that pixel block (a pixel block of m rows and n columns), a color signal similar to the image signal obtained when using a Bayer array color filter in a single-chip camera, as shown in Figures 8 and 9, is output with a 1 / f f The data is acquired at the following intervals. The signal processing unit 50 generates a video signal by performing signal processing on the acquired signal using either (a) artifact reduction method or (b) full resolution method, as shown below.

[0061] (a) Artifact reduction methods Figure 8 illustrates an example of an artifact reduction signal processing method. The signal processing unit 50 performs demosaicing on the 4 channels of signals corresponding to the Bayer array output during each frame period f1 to f4, frame by frame, and the exposure time 4 / f f , frame rate f f This generates a video signal with a resolution of 2H × 2V. The demosaicing algorithm can be any algorithm and is not limited to any particular method. In this case, the pixel values ​​of all pixels in all channels are estimated from the Bayer array image (for all pixels, R and B are 1 / 4 and G is 1 / 2) by demosaicing, so it is not strictly high-definition video, but in return the frame rate f f This method yields high frame rate video. In other words, the video obtained using this method occupies an intermediate position between high-speed shooting and high-resolution shooting.

[0062] Here, the R, G1, G2, and B channel signals obtained in each frame period have matching exposure periods, and there is no shift in image position even for moving subjects. Therefore, the image obtained by demosaicing all channel signals in each frame period has reduced artifact occurrence.

[0063] (b) Full resolution method Figure 9 illustrates an example of signal processing using the full-resolution method. The signal processing unit 50 extracts and collects the photoelectric conversion unit PD(A) to PD(D) signal values ​​for each channel (R, G1, G2, B) from the Bayer array signal output during the frame period f1 to f4 using a frame memory, and then calculates the exposure time 4 / f f , frame rate f f A 2H×2V image is generated. The G1 and G2 signals at the same position in the resulting image are added together to obtain the G signal. Through the above signal processing, a full-resolution RGB signal is generated as a video signal. This method allows for the acquisition of clear, high-resolution images of stationary subjects.

[0064] The following is a supplementary explanation regarding the selection of the two methods described above. When the video signal is acquired using the full-resolution method (b), the exposure time of the photoelectric conversion units PD(A) to PD(D) is 1 / f f Because there are differences between each, this is not a problem when the subject is stationary, but if the subject moves even slightly, artifacts caused by the shift in exposure time and the movement of the subject occur, resulting in a significant degradation of image quality. These artifacts are more likely to occur at the boundary between high-speed shooting mode and high-resolution shooting mode, where errors in motion estimation are likely to occur, for example, when specifying modes for each region using motion estimation of the subject. In this embodiment, as a countermeasure when the above situation occurs in the signal processing of method (b), the artifact reduction method signal processing of method (a) can be used to suppress significant image quality degradation due to the occurrence of artifacts.

[0065] The selection between method (a) and method (b) can be made by inputting a signal to the signal processing unit 50 to indicate which method to select, or by setting the system to automatically select the method, with at least the pixel blocks at the boundary between the high-resolution shooting mode and the high-speed shooting mode area being treated as method (a) artifact reduction, and the other areas being treated as method (b) full-resolution. Furthermore, since the selection between method (a) and method (b) can be made after shooting in high-resolution shooting mode, for example, the output signal from the image sensor 100 may be stored in memory, and the generated images from each method may be compared to select the signal processing method deemed preferable.

[0066] As described above, the imaging device of this embodiment enables imaging by localized high-frame-rate shooting or high-resolution shooting for each imaging area, reduces artifacts that may occur at the boundaries of areas with different shooting modes, and improves the overall image quality of the screen.

[0067] (Modified examples of the embodiment) The image sensor 100 in this embodiment may be configured as follows:

[0068] • Modification of control units In Figure 2, the control unit of a pixel block is exemplified as the smallest case, where the pixels share a floating diffusion FD (2x2 pixels), but the size of the control unit is arbitrary. For example, if an 8x8 pixel (4x4 pixel block) is used as the control unit, the number of read train control lines (TXA~TXD) can be reduced to 1 / 4. In addition, the transfer operation control transistor TGC can be shared by the 4x4 pixel block, and the number of such transistors can be reduced to 1 / 16, resulting in a significant reduction in pixel area.

[0069] • Modified version of the column reading circuit In the embodiment shown in Figure 2, H column readout circuits 140 are provided below the pixel array 110, and the column readout circuits are connected in parallel to the pixel block rows. However, the number of parallel column readout circuits for a pixel block row is arbitrary. That is, if two column readout circuits (2H in total) are provided for each pixel block row, the number of pixel blocks 111 processed by one column readout circuit 140 is halved. Furthermore, even more column readout circuits can be provided. This allows for sufficient processing time for the ADC, thereby improving the performance of the ADC.

[0070] Furthermore, although the configuration and operation of the image sensor and imaging device have been described in the above embodiments, the present invention is not limited to these, and may be configured as an imaging method for generating video signals. That is, it may be configured as a method for generating high-speed or high-resolution video signals using each block of the imaging device according to the data flow in Figure 1.

[0071] Although the embodiments described above are representative examples, it will be apparent to those skilled in the art that many modifications and substitutions are possible within the spirit and scope of the present invention. Therefore, the present invention should not be interpreted as being limited by the embodiments described above, and various modifications or changes are possible without departing from the scope of the claims. For example, the functions, etc., included in each block, step, etc., described in the embodiments can be rearranged in a logically consistent manner, and multiple constituent blocks, steps, etc., can be combined into one or divided. [Explanation of Symbols]

[0072] 10 lenses 20 4-plate prism 30 Feedback signal generation unit 40 Drive signal generation unit 50 Signal Processing Unit 100 image sensors 110-pixel array 111 pixel blocks 120-row selection circuit 130 Readout Column Control Circuit 140 Column readout circuit

Claims

1. A pixel array is formed by dividing a pixel equipped with a photoelectric conversion unit into a pixel block for every 2x2 adjacent pixels, and configuring the pixel array with such pixel blocks. The system includes a column readout circuit provided in correspondence with the rows of pixel blocks, which reads out the output signal of each pixel block, An image sensor having a function to switch, according to a mode selection signal set for each pixel block, between binning and reading out all the charge accumulated in the photoelectric conversion section of the four pixels of the pixel block for each frame, or sequentially reading out the charge for multiple frames accumulated in the photoelectric conversion section of one pixel of the pixel block, one pixel at a time, for each frame. A signal processing unit that performs calculations corresponding to the mode selection signal on the output signal of the image sensor for each output signal corresponding to each pixel block to generate a video signal, In an imaging device comprising, A prism that separates incident light into its respective colors, A plurality of image sensors arranged in accordance with the colors separated by the prism, A drive signal generation unit that outputs a drive signal to the image sensor in accordance with the mode selection signal, Equipped with, The signal processing unit performs calculations on the output signals of the multiple image sensors according to the mode selection signal for each output signal corresponding to each pixel block, and generates a video signal. The prism is a four-plate prism, The aforementioned image sensors are arranged in four locations corresponding to Rch, G1ch, G2ch, and Bch. The drive signal generation unit controls the pixel blocks of a plurality of image sensors to output pixels in a high-resolution shooting mode, so that the positions of the pixels output by each image sensor are shifted relative to each other.

2. In the imaging apparatus according to claim 1, The signal processing unit generates a video signal for high-speed shooting mode, which outputs a signal for each frame based on the output signal obtained by binning and reading out all the charges accumulated in the photoelectric conversion section of the four pixels of the pixel block for each frame. An imaging device that generates a video signal in a high-resolution shooting mode with a resolution corresponding to the pixels of a pixel block, based on an output signal obtained by sequentially reading out the charge for multiple frames accumulated in the photoelectric conversion unit of one pixel of the pixel block, one pixel at a time for each frame.

3. In the imaging apparatus according to claim 1, The imaging device includes a drive signal generation unit that controls the pixel blocks of the multiple image sensors to output signals for pixel positions where the corresponding pixel blocks of the multiple image sensors together form a Bayer array, for the pixel blocks of the high-resolution shooting mode.

4. In the imaging device according to claim 3, The imaging device, in the high-resolution shooting mode, selects one of the following methods: generating a video signal by demosaicing output signals read sequentially from one pixel at a time for each frame from a plurality of image sensors, or generating a video signal by collecting the output signals corresponding to each pixel in the pixel block of the image sensor as signals for each pixel position every four frames.

5. In the imaging apparatus according to claim 4, The imaging device is configured such that the signal processing unit demosaices at least the portion of the pixel block at the boundary with the pixel block in high-speed shooting mode, among the pixel blocks in which the high-resolution shooting mode is selected, for each frame to generate a video signal.

Images