Imaging device, electronic endoscope system, data processing method, and program

Abstract

To effectively reduce fixed pattern noise included in an image obtained from an imaging element relative to prior arts.SOLUTION: One aspect of the present invention relates to an imaging device comprising: an imaging element that is configured so as to share a readout line between adjacent pixel columns; a readout unit that sequentially reads out data of G pixels of the adjacent pixel columns from a common terminal, and sequentially reads out data of R pixels and B pixels of the adjacent pixel columns from a common terminal; an acquisition unit that, for a light blocking pixel region set in a plurality of pixel rows in the imaging element, acquires reference data corresponding to each terminal on the basis of data of pixels read out from the same terminal; and a correction unit that, for an effective pixel region excluding the light blocking pixel region in the imaging element, corrects the data of each pixel in the effective pixel region by subtracting the corresponding reference data from the data of the pixels read out from each terminal.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an imaging device configured to capture images of biological tissue, and an electronic endoscope system including an electronic endoscope having an imaging device at its tip. [Background technology]

[0002] In the field of medical equipment, electronic endoscope systems are known that can generate images suitable for diagnosing lesions hidden within a body cavity by illuminating biological tissue within the body cavity and capturing an image of the illuminated biological tissue within the body cavity as a subject. When imaging biological tissue, if the imaging element used for imaging has defective pixels that do not output image data normally during the manufacturing process, or if there is an abnormality in the transmission path within the imaging element, vertical line-shaped scratches may appear on the captured image. Such vertical line-shaped scratches are called fixed pattern noise (FPN). Since the occurrence of FPN is undesirable for image quality, various correction methods have been proposed to reduce FPN. For example, Patent Document 1 describes an imaging device that includes a memory that stores a gain that exceeds the gain obtained by a CMOS imaging element and at which line-shaped FPN begins to occur in the captured image as a correction reference value, and an image processing unit that performs line correction on the captured image to suppress FPN when the gain provided when exceeding the gain obtained by the CMOS imaging element exceeds the correction reference value. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2022 / 064609 Summary of the Invention [Problem to be solved by the invention]

[0004] However, when mounting a CMOS image sensor at the tip of an endoscope, it is necessary to use a small image sensor, and it is also difficult to mount an analog circuit board to sufficiently reduce FPN. In addition, when an image sensor is used in an endoscope, the brightness of the illumination light is limited, and when the gain set in the image processing at the subsequent stage is increased, the FPN level also increases, making it difficult to reduce FPN in image processing. Therefore, when mounting a CMOS image sensor in an endoscope, it may be difficult to obtain the original image quality of the image sensor.

[0005] SUMMARY OF THE PRESENT EMBODIMENTS An object of the present invention is to reduce fixed pattern noise contained in an image obtained from an imaging element more effectively than ever before. [Means for solving the problem]

[0006] The first aspect of the present disclosure is an image sensor in which a color filter of R (red), G (green) or B (blue) is arranged in a predetermined pattern for each of a plurality of pixels arranged in a matrix, and a readout line is shared between adjacent pixel columns; a readout unit that reads out data of each pixel from a plurality of terminals through readout lines of the imaging element, the readout unit sequentially reading out data of pixels (G pixels) corresponding to G color filters in adjacent pixel columns from a common terminal, and sequentially reading out data of pixels (R pixels) corresponding to R color filters and pixels (B pixels) corresponding to B color filters in adjacent pixel columns from a common terminal; an acquisition unit that acquires reference data corresponding to each terminal based on pixel data read from a same terminal, for light-shielded pixel regions set in a plurality of pixel rows in the imaging element; and a correction unit that corrects the data of each pixel in the effective pixel area excluding the light-shielded pixel area of ​​the imaging element by subtracting corresponding reference data from the pixel data read out from each terminal.

[0007] A second aspect of the present disclosure is an image sensor in which a color filter of R (red), G (green) or B (blue) is arranged in a predetermined pattern for each of a plurality of pixels arranged in a matrix, and a readout line is shared between adjacent pixel columns; a first read mode in which a read unit reads out data of each pixel from a plurality of terminals through a read line of the imaging element, the first read mode being for sequentially reading out data of pixels (G pixels) corresponding to a G color filter in adjacent pixel columns from a common terminal, and sequentially reading out data of pixels (R pixels) corresponding to an R color filter and pixels (B pixels) corresponding to a B color filter in adjacent pixel columns from a common terminal; or a first readout mode in which data of G pixels and R pixels in adjacent pixel columns is sequentially read out from a common terminal, and a second readout mode in which data of G pixels and B pixels in adjacent pixel columns is sequentially read out from a common terminal; an acquisition unit that acquires reference data corresponding to each terminal based on pixel data read from each terminal, for a light-shielded pixel region set in a plurality of pixel rows in the imaging element; a control unit that sets a readout mode when the readout unit reads out data of each pixel in the image sensor based on reference data acquired by the acquisition unit in each of the first readout mode and the second readout mode; and a correction unit that corrects data of each pixel in an effective pixel area excluding the light-shielded pixel area of ​​the imaging element by subtracting corresponding reference data from pixel data read out from each terminal in a read mode set by the control unit.

[0008] The readout unit may have an analog-to-digital conversion unit including a plurality of analog-to-digital converters that convert data read out from the plurality of terminals at every predetermined number of terminals along a pixel column of the image sensor into digital data, in which case each analog-to-digital converter is configured to sequentially process the data read out from the predetermined number of terminals.

[0009] A third aspect of the present disclosure is an electronic endoscope system including the imaging device for imaging biological tissue, and a light source device for emitting illumination light to the biological tissue. The light source device can select either white light or special light as the illumination light, and the control unit sets a read mode when the readout unit reads out data of each pixel in the imaging element at startup or when the illumination light is selected.

[0010] When the reference data acquired by either the first readout mode or the second readout mode is less than a predetermined level, the control unit may set the corresponding mode as the readout mode when the readout unit reads out data of each pixel in the imaging element.

[0011] When the reference data acquired by either the first readout mode or the second readout mode is equal to or higher than a predetermined level, the control unit may compare the reference data with the reference data acquired by the other mode, and set the readout mode corresponding to the smaller reference data as the readout mode when the readout unit reads out data of each pixel in the image sensor.

[0012] A fourth aspect of the present disclosure is a data processing method for capturing an image of a subject using an imaging element in which a color filter of either R (red), G (green), or B (blue) is arranged in a predetermined pattern for each pixel of a plurality of pixels arranged in a matrix, and a readout line is shared between adjacent pixel columns. This data processing method is a step of reading out data of each pixel from a plurality of terminals through readout lines of the imaging element for light-shielded pixel regions set in a plurality of pixel rows in the imaging element using a first readout mode in which data of pixels (G pixels) corresponding to G color filters in adjacent pixel columns are read out sequentially from a common terminal, and data of pixels (R pixels) corresponding to R color filters and pixels (B pixels) corresponding to B color filters in adjacent pixel columns are read out sequentially from a common terminal, and acquiring reference data corresponding to each terminal based on the pixel data read out from each terminal; a step of reading out data of each pixel in the light-shielded pixel region from a plurality of terminals through readout lines of the imaging element in a second readout mode in which data of G pixels and R pixels in adjacent pixel columns are sequentially read out from a common terminal, and data of G pixels and B pixels in adjacent pixel columns are sequentially read out from a common terminal, and acquiring reference data corresponding to each terminal based on the pixel data read out from each terminal; setting a readout mode based on reference data acquired in each of the first readout mode and the second readout mode; and correcting data for each pixel in an effective pixel area of ​​the imaging element excluding the light-shielded pixel area by subtracting corresponding reference data from pixel data read out from each terminal in the set readout mode.

[0013] A fifth aspect of the present disclosure relates to a method for capturing an image of a subject using an image sensor configured such that a color filter of R (red), G (green), or B (blue) is arranged in a predetermined pattern for each of a plurality of pixels arranged in a matrix, and a readout line is shared between adjacent pixel columns, On the computer, a step of reading out data of each pixel in the imaging element from a plurality of terminals through readout lines of the imaging element for light-shielded pixel regions set in a plurality of pixel rows, in a first readout mode in which data of pixels (G pixels) corresponding to G color filters in adjacent pixel columns are read out sequentially from a common terminal, and data of pixels (R pixels) corresponding to R color filters and pixels (B pixels) corresponding to B color filters in adjacent pixel columns are read out sequentially from a common terminal, and acquiring reference data corresponding to each terminal based on the pixel data read out from each terminal; a step of reading out data of each pixel in the image sensor from a plurality of terminals through readout lines of the image sensor for the light-shielded pixel region in a second readout mode in which data of G pixels and R pixels in adjacent pixel columns is read out sequentially from a common terminal, and data of G pixels and B pixels in adjacent pixel columns is read out sequentially from a common terminal, and acquiring reference data corresponding to each terminal based on the pixel data read out from each terminal; setting a readout mode based on reference data acquired in each of the first readout mode and the second readout mode; a step of correcting data of each pixel in an effective pixel area excluding the light-shielded pixel area in the imaging element by subtracting corresponding reference data from pixel data read out from each terminal in the set readout mode; This is a program for executing the above. Effect of the Invention

[0014] According to the imaging device of the present disclosure, fixed pattern noise contained in an image obtained from an imaging element can be reduced more effectively than ever before. [Brief description of the drawings]

[0015] [Figure 1] 1 is a block diagram showing an example of a configuration of an electronic endoscope system according to an embodiment; [Diagram 2] FIG. 1 is a diagram showing a pixel array of a CMOS image sensor. [Diagram 3] 2 is a block diagram showing an example of the configuration of a CMOS image sensor and a control unit in the electronic endoscope system of one embodiment. FIG. [Figure 4] 4 is a diagram for specifically explaining a pixel array, a switch section, and an ADC section in the CMOS image sensor of FIG. 3. [Diagram 5] 4 is an example of a timing chart during normal readout of pixel data in the CMOS image sensor of FIG. 3. [Figure 6] 4 is an example of a timing chart when shifting and reading pixel data in the CMOS image sensor of FIG. 3. [Figure 7] 4 is a diagram for explaining an example of reading pixel data in the CMOS image sensor of FIG. 3. [Figure 8] 4 is a diagram for explaining an example of reading pixel data in the CMOS image sensor of FIG. 3. [Figure 9] 5 is a flowchart showing a process of setting a readout mode in the electronic endoscope system of the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An electronic endoscope system, which is an embodiment of an imaging system according to the present invention, will be described in detail below with reference to the drawings. 1 is a block diagram showing an example of the configuration of an electronic endoscope system 1 according to an embodiment. The electronic endoscope system 1 is a system specialized for medical use, and is used for observing and diagnosing biological tissue within a body cavity. 1, an electronic endoscope system 1 according to one embodiment includes an electronic scope (endoscope) 10, a processor 20 incorporating a light source device, and a monitor 30. The electronic scope 10 and the processor 20 are connected by a connector, and the processor 20 is connected to the monitor 30. A portion of the electronic endoscope system 1 excluding the light source device 28 is an example of an imaging device.

[0017] The processor 20 includes a system controller 21, an image input processing unit 22, an image buffer 23, an image output processing unit 25, an operation panel 26, and a light source device . The system controller 21 executes various programs and comprehensively controls the entire electronic endoscope system 1. The system controller 21 executes each operation of the electronic endoscope system 1 and changes parameters for each operation in response to instructions from an operator (observer) inputted to an operation panel 26. The system controller 21 supplies each circuit in the electronic endoscope system 1 with a clock pulse for adjusting the timing of the operation of each part.

[0018] The light source device 28 generates emitted light EL (illumination light) for illuminating the subject. The emitted light EL may be white light, pseudo-white light, or special light in a specific wavelength band. In one embodiment, the light source device 28 can switch the emitted light EL between white light and special light (select illumination light) in response to an instruction from the system controller 21. The light EL emitted by the light source device 28 is focused by a focusing lens on an incident end surface of an LCB (Light Carrying Bundle) 11 and is incident on the LCB 11. The type of light source of the light source device 28 is not limited, but may be, for example, an LED, a laser diode, or a high-intensity lamp (for example, a xenon lamp, a metal halide lamp, a mercury lamp, or a halogen lamp). 1, the light source device 28 is built into the processor 20, but this is not the only possibility. The light source device 28 may be built into the electronic scope 10, or may be provided separately from the electronic scope 10 and the processor 20 and connected to each other by wires.

[0019] The emitted light EL incident on the LCB 11 propagates through the LCB 11. The emitted light EL propagated through the LCB 11 is emitted from the exit end face of the LCB 11 disposed at the tip of the electronic scope 10, and is irradiated onto the subject via the light distribution lens 12. Return light from the subject illuminated by the emitted light EL from the light distribution lens 12 forms an optical image on the light receiving surface of the CMOS image sensor 14 via the objective lens 13.

[0020] The CMOS image sensor 14 is an imaging element configured to capture an image of a subject using a rolling shutter method. The CMOS image sensor 14 has a pixel array with a Bayer pixel arrangement in which a color filter of R (red), G (green), or B (blue) is arranged in a predetermined pattern for each of a plurality of pixels arranged in a matrix on the light receiving surface. In the following description, the pixels corresponding to the R, G, and B color filters are referred to as "R pixels," "G pixels," and "B pixels," respectively. The CMOS image sensor 14 accumulates an optical image formed by each pixel of the pixel array as an electric charge according to the amount of light, and generates and outputs data for R pixels, G pixels, and B pixels. Note that, instead of the CMOS image sensor 14, a CCD (Charge-Coupled Device) image sensor or other types of imaging devices may be used.

[0021] FIG. 2 shows the configuration of a pixel array of the CMOS image sensor 14. 2, the pixel array of the CMOS image sensor 14 has a plurality of pixels arranged in a matrix of pixel rows and pixel columns on the light receiving surface. The pixel array includes an effective pixel area and an OB area (an example of a light-shielded pixel area). The effective pixel area is an area of ​​the pixel array that is not shielded from light. Image data obtained based on data of pixels in the effective pixel area is sent to the image input processing unit 22. The OB (optical black) region is a strip-shaped light-shielding region along the horizontal direction (i.e., column direction) in the pixel array, and is set in the upper end region and the lower end region of the pixel array and includes a predetermined number of pixel rows. The number of pixel rows corresponding to the OB region is not limited, but for example, several tens of rows are set in each of the upper end region and the lower end region. In the example shown in FIG. 2, the OB region is set in both the upper end region and the lower end region of the pixel array, but this is not limited, and the OB region may be set in either the upper end region or the lower end region. Note that the OB region may be set not only in the horizontal direction in the pixel array but also in the left end region and the right end region of the pixel array along the vertical direction. As will be described later, pixel data obtained in the OB region is used when detecting FPN and correcting the data of each pixel in the effective pixel region.

[0022] 1 again, the control unit 15 provided in the electronic scope 10 drives and controls the CMOS image sensor 14 at a timing synchronized with the frame rate of the video processed by the processor 20, in accordance with a clock pulse supplied from the system controller 21. An image signal capturing an image of a subject is input from the CMOS image sensor 14 to the control unit 15 at a predetermined frame period. The frame period (one frame period) is, for example, 1 / 120 seconds, 1 / 60 seconds, or 1 / 30 seconds. The control unit 15 also detects fixed pattern noise (hereinafter referred to as "FPN") from the image data input from the CMOS image sensor 14, and corrects the image data to reduce the FPN. The control unit 15 outputs the corrected image data (captured image) to the image input processing unit 22 of the processor 20. The control unit 15 will be described in further detail later.

[0023] In the processor 20 , an image input processing unit 22 , an image buffer 23 , and an image output processing unit 25 are provided to generate an image to be displayed on a monitor 30 . The image input processing unit 22 performs predetermined processing such as demosaic processing and matrix calculation on image data (captured image) in frame units sent from the control unit 15, and sends the processed image data to an image buffer 23 and an image output processing unit 25. The image buffer 23 is a memory (frame buffer) that buffers the captured image sent from the image input processing unit 22 on a frame-by-frame basis. The image buffer 23 temporarily stores captured images of past frames, such as frames one frame before the current frame and frames two frames before the current frame, for processing by the image output processing unit 25.

[0024] The image output processing unit 25 performs image processing such as filtering and edge enhancement on the image data of the current frame output from the image input processing unit 22 and the image data of the past frame acquired from the image buffer 23 to generate screen data for monitor display, and converts the generated screen data for monitor display into a predetermined video format signal. The converted video format signal is output to the monitor 30. As a result, the image of the subject is displayed on the display screen of the monitor 30.

[0025] Next, the CMOS image sensor 14 and the control unit 15 will be described in more detail with reference to FIGS. FIG. 3 is a block diagram showing an example of the configuration of the CMOS image sensor 14 and the control unit 15 in the electronic endoscope system 1. As shown in FIG. As shown in Fig. 3, the CMOS image sensor 14 includes a pixel array 141, a vertical selection circuit 142, a column readout circuit 143, and an ADC (Analogue to Digital Converter) unit 144. Fig. 4 is a diagram for more specifically explaining the pixel array 141, the column readout circuit 143, and the ADC unit 144 in the CMOS image sensor 14 of Fig. 3.

[0026] 3, the pixel array 141 has a plurality of pixels arranged in a matrix as shown in FIG. 2, and includes an effective pixel region and an OB region. 4 shows a part of a plurality of pixels arranged in a matrix in the pixel array 141. In the pixel array 141, a color filter of R (red), G (green), or B (blue) is arranged for each of the plurality of pixels arranged in a matrix by the Bayer type pixel arrangement method. That is, the G pixels indicated by diagonal lines are arranged in a checkerboard pattern. In the pixel array 141 of FIG. i,j (i=0, 1, ..., j=0, 1, ...) indicates pixels (B pixels) on which a B (blue) color filter is arranged. i,j (i=0, 1, ..., j=0, 1, ...) indicates pixels (G pixels) on which a G (green) color filter is arranged.i,j (i=0, 1, . . . , j=0, 1, . . . ) indicate pixels (R pixels) on which R (red) color filters are arranged.

[0027] A plurality of readout lines L0, L1, ..., L9, ... are provided corresponding to each pixel column of the pixel array 141. In the pixel array 141, the readout lines are configured to be shared between adjacent pixel columns. In one embodiment, as shown in FIG. 4, each readout line is connected so as to be shared by pixels arranged diagonally. For example, the diagonally arranged pixels G 0,1 ,Pixel G 1,0 ,Pixel G 2,1 , and pixel G 3,0 share one read line L1.

[0028] The vertical selection circuit 142 is a circuit that sequentially selects pixel rows in the pixel array 141. The column readout circuit 143 is a circuit that sequentially selects pixel columns in the pixel array 141. For example, the vertical selection circuit 142 scans pixel rows in sequence, and pixel data output from the pixel row to be scanned is input to the column readout circuit 143 through the corresponding readout line. In the example shown in FIG. 4, pixel B 0,0 ,G 0,1 ,B 0,2 , ... are selected, and the data of each selected pixel is output to the corresponding readout line. Then, the pixel G 1,0 ,R 1,1 ,G 1,2 , . . . are selected, and the data of each selected pixel is output to a corresponding readout line, and so on. The pixel may have a known configuration, for example, including a photodiode that performs photoelectric conversion, a floating diffusion (FD), an amplifying transistor, and a plurality of switches including a transfer switch that transfers the charge generated in the photodiode to the FD.

[0029] The column readout circuit 143 and the ADC unit 144 are an example of a readout unit that reads out data of each pixel from ADC input terminals T1 to T4 through readout lines of the CMOS image sensor 14, and sequentially reads out data of G pixels in adjacent pixel columns from a common terminal (a common ADC input terminal), and sequentially reads out data of R pixels and B pixels in adjacent pixel columns from a common terminal (a common ADC input terminal).

[0030] The column readout circuit 143 includes a plurality of switches that operate in accordance with either a normal readout or a shift readout mode, which will be described later. The normal readout is a mode in which each pixel of the pixel array 141 is read out according to the arrangement of the pixel columns, and is an example of a second readout mode. The shift readout is a mode in which each pixel of the pixel array 141 is read out by shifting it in the pixel column direction according to the pixel row, and is an example of a first readout mode. In the example shown in Fig. 4, the column readout circuit 143 has a plurality of switch pairs, with the switches Sa and Sb forming one switch pair. In each switch pair, one end of the switch Sa and the switch Sb are connected to the adjacent readout line, and the other end is a common output end (input terminal of the ADC). For example, when focusing on the adjacent readout lines L0 and L1, one end of the switch Sa in the switch pair is connected to the readout line L0, one end of the switch Sb is connected to the readout line L1, and the other ends of the switches Sa and Sb are a common output end (input terminal T1).

[0031] The ADC unit 144 converts data read from every predetermined number of terminals (four input terminals T1 to T4 in the example of FIG. 4) along the pixel columns of the pixel array 141 into digital data. Specifically, the ADC unit 144 has a plurality of ADCs 144_1, 144_2, ..., each of which has four input terminals T1 to T4. Each ADC is configured to sequentially process data read from the four input terminals T1 to T4. The input terminals T1 to T4 of each ADC are connected to the output terminals of four switch pairs that are consecutive in the column direction. That is, one ADC is shared by four adjacent pixel columns. The number of pixel columns that share one ADC is not limited to four, but the readout speed can be increased by sharing one ADC among a plurality of pixel columns. In the example shown in FIG. 4, the ADC unit 144 sequentially converts pixel data input to the input terminal T1 of the ADCs 144_1, 144_2, ... into digital data with a time difference and outputs the digital data, and then captures pixel data input to the input terminal T2 of the ADCs 144_1, 144_2, ..., converts the pixel data into digital data with a time difference, and outputs the digital data.

[0032] Next, the read operation in shift read and normal read will be described with reference to FIG. 5 and FIG. Fig. 5 is an example of a timing chart for normal readout of pixel data in the CMOS image sensor 14 of Fig. 3. Fig. 6 is an example of a timing chart for shift readout of pixel data in the CMOS image sensor 14 of Fig. 3. In each of Figs. 5 and 6, (a) indicates a reset pulse P RESET , (b) is the read pulse P READ , (c) to (f) respectively indicate the sampling (analog input) Samp1 to Samp4 input to the input terminals T1 to T4 of each ADC of the ADC section 144, (g) indicates the ADC analog lamp ADC_Lamp, (h) indicates the state (ON or OFF) of the switch Sa, and (i) indicates the state (ON or OFF) of the switch Sb.

[0033] 5 and 6 show data readout of the pixels in the 0th and 1st rows, and a reset pulse P RESET is sent out, and the FD of each pixel is reset (the FD becomes the reset level potential). 5 and 6, the control of the switches Sa and Sb differs depending on the selection result of either shift readout or normal readout. In the normal readout shown in Fig. 5, when row 0 is selected, switch Sa is ON and switch Sb is OFF, and when row 1 is selected, switch Sa is OFF and switch Sb is ON. Although not shown in Fig. 5 for the second and subsequent rows, the state of each switch is changed every time a new row is selected. On the other hand, in the shift readout shown in Fig. 6, when the 0th row or the 1st row is selected, the switch Sa is turned ON and the switch Sb is turned OFF. Although not shown in Fig. 6, for the 2nd row and onwards, even when a new row is selected, the switch Sa is always turned ON and the switch Sb is always turned OFF.

[0034] When a row is selected, the input terminals T1 to T4 of each ADC in the ADC section 144 are each connected to one of the readout lines depending on the state of the switch. When the 0th row is selected, data (electric potential) corresponding to the reset level of the FD of each pixel in the 0th row is output to the corresponding readout line and sampled from the input terminal of the ADC connected to the readout line (period Ts1). Next, a readout pulse P READ In synchronization with this, the charge accumulated in the photodiode of each pixel is transferred to the FD and output to the readout line. The data (electric potential) output to the readout line is sampled from the input terminals T1 to T4 of each ADC with a time difference (period Ts2). Each ADC subtracts the reset level data input during period Ts1 from the data input during period Ts2 (CDS (Correlated Double Sampling) processing) and converts it into a digital value. Since processing is performed for each input terminal with a time difference during period Ts2, the processing load on each ADC is reduced. Next, when the first row is selected, the states of the switches Sa and Sb change, but the operation of each section is the same as when the zeroth row is selected.

[0035] Next, normal readout and shift readout in response to control over the column readout circuit 143 will be described with reference to FIGS. 7 and 8 show switch control when the pixels of the pixel array 141 shown in FIG. 4 are selected and scanned in the order of row 0 to row 3, and data arrangement of the pixels after being read out when each row is selected, in association with each other. Unlike the actual arrangement of pixels in the pixel array 141, the same column indicates pixels corresponding to data input from the same ADC input terminal in FIGS. 7 and 8. In other words, the "columns" in FIGS. 7 and 8 refer to columns of the input terminals of each ADC of the ADC unit 144, unlike columns in the pixel arrangement of the pixel array 141. Although only the 0th to 3rd lines are shown in FIG. 7 and FIG. 8, the same applies to the 4th and subsequent lines.

[0036] Referring to FIG. 7, normal read R N1 In the normal readout R, the state of each switch pair is switched every time a pixel row is selected. When the 0th row is selected, the switch Sa is turned ON and the switch Sb is turned OFF, and when the 1st row is selected, the switch Sa is turned OFF and the switch Sb is turned ON. N1 The switch control in corresponds to the flowchart in Figure 5. N1 In this case, the G pixels after readout are arranged in a checkerboard pattern.

[0037] Shift read R S1 In this case, regardless of the pixel row selection, the switch Sa is always ON and the switch Sb is always OFF. S1 The switch control in corresponds to the flowchart in Figure 6. S1 In this example, data from pixels connected to odd-numbered rows (first row, third row, ...) in pixel array 141 is shifted one column to the right and read out. As a result, data from G pixels diagonally arranged with respect to a common readout line is read out from a common terminal, and data from B and R pixels diagonally arranged with respect to the common readout line in pixel array 141 is read out from a common terminal.

[0038] FIG. 8 shows a case where normal readout and shift readout are performed with switch control different from that in FIG. Referring to FIG. 8, normal read R N2 So, normal read R N1 Similarly, the state of each switch pair is switched each time a pixel row is selected, but the order of ON and OFF for the switch pairs is the same as that of the normal readout R N1 When the 0th row is selected, the switch Sa is turned OFF and the switch Sb is turned ON, and when the 1st row is selected, the switch Sa is turned ON and the switch Sb is turned OFF. N2 So, normal read R N1 Similarly, the G pixels after readout are arranged in a checkerboard pattern. Shift read R S2 In this case, regardless of the pixel row selection, switch Sa is always OFF and switch Sb is always ON. At this time, data of pixels connected to even-numbered rows (row 0, row 2, ...) is shifted one column to the left and read out. As a result, data of G pixels diagonally arranged with respect to the common readout line is read out from a common terminal, and data of B and R pixels diagonally arranged with respect to the common readout line is read out from a common terminal.

[0039] Returning to the explanation of Figure 3. 3, the control unit 15 includes a CPU 151, an FPN detection unit 152, a correction unit 153, and an image processing unit 154. The CPU 151 executes a predetermined program to control the overall operation of the control unit 15 under the command of the system controller 21, and also controls the vertical selection circuit 142 and the column readout circuit 143 of the CMOS image sensor 14. Note that, in one embodiment, the processing of the control unit 15 may be performed by a device such as a DSP (digital signal processor), an ASIC, or an FPGA (field programmable gate array) instead of the CPU.

[0040] The FPN detection unit 152 detects FPN (fixed pattern noise; an example of reference data) of the CMOS image sensor 14 based on pixel data sequentially output by the ADC unit 144. Specifically, the FPN detection unit 152 targets the OB region (see FIG. 2) set in multiple pixel rows of the pixel array 141, and acquires reference data corresponding to each terminal based on pixel data read from the same input terminal of each ADC. As described above, the OB region is a strip-shaped (predetermined number of rows) light-shielded region along the horizontal direction (i.e., the column direction), so data read from pixels in the OB region is data in which FPN is superimposed on the black level. The FPN detection unit 152 detects FPN by averaging the data of pixels in this predetermined number of rows. Since FPN is a vertical line-shaped scratch that appears on a captured image, FPN is detected for each pixel column. Correction unit 153 corrects the data of each pixel in the effective pixel area excluding the OB area of ​​pixel array 141 by subtracting the corresponding reference data (FPN detected by FPN detection unit 152) from the pixel data read from each input terminal of each ADC. As described above, since FPN is detected for each pixel column, correction unit 153 corrects the data of each pixel by subtracting the FPN detected corresponding to a specific pixel column from the data of each pixel in that pixel column.

[0041] The detection of FPN by the FPN detection unit 152 and the correction by the correction unit 153 are performed on pixel data read out by the CMOS image sensor 14. S1 , or R in Fig. 8 S2 ), the data of the G pixels is read out from a common terminal. That is, as shown in FIG. 4, the data of the diagonally arranged G pixels (for example, G 0,1 ,G 1,0 ,G 2,1 ,G 3,0) data is read out through a common read-out line (e.g., L1) and input to a common input terminal, so that FPN resulting from the same path is superimposed on the read-out data. Therefore, the number of samples required to detect (calculate) FPN is doubled compared to normal read-out, making it possible to detect FPN with high accuracy for G pixels, and accordingly improving the accuracy of correction of G pixel data. Generally, the contribution of G color to the luminance of a captured image obtained by white or pseudo-white emitted light EL is high, so that a good captured image can be obtained by correction by the correction unit 153.

[0042] In the example shown in FIG. 4, the ADC unit 144 converts pixel data (electric potential) input from each input terminal in each ADC into digital data with a time difference and sweeps it out. For example, in the ADC 144_1, pixel data is processed in the order of input terminals T1, T2, T3, and T4 with a time difference, so that an offset according to the time difference is superimposed on the data input at each input terminal. Therefore, in the past, even when pixel data of the same level was input, FPN tended to increase in the order of input terminals T1, T2, T3, and T4. This difference in offset due to the input terminals can be reduced by taking in G pixel data at a common input terminal as described above.

[0043] The image processing unit 154 generates image data (captured image) including pixel data according to the original pixel arrangement (Bayer type pixel arrangement) based on the pixel data corrected by the correction unit 153, and sends the image data (captured image) in units of frames to the image input processing unit 22. For example, when shift readout is selected, as shown in Figures 7 and 8, the data of the G pixels is read out from the common input terminal, but is returned to the data of the Bayer type pixel arrangement (i.e., data according to the arrangement of the pixel array 141) by the image processing unit 154.

[0044] As described above, the electronic endoscope system 1 of one embodiment executes shift readout, in which data of G pixels is sequentially read out from a common terminal from the pixel array of the Bayer type pixel arrangement of the CMOS image sensor 14, and data of R pixels and B pixels in adjacent pixel rows is sequentially read out from a common terminal. Furthermore, the electronic endoscope system 1 acquires FPNs corresponding to each terminal based on pixel data read out from the same terminal for the OB region of the CMOS image sensor 14, and corrects data of each pixel in the effective pixel region by subtracting the corresponding FPN from pixel data read out from each input terminal of the ADC for the effective pixel region. In the shift readout, data of G pixels is taken into a common input terminal of the ADC, so that FPN can be detected with high accuracy for G pixels, and the accuracy of correction for the data of G pixels is accordingly improved. For example, in the case of white or pseudo-white emitted light in which the contribution of G color is high in luminance, a good captured image (i.e., an image in which fixed pattern noise is effectively reduced) can be obtained by correction.

[0045] In addition, when the illumination light is, for example, a special light with high sensitivity to the color B, or when the illumination light is white but depending on its hue, it may not be appropriate to aggregate and read out the data of G pixels as data of the same column. In other words, in one embodiment, normal readout may be selected instead of shift readout depending on the illumination light irradiating the subject.

[0046] There are individual differences in the occurrence of FPN in the CMOS image sensor 14, and the offset occurring at each input terminal of each ADC is not limited to the case where it increases in the order of input terminals T1, T2, T3, and T4, but there are also cases where the offset occurs largely only at specific two or three terminals. Therefore, in one embodiment, it is also preferable to set the readout mode by comparing the FPN obtained in both readout modes of shift readout and normal readout. An example of this readout mode setting process will be described below with reference to the flowchart of FIG. 9.

[0047] 9 is a process that is mainly executed by the control unit 15 and the system controller 21 of the electronic endoscope system 1 in cooperation with each other, and shows a process that is executed immediately after the system is started up. Note that the readout mode setting process may be performed when the illumination light is switched (illumination light is selected) by the surgeon during system operation.

[0048] First, when the initial setting is completed (step S2), the system controller 21 sets the illumination light based on the user setting of the illumination light input from the operation panel 26 (step S4). If the illumination light set here is white light (step S6), the system controller 21 instructs the control unit 15 to acquire the FPN of the OB region by normal readout (step S8). If the illumination light set in step S4 is special light (step S6), the system controller 21 instructs the control unit 15 to acquire the FPN of the OB region by shift readout (step S10). If the FPN acquired in step S8 or S10 is smaller than the predetermined threshold Th (step S12: YES), the system controller 21 sets the read mode to the one in which the FPN was last acquired (i.e., the FPN was acquired in step S8 or S10) (step S14). That is, if the FPN acquired in either the shift read mode or the normal read mode is smaller than the threshold Th (an example of a predetermined level), the system controller 21 sets the read mode to the mode in which the FPN is smaller than the predetermined level. The comparison of the FPN obtained in step S8 or S10 with the predetermined threshold value Th may be performed based on the average value of the FPN obtained for each pixel column, or based on the maximum value of the FPN obtained for each pixel column.

[0049] On the other hand, if the FPN acquired in step S8 or S10 is equal to or greater than the predetermined threshold Th (step S12: NO), the system controller 21 acquires the FPN in a readout mode opposite to that of steps S6 to S10. That is, if the illumination light set in step S4 is white light (step S16), the system controller 21 instructs the control unit 15 to acquire the FPN of the OB region by shift readout (step S18). If the illumination light set in step S4 is special light (step S16), the system controller 21 instructs the control unit 15 to acquire the FPN of the OB region by normal readout (step S20).

[0050] If the FPN acquired in step S18 or S20 is smaller than the FPN acquired in step S8 or S10 (i.e., the FPN has improved) (step S22: YES), the system controller 21 sets the read mode to the one in which the FPN was last acquired (i.e., the FPN was acquired in step S18 or S20) (step S14). That is, if the FPN acquired in step S12 by either the shift readout or normal readout mode is equal to or greater than the threshold Th, the system controller 21 compares the FPN with the FPN acquired in the other mode, and sets the read mode to the one corresponding to the smaller FPN. This makes it possible to set the mode to either the shift readout or normal readout mode, whichever provides the better (smaller) FPN. If the FPN has not improved (step S22: NO), the system controller 21 sets the normal readout (step S24).

[0051] According to the readout mode setting process shown in Fig. 9, a readout mode in which the FPN is at a low level is set based on the FPN acquired in each of the shift readout and normal readout. Therefore, an optimal readout mode can be set according to the type of illumination light and individual differences of the image sensor. Note that, unlike Fig. 9, the FPN may be acquired by the shift readout and normal readout, respectively, in steps S8 and S10, and the FPN may be acquired by the normal readout and shift readout, respectively, in steps S18 and S20. In either case, the mode in which the FPN is better, either the shift readout or the normal readout, can be set. When the read mode is set by the read mode setting process, the system controller 21 instructs the CPU 151 to operate the control unit 15 in the set read mode. In response to this instruction, the control unit 15 corrects the data of each pixel in the effective pixel area by subtracting the FPN (reference data) already acquired by the read mode setting process from the pixel data read from each input terminal of each ADC for the effective pixel area excluding the OB area of ​​the pixel array 141.

[0052] 4 illustrates a pixel array in which the readout lines are connected so as to be shared by pixels arranged diagonally opposite each other, but this is not limited thereto. The readout lines may also be configured so that adjacent pixels in the horizontal or vertical direction share the same readout lines. 4, the pixels are arranged in a Bayer pixel array, but the color filter arrangement pattern is not limited to this and can be any pattern. Regardless of the color filter arrangement pattern, the column readout circuit can be modified to sequentially read out only G pixels from a common terminal, for example.

[0053] In one embodiment, a data processing method is disclosed, comprising the following steps (i) to (iv). (i) When data of each pixel is read from a plurality of input terminals of the ADC through the read lines of the CMOS image sensor 14, the data of each pixel of the pixel array 141 is read by shift reading, and an FPN (an example of reference data) corresponding to each input terminal is obtained based on the pixel data read from each input terminal, for the OB region of the pixel array 141. (ii) When reading out data of each pixel from a plurality of input terminals of the ADC through the readout lines of the CMOS image sensor 14 for the OB region of the pixel array 141, the data of each pixel of the pixel array 141 is read out by normal readout, and an FPN (an example of reference data) corresponding to each input terminal is obtained based on the pixel data read out from each input terminal. (iii) setting a read mode based on the FPN obtained in each of the shift read and the normal read; and (iv) For the effective pixel area of ​​the pixel array 141, the data of each pixel in the effective pixel area is corrected by subtracting the corresponding FPN from the pixel data read from each input terminal in the set read mode.

[0054] In one embodiment, a program for causing a computer to execute the above steps (i) to (iv) is disclosed.

[0055] The imaging device, electronic endoscope system, data processing method, and program of the present invention have been described in detail above. However, the present invention is not limited to the above-described embodiments, and various improvements and modifications may be made without departing from the spirit and scope of the present invention. [Explanation of symbols]

[0056] 1. Electronic endoscope system 10...Electronic scope 11...LCB 12...Light distribution lens 13...Objective lens 14…CMOS image sensor 141…Pixel array 142...Drive unit 143…Switch section 144…ADC section 144_1~141_3…ADC 15...Control section 151…CPU 152...FPN detector 153...Correction section 154...Image processing unit 20…Processor 21…System controller 22...Image input processing unit 23...Image buffer 25...Image output processing section 26...Operation panel 28…Light source device 30…Monitor L…Outgoing light L 0 ~L 9 …Readout line Sa, Sb…Switch T1~T4…Terminals

Claims

1. an imaging element in which a color filter of any one of R (red), G (green) or B (blue) is arranged in a predetermined pattern for each of a plurality of pixels arranged in a matrix, and a readout line is shared between adjacent pixel columns; a readout unit that reads out data of each pixel from a plurality of terminals through readout lines of the imaging element, the readout unit sequentially reading out data of pixels (G pixels) corresponding to G color filters in adjacent pixel columns from a common terminal, and sequentially reading out data of pixels (R pixels) corresponding to R color filters and pixels (B pixels) corresponding to B color filters in adjacent pixel columns from a common terminal; an acquisition unit that acquires reference data corresponding to each terminal based on pixel data read from a same terminal, for light-shielded pixel regions set in a plurality of pixel rows in the imaging element; a correction unit that corrects data of each pixel in an effective pixel area excluding the light-shielded pixel area in the image sensor by subtracting corresponding reference data from pixel data read out from each terminal; An imaging device comprising:

2. an imaging element in which a color filter of any one of R (red), G (green) or B (blue) is arranged in a predetermined pattern for each of a plurality of pixels arranged in a matrix, and a readout line is shared between adjacent pixel columns; a first read mode in which a read unit reads out data of each pixel from a plurality of terminals through a read line of the imaging element, the first read mode being for sequentially reading out data of pixels (G pixels) corresponding to G color filters in adjacent pixel columns from a common terminal, and sequentially reading out data of pixels (R pixels) corresponding to R color filters and pixels (B pixels) corresponding to B color filters in adjacent pixel columns from a common terminal; or a first readout mode in which data of G pixels and R pixels in adjacent pixel columns is sequentially read out from a common terminal, and a second readout mode in which data of G pixels and B pixels in adjacent pixel columns is sequentially read out from a common terminal; and an acquisition unit that acquires reference data corresponding to each terminal based on pixel data read from each terminal, for a light-shielded pixel region set in a plurality of pixel rows in the imaging element; a control unit that sets a readout mode when the readout unit reads out data of each pixel in the image sensor based on reference data acquired by the acquisition unit in each of the first readout mode and the second readout mode; a correction unit that corrects data of each pixel in an effective pixel area excluding the light-shielded pixel area of ​​the image sensor by subtracting corresponding reference data from pixel data read from each terminal in a read mode set by the control unit; An imaging device comprising:

3. the readout unit has an analog-to-digital conversion unit including a plurality of analog-to-digital converters that convert data read out from the plurality of terminals at every predetermined number of terminals along a pixel row of the image sensor into digital data; Each analog-to-digital converter is configured to sequentially process data read from the predetermined number of terminals.

3. The imaging device according to claim 2.

4. 3. An imaging device for imaging biological tissue according to claim 2, A light source device that emits illumination light for biological tissue, the light source device is capable of selecting either white light or special light as the illumination light, The control unit sets a read mode when the readout unit reads out data of each pixel in the image sensor at the time of startup or at the timing when the illumination light is selected. Electronic endoscope system.

5. When the reference data acquired in either the first readout mode or the second readout mode is less than a predetermined level, the control unit sets the first readout mode or the second readout mode as a readout mode when the readout unit reads out data of each pixel in the image sensor.

5. An electronic endoscope system according to claim 4.

6. When the reference data acquired in either the first readout mode or the second readout mode is equal to or higher than a predetermined level, the control unit compares the reference data with the reference data acquired in the other mode, and sets the readout mode corresponding to the smaller reference data as a readout mode when the readout unit reads out data of each pixel in the image sensor.

6. An electronic endoscope system according to claim 4 or 5.

7. A data processing method for capturing an image of a subject using an image sensor configured such that a R (red), G (green) or B (blue) color filter is arranged in a predetermined pattern for each of a plurality of pixels arranged in a matrix, and a readout line is shared between adjacent pixel columns, comprising: a step of reading out data of each pixel from a plurality of terminals through readout lines of the imaging element for light-shielded pixel regions set in a plurality of pixel rows in the imaging element using a first readout mode in which data of pixels (G pixels) corresponding to G color filters in adjacent pixel columns are read out sequentially from a common terminal, and data of pixels (R pixels) corresponding to R color filters and pixels (B pixels) corresponding to B color filters in adjacent pixel columns are read out sequentially from a common terminal, and acquiring reference data corresponding to each terminal based on the pixel data read out from each terminal; a step of reading out data of each pixel in the light-shielded pixel region from a plurality of terminals through readout lines of the imaging element in a second readout mode in which data of G pixels and R pixels in adjacent pixel columns is read out sequentially from a common terminal, and data of G pixels and B pixels in adjacent pixel columns is read out sequentially from a common terminal, and acquiring reference data corresponding to each terminal based on the pixel data read out from each terminal; setting a readout mode based on reference data acquired in each of the first readout mode and the second readout mode; correcting data of each pixel in an effective pixel area of ​​the image sensor excluding the light-shielded pixel area by subtracting corresponding reference data from pixel data read out from each terminal in the set readout mode; Data processing methods, including:

8. When capturing an image of a subject using an image sensor in which a color filter of R (red), G (green), or B (blue) is arranged in a predetermined pattern for each of a plurality of pixels arranged in a matrix, and a readout line is shared between adjacent pixel columns, On the computer, a step of reading out data of each pixel in the imaging element from a plurality of terminals via readout lines of the imaging element for light-shielded pixel regions set in a plurality of pixel rows, in a first readout mode in which data of pixels corresponding to G color filters (G pixels) in adjacent pixel columns are read out sequentially from a common terminal, and data of pixels corresponding to R color filters (R pixels) and pixels corresponding to B color filters (B pixels) in adjacent pixel columns are read out sequentially from a common terminal, and acquiring reference data corresponding to each terminal based on the pixel data read out from each terminal; a step of reading out data of each pixel in the image sensor from a plurality of terminals through readout lines of the image sensor from the light-shielded pixel region in a second readout mode in which data of G pixels and R pixels in adjacent pixel columns is read out sequentially from a common terminal, and data of G pixels and B pixels in adjacent pixel columns is read out sequentially from a common terminal, and acquiring reference data corresponding to each terminal based on the pixel data read out from each terminal; setting a readout mode based on reference data acquired in each of the first readout mode and the second readout mode; a step of correcting data of each pixel in an effective pixel area excluding the light-shielded pixel area in the imaging element by subtracting corresponding reference data from pixel data read out from each terminal in the set readout mode; A program for executing the above.

Images