Imaging system and electronic endoscope system

Abstract

To improve flexibility in setting a length of a flash irradiation period when performing imaging of a subject by irradiating the subject with flash by a rolling shutter method.SOLUTION: An electronic endoscope system 1 includes: a CMOS image sensor 14 configured to perform imaging of a subject by a rolling shutter method; and a light source device 28 that emits flash for stroboscopic imaging of the subject. In this system, the light source device 28 emits flash by setting the period from one flash end time to the next flash start time longer than a flash prohibition period in which the emission of flash is prohibited for at least one frame period. The electronic endoscope system 1 generates a display image by processing the images captured per frame acquired by the CMOS image sensor 14 based on the flash emission timing by the light source device 28.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to stroboscopy performed using a rolling shutter type image sensor. [Background technology]

[0002] Conventionally, an electronic endoscope system for performing laryngeal stroboscopy is known. For example, the electronic endoscope system described in Patent Document 1 is configured to continuously read pixel signals from a CMOS image sensor by a rolling shutter method and to strobe light an LED in accordance with the subject's voice. When strobe light (flash) is irradiated, the pixel signals of the line read after strobe light irradiation of the frame in which reading from the CMOS image sensor is being performed are stored in a frame buffer as a lower image, and in the next frame, the pixel signals of the line above the lower image of the previous frame are directly output to an image signal processing unit, and then the lower image of the previous frame held in the frame buffer is output to the image signal processing unit. In this way, in strobe photography using a rolling shutter, an image for one screen shot by the same strobe light is obtained without providing a period for strobe exposure. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6196105 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the above electronic endoscope system has a problem in that the period (flash irradiation period) during which the strobe light (flash) is irradiated must be shorter than the readout of one line of the CMOS image sensor. If the flash irradiation period is short, shorter than the readout of one line of the CMOS image sensor, the light intensity of the light source such as an LED must be increased to obtain sufficient brightness for observing the vocal cords, which may result in adverse effects such as the need to strengthen heat dissipation measures for the light source and deterioration of the durability of the light source. Furthermore, increasing the gain of the CMOS image sensor without increasing the intensity of the light source will result in a decrease in the S / N ratio.

[0005] SUMMARY OF THE PRESENT DISCLOSURE An object of the present invention is to increase the degree of freedom in setting the length of the flash irradiation period when a flash is irradiated on a subject and an image of the subject is captured by the rolling shutter method. [Means for solving the problem]

[0006] One aspect of the present disclosure is An imaging element configured to capture an image of a subject using a rolling shutter method; a light source unit that emits a flash of light to photograph the subject with a flash, the light source unit emitting the flash of light such that a period from a flash end time of one flash to a flash start time of the next flash is longer than a flash prohibition period that prohibits emission of a flash of light for at least one frame period; an image processing unit that processes the captured image obtained by the image sensor on a frame-by-frame basis based on the timing of emitting a flash of light from the light source unit to generate an image for display; The imaging system includes: When a flash is emitted on any one of the lines of the image sensor at an exposure start time of a frame one frame before the current frame, the image processing unit Among the display images of the current frame, an upper image corresponding to a line above the flash start line of the previous frame is set as a captured image obtained by the corresponding line of the current frame; Among the display images of the current frame, a lower image corresponding to a line below the flash end line of the previous frame is set as a captured image obtained by the corresponding line of the previous frame; The display image of the current frame is generated by taking the boundary image from the flash start line to the flash end line of the display image of the current frame as an image obtained by adding together the captured image obtained by the corresponding line of the current frame and the captured image obtained by the corresponding line of the previous frame. When no flash is emitted on any line of the image sensor at an exposure start time of a frame one frame before the current frame, the image processing unit Among the display images of the current frame, an upper image corresponding to a line above the flash start line of two frames before is set as a captured image obtained by the corresponding line of one frame before; Among the display images of the current frame, a lower image corresponding to a line below the flash end line of two frames before is set as a captured image obtained by the corresponding line of two frames before; The display image of the current frame is generated by taking the boundary image from the flash start line to the flash end line of the display image of the current frame as an image obtained by adding together the captured image obtained by the corresponding line of the previous frame and the captured image obtained by the corresponding line of the previous frame.

[0007] In the imaging system, the image processing unit has an amplification processing unit that amplifies a pixel value of each pixel of the display image of a current frame, The amplification processing unit may, depending on the profile of the light emission intensity over the course of the flash time and the readout period of the imaging element, set a gain to be applied to pixels of each line of the boundary image in the display image of the current frame that is greater than a gain to be applied to pixels of each line other than the boundary image.

[0008] The image processing unit may include a filter processing unit that applies a spatial filter to pixel values ​​of pixels included in a predetermined number of lines above and below an adjacent position where the border image and the upper image are adjacent, and a predetermined number of lines above and below an adjacent position where the border image and the lower image are adjacent, for the display image of the current frame.

[0009] The image processing unit may include a filter processing unit that applies a spatial filter to pixel values ​​of pixels included in a predetermined number of lines above and below a central position in a line direction of the border image, for the display image of the current frame.

[0010] The filtering unit may set a filter coefficient of a spatial filter applied to pixels included in each of the predetermined number of lines in accordance with a profile of a light emission intensity according to the passage of time of the flash.

[0011] The image processing unit may have an interpolation processing unit that performs an interpolation process between the current frame and a past frame that is two frames earlier or three frames earlier, using as target lines each line of the border image of the display image of the current frame, or each line of a plurality of lines including, in addition to each line of the border image, a predetermined number of lines above an adjacent position where the border image and the upper image are adjacent and a predetermined number of lines below an adjacent position where the border image and the lower image are adjacent. In that case, the interpolation processing unit: when a flash is emitted in any one of the lines of the imaging element at an exposure start time of one frame before the current frame, a pixel value of a pixel included in each target line in the image for display of the current frame is calculated by taking a weighted average of a pixel value of a corresponding pixel in the image for display of the current frame and a pixel value of a corresponding pixel in the captured image two frames before; If no flash is emitted in any line of the imaging element at the exposure start time of the frame prior to the current frame, the pixel values ​​of pixels included in each target line of the display image of the current frame are calculated by taking a weighted average of the pixel values ​​of the corresponding pixels in the display image of the current frame and the pixel values ​​of the corresponding pixels in the captured image three frames prior.

[0012] The interpolation processing unit may set weights for the weighted average so that the weight for the pixel value of the corresponding pixel in the captured image of the past frame is maximum at the center of the line direction of the boundary image, and the weight for the pixel value of the corresponding pixel in the captured image of the past frame decreases as the distance from the center increases above or below.

[0013] The interpolation processing unit may set weights for the weighted average so that weighting for pixel values ​​of pixels included in a line located in adjacent positions where the boundary image and the upper image are adjacent and / or a line located in adjacent positions where the boundary image and the lower image are adjacent is maximum, and weighting for pixel values ​​of corresponding pixels of captured images of previous frames decreases as the distance from the adjacent positions increases vertically.

[0014] The interpolation processing unit may set weights of the weighted average applied to pixels included in the target line in accordance with a profile of a light emission intensity according to the passage of time of the flash.

[0015] One aspect of the present disclosure is A microphone and a voice detection unit that detects a voice frequency from the voice signal acquired from the microphone; An imaging system according to any one of claims 1 to 9, The light source unit of the electronic endoscope system emits the flash of light at a period synchronized with the audio frequency detected by the audio detection unit. Effect of the Invention

[0016] According to the imaging system and electronic endoscope system described above, when a subject is irradiated with a flash of light and an image of the subject is taken by the rolling shutter method, the length of the flash irradiation period can be set with greater freedom. [Brief description of the drawings]

[0017] [Figure 1] 1 is a block diagram showing an example of a configuration of an electronic endoscope system according to an embodiment; [Diagram 2] FIG. 2 is a diagram showing the relationship between the operation of a CMOS image sensor and the timing of emitting a flash of light. [Diagram 3] 11 is a timing chart showing an operation of generating a composite image when a flash of light is emitted in accordance with a vocal cord vibration frequency of 1000 Hz in the electronic endoscope system of one embodiment. [Figure 4] 11 is a timing chart showing an operation of generating a composite image when a flash of light is emitted in accordance with a vocal cord vibration frequency of 250 Hz in the electronic endoscope system of one embodiment. [Diagram 5] 11 is a timing chart showing an operation of generating a composite image when a flash of light is emitted in accordance with a vocal cord vibration frequency of 125 Hz in the electronic endoscope system of one embodiment. [Figure 6] 11 is a timing chart showing an operation of generating a composite image when a flash of light is emitted in accordance with a vocal cord vibration frequency of 115 Hz in the electronic endoscope system of one embodiment. [Figure 7] 11 is a timing chart showing an operation of generating a composite image when a flash of light is emitted in accordance with a vocal cord vibration frequency of 63 Hz in the electronic endoscope system of one embodiment. [Figure 8] 1 is a diagram showing a relationship between a vocal cord vibration frequency and a frame rate in an electronic endoscope system according to an embodiment of the present invention. [Figure 9] 2 is a diagram for explaining an embodiment in which an amplification processing unit and a filter processing unit are added to the system shown in FIG. [Figure 10]10A to 10C are diagrams illustrating an example of a method for improving the image quality of a composite image in the electronic endoscope system according to the embodiment shown in FIG. [Figure 11] FIG. 13 is a diagram illustrating an example of spatial filtering applied to a composite image. [Figure 12] FIG. 13 is a diagram illustrating an example of spatial filtering applied to a composite image. [Figure 13] FIG. 13 is a diagram illustrating an example of spatial filtering applied to a composite image. [Figure 14] 2 is a diagram for explaining an embodiment in which an amplification processing unit and an interpolation processing unit are added to the system shown in FIG. [Figure 15] 15A to 15C are diagrams illustrating an example of a method for improving the image quality of a composite image in the electronic endoscope system of the embodiment shown in FIG. 14. [Figure 16] 15A to 15C are diagrams illustrating an example of a method for improving the image quality of a composite image in the electronic endoscope system of the embodiment shown in FIG. 14. [Figure 17] 15A to 15C are diagrams illustrating an example of a method for improving the image quality of a composite image in the electronic endoscope system of the embodiment shown in FIG. 14. [Figure 18] 11A and 11B are diagrams illustrating an example of frame interpolation processing applied to a composite image. [Figure 19] 11A and 11B are diagrams illustrating an example of frame interpolation processing applied to a composite image. [Figure 20] 11A and 11B are diagrams illustrating an example of frame interpolation processing applied to a composite image. [Figure 21] 2 is a block diagram showing an example of the configuration of an electronic endoscope system according to an embodiment different from that shown in FIG. 1. [Figure 22] 2 is a block diagram showing an example of the configuration of an electronic endoscope system according to an embodiment different from that shown in FIG. 1. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An electronic endoscope system, which is an embodiment of an imaging system according to the present invention, will be described in detail below with reference to the drawings. 1 is a block diagram showing an example of the configuration of an electronic endoscope system 1 according to an embodiment. The electronic endoscope system 1 is a system specialized for medical use, and is particularly used for laryngostroboscopy. In laryngostroboscopy, an operator uses a light source device that intermittently emits flashes of light and an electronic scope to perform stroboscopic photography and display an image to observe the vocal cords as a subject. 1, an electronic endoscope system 1 according to an embodiment includes an electronic scope (endoscope) 10, a processor 20 incorporating a light source device, a monitor 30, and a microphone 40. The processor 20 is connected to the monitor 30 and the microphone 40.

[0019] The processor 20 includes an audio processing circuit 31 and a frequency detection circuit 32 . When strobe photography of the vocal cords is performed, a flash is intermittently turned on (emitted) in accordance with the vocal cord vibration frequency based on the voice produced by the patient. The voice produced by the patient is collected by the microphone 40. The voice processing circuit 31 removes noise and the like from the voice collected by the microphone 40 to produce a voice waveform suitable for detecting the vocal cord vibration frequency. The frequency detection circuit 32 detects the vocal cord vibration frequency from the voice waveform obtained by the voice processing circuit 31.

[0020] The processor 20 includes a system controller 21, an operation panel 26, a timing control circuit 27, and a light source device 28 (an example of a light source section). The system controller 21 executes various programs and comprehensively controls the entire electronic endoscope system 1. The system controller 21 changes each operation of the electronic endoscope system 1 and parameters for each operation in response to instructions from an operator (observer) inputted to an operation panel 26. The system controller 21 supplies clock pulses to each circuit in the electronic endoscope system 1 to adjust the timing of the operation of each part.

[0021] Under the control of the system controller 21, the timing control circuit 27 determines the timing of emitting a flash of light according to the vocal cord vibration frequency detected by the frequency detection circuit 32, and sequentially transmits a signal (light emission timing signal) indicating the determined timing of emitting a flash of light to the light source device 28. The timing control circuit 27 receives data on the length of the light emission prohibition period from the system controller 21 at the time of system startup, and determines the light emission timing so as to maintain the length of the light emission prohibition period. The timing control circuit 27 sequentially transmits the light emission timing signal to the synthesis unit 24 in real time.

[0022] The flash prohibition period means the shortest period from the flash end time of one flash to the flash start time of the next flash. In other words, the emission of the next flash is prohibited from the flash end time of one flash until the flash prohibition period has elapsed. The flash prohibition period is at least one frame period (one frame cycle) or longer. The reason why the flash prohibition period is at least one frame period is to prevent the same line of CMOS image sensor 14 from being exposed two or more times within one frame period.

[0023] Light source device 28 emits a flash of light L for illuminating a subject such as vocal cords in synchronization with a light emission timing signal sent from timing control circuit 27. The flash of light L may be white light, pseudo-white light, or special light in a specific wavelength band. According to one embodiment, the flash of light L emitted from light source device 28 is focused by a focusing lens on an incident end surface of LCB (Light Carrying Bundle) 11 and enters LCB 11. The type of light source of light source device 28 is not limited, but may be, for example, an LED, a laser diode, a high-intensity lamp (for example, a xenon lamp, a metal halide lamp, a mercury lamp, or a halogen lamp), etc. In the following description, a case in which the light source of light source device 28 is an LED will be described as an example.

[0024] The flash of light L incident on the LCB 11 propagates through the LCB 11. The flash of light L propagating through the LCB 11 is emitted from the exit end face of the LCB 11 disposed at the tip of the electronic scope 10, and is irradiated onto the subject via the light distribution lens 12. The return light from the subject illuminated by the flash of light L from the light distribution lens 12 forms an optical image on the light receiving surface of the CMOS image sensor 14 via the objective lens 13.

[0025] The CMOS image sensor 14 is an example of an imaging element configured to capture an image of a subject using a rolling shutter method. The CMOS image sensor 14 has, for example, a Bayer type pixel arrangement, and accumulates an optical image formed at each pixel on the light receiving surface as an electric charge according to the amount of light, and generates and outputs image signals of R (Red), G (Green), and B (Blue). Note that, instead of the CMOS image sensor 14, a CCD (Charge-Coupled Device) image sensor or other types of imaging devices may be applied. The CMOS image sensor 14 may be equipped with a complementary color filter.

[0026] The CMOS driver 15 provided in the electronic scope 10 drives and controls the CMOS image sensor 14 in accordance with a clock pulse supplied from the system controller 21 at a timing synchronized with the frame rate of the video processed by the processor 20. The CMOS driver 15 performs predetermined processing including A / D conversion on the captured image input from the CMOS image sensor 14 and outputs the image to the image input processing unit 22 of the processor 20. An image signal representing an image of an object is input to the CMOS driver 15 from the CMOS image sensor 14 at a predetermined frame period.

[0027] The frame period (one frame period) is, for example, 1 / 120 seconds, 1 / 60 seconds, or 1 / 30 seconds, but the following description will be given taking the frame period of 1 / 60 seconds as an example, i.e., the length of the flash prohibition period is at least 1 / 60 seconds.

[0028] The processor 20 includes an image input processing unit 22 , a frame buffer 23 , a synthesis unit 24 , and an image output processing unit 25 in order to generate an image to be displayed on the monitor 30 . The image input processing unit 22 performs predetermined signal processing such as noise removal, demosaic processing, matrix operation, etc. on the captured image sent from the CMOS driver 15 on a frame-by-frame basis, and transmits the captured image on a frame-by-frame basis (the captured image of the current frame) to a frame buffer 23 and a synthesis unit 24. The frame buffer 23 is a memory that buffers the captured images sent from the image input processing unit 22 on a frame-by-frame basis. In one embodiment, the frame buffer 23 temporarily stores three frames of captured images, i.e., frames one frame before the current frame, frames two frames before, and frames three frames before the current frame. When frame interpolation processing, which will be described later, is not performed, the frame buffer 23 only needs to temporarily store two frames of captured images, i.e., frames one frame before the current frame and frames two frames before the current frame.

[0029] The synthesis unit 24 generates a composite image by synthesizing the captured images of a plurality of frames stored in the frame buffer 23 based on the light emission timing indicated by the light emission timing signal received from the timing control circuit 27. A specific example of generating a composite image will be described in detail later. The image output processing unit 25 processes the composite image generated by the composition unit 24 on a frame-by-frame basis to generate screen data for monitor display, and converts the generated screen data for monitor display into a predetermined video format signal. The converted video format signal is output to the monitor 30. As a result, an image of the subject (vocal cords) is displayed on the display screen of the monitor 30. The frame buffer 23, the synthesis unit 24, and the image output processing unit 25 constitute an example of an image processing unit of the present invention.

[0030] Next, the operation of the CMOS image sensor 14 based on a flash of light will be described with reference to FIG. In Figure 2, (a) shows the operation of the CMOS image sensor 14 during a two-frame period (1 / 60 sec + 1 / 60 sec) of consecutive Nth and N+1th frames, (b) shows a flash of light L, and (c) shows an enlarged portion of (a). In the CMOS image sensor 14, exposure for each frame is started with a time difference between each of a plurality of lines in the effective pixel area excluding the ineffective pixel area. Note that the ineffective pixel area consists of one line or a plurality of lines. As shown in Fig. 2(a), in the CMOS image sensor 14, exposure in the Nth frame is started with a time difference for each line (from top to bottom) included in the effective pixel area excluding the invalid pixel area, LN1, LN2, ..., LNn (the last line of the effective pixel area). As shown in an enlarged view in Fig. 2(c), each frame period includes a charge accumulation period T INT Then, the charge accumulation for one frame period ends, and a signal readout period T RO The signal readout period T RO Charge storage is not possible in the

[0031] As shown in Figures 2(a) and (b), flash L is light emitted during the short period from flash start time Ts to flash end time Te, and is used for charge accumulation in a portion of the lines in the effective pixel area in the Nth frame and in a portion of the lines in the effective pixel area in the N+1th frame. The period from the flash start time Ts to the flash end time Te is the flash illumination period. Referring to the enlarged view of Figure 2(c), the line corresponding to the flash start time Ts of the flash L is defined as the flash start line Ls, and the line corresponding to the flash end time Te of the flash L is defined as the flash end line Le. For each line from line LN1 to the flash start line Ls, the charge generated by the flash L is accumulated during the charge accumulation period T INT On the other hand, for each line from the flash end line Le to the last line LNn of the effective pixel area, the charge generated by the flash L is accumulated during the charge accumulation period T INTFor each line from the flash start line Ls to the flash end line Le, the charge generated by a portion of the flash L is accumulated during the charge accumulation period T INT The charge generated by the remaining part of the flash L is accumulated in the charge accumulation period T INT is accumulated.

[0032] As can be seen from FIG. 2, one display image obtained by flashing light L can be obtained based on a composite image obtained by appropriately combining the following upper image, lower image, and border image. The upper image obtained during the N+1th frame by each line from line LN1 to the flash start line Ls The lower image obtained during the Nth frame period by each line from the flash end line Le to the last line LNn of the effective pixel area. The boundary image between the upper and lower images obtained during the Nth and N+1th frames by each line from the flash start line Ls to the flash end line Le.

[0033] In the electronic endoscope system 1 of this embodiment, as long as the emission prohibition period is ensured, the emission timing of the flash L can be set arbitrarily, and the flash irradiation period can also be set arbitrarily. Therefore, the charge accumulation period T INT There may be cases where flash L does not occur even at the start time of the process. Taking such cases into consideration, more specifically, the synthesis unit 24 of the processor 20 generates a synthetic image that serves as the basis for the display image by dividing the process into the following processes Pa and Pb.

[0034] [Process Pa] When a flash is emitted on any line of the CMOS image sensor 14 at the exposure start time of the frame immediately preceding the current frame, the synthesis unit 24 generates a synthetic image of the current frame as follows. (a-1) Of the composite image of the current frame, the upper image corresponding to the line above the flash start line of the previous frame is set as the captured image obtained by the corresponding line of the current frame. (a-2) Of the composite image of the current frame, the lower image corresponding to the line below the flash end line of the previous frame is set as the captured image obtained by the corresponding line of the previous frame. (a-3) Among the composite images of the current frame, the boundary image from the flash start line to the flash end line is an image obtained by adding together the captured image obtained by the corresponding line of the current frame and the captured image obtained by the corresponding line of the previous frame.

[0035] [Process Pb] If no flash was emitted on any of the lines of the CMOS image sensor 14 at the exposure start time of the frame immediately preceding the current frame, the synthesis unit 24 generates a synthetic image of the current frame as follows. (b-1) Of the composite image of the current frame, the upper image corresponding to the line above the flash start line of two frames before is taken as the captured image obtained by the corresponding line in the previous frame. (b-2) Of the composite image of the current frame, the lower image corresponding to the line below the flash end line of two frames before is set as the captured image obtained by the corresponding line of two frames before. (b-3) Among the composite images of the current frame, the boundary image from the flash start line to the flash end line is an image obtained by adding together the captured image obtained by the corresponding line in the previous frame and the captured image obtained by the corresponding line in the previous frame.

[0036] Next, with reference to the timing charts of Figures 3 to 7, a specific example of the operation of generating a composite image based on the above processes Pa and Pb in the electronic endoscope system 1 when the vocal cord vibration frequencies obtained from the microphone 40 are different will be described. Note that Figures 3 to 7 show an example in which the emission prohibition period is set to approximately one frame period (1 / 60 seconds). 3 to 7 show, for each frame of the Mth frame, the M+1th frame, etc., (a) the operation of the CMOS image sensor 14 (CMOS operation), (b) the timing of emitting the flash, (c) the captured image obtained by the CMOS image sensor 14, and (d) the synthesized image by the synthesis unit 24. In (a) of FIG. 3 to FIG. 7, the emission period of the flash shown in (b) is shown by a vertical line. In (b) of FIG. 3 to FIG. 7, the waveform of the vocal cord vibration is also shown. The emission timing of the flashes L1, L2, L3, etc. is adjusted according to the vocal cord vibration frequency Fv so as to satisfy the conditions of the emission prohibition period.

[0037] (I) When the vocal fold vibration frequency Fv=1000Hz (Figure 3) 3, because flash L1 is emitted during the Mth frame period, a lower image IM_L1 due to flash L1 is obtained as the captured image for the Mth frame, and an upper image IM_L1a due to flash L1 is obtained as the captured image for the (M+1)th frame. Similarly, a lower image IM_L2 due to flash L2 is obtained as the captured image for the (M+1)th frame, and an upper image IM_L2a due to flash L2 is obtained as the captured image for the (M+2)th frame. A lower image IM_L3 due to flash L3 is obtained as the captured image for the (M+2)th frame, and an upper image IM_L3a due to flash L3 is obtained as the captured image for the (M+3)th frame. 3, flash L4 is emitted during the (M+4)th frame period, so a lower image IM_L4 due to flash L4 is obtained as the captured image for the (M+4)th frame, and an upper image IM_L4a due to flash L4 is obtained as the captured image for the (M+5)th frame. Similarly, a lower image IM_L5 due to flash L5 is obtained as the captured image for the (M+5)th frame, and an upper image IM_L5a due to flash L5 is obtained as the captured image for the (M+6)th frame.

[0038] When creating a composite image of each of the M+1 to M+3 frames, since a flash is emitted on one of the lines of the CMOS image sensor 14 at the exposure start time of the frame immediately preceding each frame, a composite image is generated in the following manner according to the above process Pa. For example, when generating a composite image of the (M+1)th frame as the current frame, an upper image of the composite image corresponding to the line above the flash start line of the Mth frame is set as an upper image IM_L1a obtained from the corresponding line of the (M+1)th frame (current frame), and a lower image of the composite image corresponding to the line below the flash end line of the Mth frame is set as a lower image IM_L1 obtained from the corresponding line of the Mth frame (one frame before). Note that, although not visible in Fig. 3, a boundary image from the flash start line to the flash end line of the composite image of the (M+1)th frame is set as an image obtained by adding together the captured image obtained from the corresponding line of the (M+1)th frame (current frame) and the captured image obtained from the corresponding line of the frame one frame before. The same applies to the case where a composite image is generated for each of the (M+2)th and (M+3)th frames as the current frame.

[0039] When creating a composite image of the (M+4)th frame as the current frame, since no flash was emitted on any of the lines of the CMOS image sensor 14 at the exposure start time of the (M+3)th frame, which is one frame before, the composite image is generated as follows. Of the composite image for the M+4th frame, an upper image corresponding to the line above the flash start line in the M+2th frame, two frames before, is taken as captured image IM_L3a obtained from the corresponding line in the M+3th frame, one frame before, and a lower image of the composite image corresponding to the line below the flash end line in the M+2th frame, two frames before, is taken as captured image IM_L3 obtained from the corresponding line in the M+2th frame, two frames before. Although not visible in Figure 3, of the composite image for the M+4th frame, the boundary image from the flash start line to the flash end line is taken as an image obtained by adding together the captured image obtained from the corresponding line in the M+3th frame, one frame before, and the captured image obtained from the corresponding line in the M+2th frame, two frames before. In short, when creating a composite image of the (M+4)th frame, the captured image IM_L4a has not been obtained, and a composite image based on the captured images IM_L4, IM_L4a cannot be created, so the composite image of the (M+3)th frame is reproduced as shown in Figure 3 (i.e., it will be identical to the composite image of the (M+3)th frame).

[0040] For each of the M+5th and M+6th frames as the current frames, a composite image is generated according to the above process Pa in the same manner as in the case of creating a composite image of each of the M+1th to M+3rd frames.

[0041] (II) When the vocal fold vibration frequency Fv=250Hz (Figure 4) As shown in FIG. 4(b), when the vocal cord vibration frequency Fv is 250 Hz, the period from when the no-emission period following a flash has elapsed until the next flash is emitted is longer than when the vocal cord vibration frequency Fv is 1000 Hz. 4, flash L1 is emitted during the Mth frame period, so that a lower image IM_L1 due to flash L1 is obtained as the captured image of the Mth frame, and an upper image IM_L1a due to flash L1 is obtained as the captured image of the M+1th frame. Similarly, a lower image IM_L2 due to flash L2 is obtained as the captured image of the M+1th frame, and an upper image IM_L2a due to flash L2 is obtained as the captured image of the M+2th frame. A lower image IM_L3 due to flash L3 is obtained as the captured image of the M+2th frame, and an upper image IM_L3a due to flash L3 is obtained as the captured image of the M+3th frame. In FIG. 4, a flash of light L4 is emitted during the (M+4)th frame period, and an image IM_L4 of the entire effective pixel area due to the flash of light L4 is obtained as the captured image of the (M+4)th frame.

[0042] When creating a composite image of each of the M+1th to M+3th frames as the current frame, a flash is emitted on one of the lines of the CMOS image sensor 14 at the exposure start time of the frame immediately preceding each frame, so the composite image is created according to the above process Pa. The creation of this composite image is the same as the creation of a composite image of each of the M+1th to M+3th frames when the vocal cord vibration frequency Fv=1000 Hz in (I). For the current frame, the M+4th frame, a composite image is also generated in accordance with the above process Pa. However, since a flash of light is emitted in the invalid pixel area of ​​the CMOS image sensor 14 at the exposure start time of the previous M+3th frame, no actual synthesis is performed, and the composite image of the M+4th frame becomes the captured image IM_L4. For the (M+5)th frame as the current frame, this corresponds to a case where no flash was emitted on any line of the CMOS image sensor 14 at the exposure start time of the (M+4)th frame, which is one frame before, so a composite image is generated according to the above process Pb. As a result, the composite image of the (M+5)th frame is the same as the composite image of the (M+4)th frame.

[0043] (III) When the vocal fold vibration frequency Fv=125Hz (Figure 5) As shown in FIG. 5(b), when the vocal cord vibration frequency Fv is 125 Hz, the period from when the no-emission period following a flash has elapsed until the next flash is emitted is longer than when the vocal cord vibration frequency Fv is 250 Hz. 5, since flash L1 is emitted during the Mth frame period, a lower image IM_L1 due to flash L1 is obtained as the captured image of the Mth frame, and an upper image IM_L1a due to flash L1 is obtained as the captured image of the M+1th frame. Similarly, a lower image IM_L2 due to flash L2 is obtained as the captured image of the M+1th frame, and an upper image IM_L2a due to flash L2 is obtained as the captured image of the M+2th frame. A lower image IM_L3 due to flash L3 is obtained as the captured image of the M+3th frame, and an upper image IM_L3a due to flash L3 is obtained as the captured image of the M+4th frame. A lower image IM_L4 due to flash L4 is obtained as the captured image of the M+4th frame.

[0044] When creating a composite image of each of the M+1th to M+2nd frames as the current frame, a flash is emitted on one of the lines of the CMOS image sensor 14 at the exposure start time of the frame immediately preceding each frame, so the composite image is created according to the above process Pa. The creation of this composite image is the same as when creating a composite image of each of the M+1th to M+3th frames when the vocal cord vibration frequency Fv=1000 Hz in (I). For the (M+3)th frame as the current frame, this corresponds to a case where no flash was emitted on any line of the CMOS image sensor 14 at the exposure start time of the (M+2)th frame, which is one frame before, so a composite image is generated according to the above process Pb. As a result, the composite image of the (M+3)th frame is the same as the composite image of the (M+2)th frame. When creating a composite image of each of the M+4th to M+5th frames as the current frame, a flash is emitted on any line of the CMOS image sensor 14 at the exposure start time of the frame immediately preceding each frame, so the composite image is generated according to the above process Pa. That is, the composite images of the M+4th to M+5th frames are created in the same manner as when creating a composite image of the M+1th to M+2nd frames.

[0045] (IV) When the vocal fold vibration frequency Fv=115Hz (Figure 6) As shown in FIG. 6(b), when the vocal cord vibration frequency Fv is 115 Hz, the period from when the emission prohibition period following a certain flash has elapsed until the next flash is emitted is very short. In Fig. 6, since the flash L1 is emitted during the Mth frame period, a lower image IM_L1 due to the flash L1 is obtained as the captured image of the Mth frame, and an upper image IM_L1a due to the flash L1 is obtained as the captured image of the M+1th frame. Similarly, a lower image IM_L2 due to the flash L2 is obtained as the captured image of the M+1th frame, and an upper image IM_L2a due to the flash L2 is obtained as the captured image of the M+2th frame. A lower image IM_L3 due to the flash L3 is obtained as the captured image of the M+2th frame, and an upper image IM_L3a due to the flash L3 is obtained as the captured image of the M+3th frame. A lower image IM_L4 due to the flash L4 is obtained as the captured image of the M+3th frame, and an upper image IM_L4a due to the flash L4 is obtained as the captured image of the M+4th frame. A lower image IM_L5 due to the flash L5 is obtained as the captured image of the M+4th frame.

[0046] When creating a composite image of each of the M+1th to M+5th frames as the current frame, a flash is emitted on one of the lines of the CMOS image sensor 14 at the exposure start time of the frame immediately preceding each frame, so the composite image is created according to the above process Pa. The creation of this composite image is the same as the creation of a composite image of each of the M+1th to M+3th frames when the vocal cord vibration frequency Fv=1000 Hz in (I).

[0047] (V) When the vocal fold vibration frequency Fv=63Hz (Figure 7) As shown in FIG. 7(b), when the vocal cord vibration frequency Fv is 63 Hz, the period from when the emission prohibition period following a certain flash has elapsed until the next flash is emitted is very long. 7, since flash L1 is emitted during the Mth frame period, a lower image IM_L1 due to flash L1 is obtained as the captured image of the Mth frame, and an upper image IM_L1a due to flash L1 is obtained as the captured image of the (M+1)th frame. Similarly, a lower image IM_L2 due to flash L2 is obtained as the captured image of the (M+2)th frame, and an upper image IM_L2a due to flash L2 is obtained as the captured image of the (M+3)th frame. A lower image IM_L3 due to flash L3 is obtained as the captured image of the (M+4)th frame, and an upper image IM_L3a due to flash L3 is obtained as the captured image of the (M+5)th frame.

[0048] When creating a composite image of the M+1th frame as the current frame, a flash is emitted on one of the lines of the CMOS image sensor 14 at the exposure start time of the frame immediately preceding the current frame (the Mth frame), so the composite image is generated according to the above process Pa. When creating a composite image of the M+2th frame as the current frame, since no flash was emitted on any line of the CMOS image sensor 14 at the exposure start time of the frame immediately preceding that frame (the M+1st frame), the composite image is generated according to the above process Pb. When creating a composite image of the (M+3)th frame as the current frame, a flash was emitted on one of the lines of the CMOS image sensor 14 at the exposure start time of the frame immediately preceding the current frame (the (M+2)th frame), so the composite image is generated according to the above process Pa. When creating a composite image of the (M+4)th frame as the current frame, since no flash was emitted on any line of the CMOS image sensor 14 at the exposure start time of the frame immediately preceding that frame (the (M+3)th frame), the composite image is generated according to the above process Pb. When creating a composite image of the (M+5)th frame as the current frame, a flash was emitted on one of the lines of the CMOS image sensor 14 at the exposure start time of the frame immediately preceding that frame (the (M+4)th frame), so the composite image is generated according to the above process Pa. As a result, a composite image of each frame is generated as shown in FIG.

[0049] As specifically described above with reference to Figures 3 to 7, in the electronic endoscope system 1, the timing of emitting the flash of light is determined so that the interval between successive flashes of light is longer than a predetermined light emission prohibition period and is synchronized with the vocal cord vibration frequency. At this time, if no flash is emitted on any line of the CMOS image sensor 14 at the exposure start time of the CMOS image sensor 14 operating in the rolling shutter system, the composite images of two consecutive frames will be the same, and the frame rate will be low. For example, as shown in Fig. 6, when the vocal cord vibration frequency Fv is 115 Hz, a flash is emitted on any line of the CMOS image sensor 14 at the exposure start time for each frame, so the composite images of each frame are not the same, and the frame rate is high (about 60 fps). On the other hand, as shown in Fig. 7, when the vocal cord vibration frequency Fv is 63 Hz, a situation in which no flash is emitted on any line of the CMOS image sensor 14 at the exposure start time occurs every other frame, so the composite images of two consecutive frames will be the same, and the frame rate is low (about 30 fps).

[0050] FIG. 8 shows the relationship between the vocal cord vibration frequency Fv and the frame rate of the synthetic image. In the electronic endoscope system 1 of this embodiment, the frame rate varies discontinuously based on the vocal cord vibration frequency Fv and the timing of the flash, which is set according to a predetermined light emission prohibition period. This is because the period from the elapse of the light emission prohibition period after a certain flash to the emission of the next flash varies discontinuously according to the vocal cord vibration frequency. 8, a frame rate of at least 30 fps is ensured for any vocal cord vibration frequency, and as the vocal cord vibration frequency increases, the frame rate approaches 60 fps, resulting in a high frame rate. When the total length of the flash on period and the flash off period matches the period corresponding to the vocal cord vibration frequency Fv, the frame rate increases, so that the frame rate periodically increases with respect to the vocal cord vibration frequency Fv, as shown in FIG. Note that FIG. 8 shows the results when the light emission prohibition period is set to approximately one frame period (1 / 60 seconds) as shown in FIGS. 3 to 7. If the light emission prohibition period is made longer, the results will differ from those in FIG. 8.

[0051] Next, a post-processing that is preferably performed on the composite image obtained by the above processes Pa and Pb will be described. As described above, in the process Pa, the boundary image from the flash start line to the flash end line of the composite image of the current frame is taken as an image obtained by adding together the captured image obtained by the corresponding line of the current frame and the captured image obtained by the corresponding line of the previous frame. On the other hand, in the process Pb, the boundary image from the flash start line to the flash end line of the composite image of the current frame is taken as an image obtained by adding together the captured image obtained by the corresponding line of the previous frame and the captured image obtained by the corresponding line of two frames previous. The boundary image of the composite image is the area between the upper image and the lower image of the composite image. As shown in FIG. 2, this boundary image is a region between the upper image and the lower image of the composite image. RO at least partially overlap with each other, and is an image taken during a period in which exposure by the flash of light L is insufficient. For this reason, if no processing were performed on this boundary image, it would stand out as a horizontal black line in the composite image, as shown in (d) of Figures 3 to 7, and would be visually recognized by the viewer as a black line moving in the vertical direction. Therefore, in order to make the horizontal black lines of the boundary image less noticeable in the composite image, it is preferable to perform the image processing described below on the boundary image and the image of the adjacent lines as post-processing on the composite image.

[0052] In one embodiment, the post-processing for the composite image is any one of the following three processes Pc1 to Pc3, or a combination of two or more of these processes. [Process Pc1] Digital gain addition process The digital gain application process amplifies the pixel value of each pixel in each line of the boundary image. By applying the digital gain, the brightness of the boundary image obtained in a state where the exposure due to the flash is insufficient is corrected. [Processing Pc2] Spatial filter processing The spatial filtering process is a process in which a spatial filter is applied to the pixel values ​​of the pixels of a boundary image included in the composite image of the current frame, or the boundary image and a predetermined number of lines in its vicinity, for the purpose of smoothing (blurring) the image. [Process Pc3] Frame interpolation processing Frame interpolation processing is processing that performs interpolation between the current frame and a previous frame that is two frames or three frames earlier, targeting a boundary image included in the current frame, or the boundary image and a predetermined number of lines in its vicinity.

[0053] Although only the process Pc1 may be used, a combination of the processes Pc1 and Pc2, or a combination of the processes Pc1 and Pc3 is preferable in order to more effectively make the horizontal black lines of the boundary images less noticeable in the composite image.

[0054] (i) First aspect of post-processing of composite image The first mode of post-processing for the composite image is a combination of the above-mentioned processes Pc1 and Pc2. A system configuration in which this first aspect is implemented is shown in Fig. 9. As shown in Fig. 9, in the first aspect, an amplification processing unit 51 and a filter processing unit 52 are further included between the synthesis unit 24 and the image output processing unit 25 in the processor 20 shown in Fig. 1. Fig. 10 is a diagram for explaining image processing when the first mode of post-processing for a composite image is applied. In Fig. 10, (a) shows the vertical synchronization signal Vsync of the CMOS image sensor 14, (b) shows the operation of the CMOS image sensor 14 (CMOS operation), (c) shows the flash L, (d) shows the captured image acquired by the CMOS image sensor 14, (e) shows the composite image, (f) shows the corrected image after the digital gain is applied, and (g) shows the corrected image after the spatial filter processing.

[0055] 10, as shown in Fig. 10(a) and (b), the CMOS image sensor 14 starts exposure by a rolling shutter method in frame units in synchronization with a vertical synchronization signal Vsync. As shown in Fig. 10(d), when the current frame is taken as a reference, a lower image IM_L is obtained as an image captured in the previous frame for a flash of light L emitted in the previous frame, and an upper image IM_La is obtained as an image captured in the current frame. As shown in Fig. 10(e), the composite image of the current frame is an image obtained by combining the lower image IM_L and the upper image IM_La, but this composite image has a noticeable horizontal black line in the boundary image.

[0056] The amplification processing unit 51 shown in Fig. 9 generates a corrected image after digital gain is applied (Fig. 10(f)) by amplifying the pixel values ​​(digital values) of each pixel of each line of the boundary image in the composite image of the current frame generated by the composition unit 24. This increases the luminance of each pixel of each line of the boundary image, making horizontal black lines less noticeable in the boundary image.

[0057] The filter processing unit 52 shown in Fig. 9 applies a spatial filter to the pixel values ​​of pixels included in a predetermined number of lines above and below the adjacent position where the boundary image and the upper image IM_La are adjacent, and a predetermined number of lines above and below the adjacent position where the boundary image and the lower image IM_L are adjacent, for the composite image of the current frame, to generate a spatially filtered corrected image (Fig. 10(g)). This smooths the boundary image and the image in its vicinity, making horizontal black lines in the boundary image even less noticeable. The spatial filter applied here may be any known spatial filter that smoothes between the boundary image and the upper image IM_La, and between the boundary image and the lower image IM_L, such as a Gaussian filter or an averaging filter.

[0058] Next, a specific example of setting a spatial filter in the first aspect of post-processing on a composite image will be described with reference to FIGS. In each of Figures 11 to 13, (a) shows the digital gain setting for each pixel on each line of the boundary image, (b) shows the flash profile, (c) shows the processing target pixel for the spatial filter, and (d) shows the filter coefficient (kernel) of the spatial filter. In each figure (a), as in Figure 2 (c), the charge accumulation period T IN T and signal readout period T RO In addition, the flash profile shown in (b) of each figure is compressed vertically. The flash profile is a profile of the emission intensity over time of the flash, in other words, the emission characteristics of the flash when the horizontal axis is time and the vertical axis is the emission intensity of the flash.

[0059] In Fig. 2(b), an ideal pulsed flash profile is shown for convenience of explanation, but the actual flash profile may vary depending on the magnitude of the light emission intensity. For example, Figs. 11 to 13(b) show flashes caused by the same LED, and are flash profiles obtained when the light emission intensity is reduced (i.e., the LED current is reduced) in the order of Fig. 11(b), Fig. 12(b), and Fig. 13(b). In this way, when the light emission intensity of the LED is varied, the flash profile of the LED (particularly at the rise and fall) varies depending on changes in the characteristics of the drive circuit, power supply circuit, and the like within light source device 28.

[0060] 11 to 13(a), the flash profile and the signal readout period T ROThe exposure of each line of the boundary image varies depending on the degree of overlap between and . Therefore, preferably, the amplification processing unit 51 makes the gain applied to the pixels of each line of the boundary image in the composite image of the current frame larger than the gain applied to the pixels of each line other than the boundary image, depending on the flash profile and the signal readout period of the CMOS image sensor 14.

[0061] For example, in the digital gain setting example in Fig. 11(a), the flash profile and the signal readout period T RO Since there is almost no overlap, the digital gain is set to "1.0". RO is largely overlapped with the flash profile. In other words, the peak of the flash is within the signal readout period T RO Since this corresponds to a line that is not exposed in either the previous or next frame, the digital gain is set to "1.2." Since there are portions in the flash profile between Line X+2 and Line X+7 that are not exposed in either the previous or next frame, if the boundary image is not amplified, horizontal black lines will be easily noticeable in the boundary image. Therefore, a digital gain greater than 1 (i.e., a gain that substantially amplifies the pixel value) is set for these lines to compensate for the reduction in the amount of exposure. In this case, as shown in FIG. 11(a), the smaller the charge accumulation amount (or the integral amount of light emission intensity) of each line in the boundary image, the larger the gain is set, and the brightness of each pixel can be effectively increased.

[0062] 11 to 13(c) show a pixel group P ADJ An example of this is shown. Processing target pixel group P ADJ is a pixel group included in a predetermined number of lines above and below the adjacent position where the boundary image and the upper image are adjacent, and a predetermined number of lines above and below the adjacent position where the boundary image and the lower image are adjacent. ADJThe pixels included in the matrix are subjected to filtering using the filter coefficients shown in (d) of FIGS. The 5×5 filter coefficients shown in (d) of FIGS. 11 to 13 are filter coefficients when color filters in a Bayer array are arranged in each pixel, and are essentially 3×3 filter coefficients. Preferably, the filter processing unit 52 sets the filter coefficients of the spatial filter applied to the pixels included in each of the predetermined number of lines in accordance with the flash profile. For example, the filter coefficients shown in Fig. 13(d) have a higher weight for the pixel of interest and a lower weight for pixels in the vertical direction compared to the filter coefficients shown in Fig. 11(d) and Fig. 12(d). By optimizing the filter coefficients according to the flash profile, it is possible to perform appropriate smoothing processing for the flash profile.

[0063] In one embodiment, the filter processing unit 52 applies a spatial filter to the pixel values ​​of pixels included in a predetermined number of lines above and below the center position in the line direction of the boundary image, for the composite image of the current frame. ADJ Alternatively, the group of pixels to be processed may be a group of pixels contained in a predetermined number of lines above and below the center position in the line direction of the border image. In this case too, smoothing is performed on the border image and its neighboring images, making horizontal black lines in the border image even less noticeable.

[0064] (ii) Second aspect of post-processing of composite images The second mode of post-processing for the composite image is a combination of the above-mentioned processes Pc1 and Pc3. A system configuration in which this second aspect is implemented is shown in Fig. 14. As shown in Fig. 14, in the second aspect, an amplification processing unit 51 and an interpolation processing unit 53 are further included between the synthesis unit 24 and the image output processing unit 25 in the processor 20 shown in Fig. 1. 15 to 17 are diagrams for explaining image processing when the second aspect of post-processing for a composite image is applied. In each of FIGS. 15 to 17, (a) shows the vertical synchronization signal Vsync of the CMOS image sensor 14, (b) shows the operation of the CMOS image sensor 14 (CMOS operation), (c) shows the flash L, (d) shows the captured image acquired by the CMOS image sensor 14, (e) shows the composite image, (f) shows the corrected image after the digital gain is applied, and (g) shows the corrected image after the frame interpolation process. Note that FIG. 15(c) and FIG. 16(c) show the waveform of vocal cord vibration that is the basis of the timing of emitting the flash. FIG. 15(c) shows the timing of the flash when the vocal cord vibration frequency Fv is relatively high, and FIG. 16(c) shows the timing of the flash when the vocal cord vibration frequency Fv is low (e.g., Fv=63Hz).

[0065] 15, as shown in Fig. 15(a) and (b), the CMOS image sensor 14 starts exposure by a rolling shutter method in units of frames in synchronization with a vertical synchronization signal Vsync. As shown in Fig. 15(d), when the current frame is taken as a reference, a lower image IM_L1 is obtained as an image captured two frames before for the flash L1, an upper image IM_L1a is obtained as an image captured one frame before for the flash L1, a lower image IM_L2 is obtained as an image captured one frame before for the flash L2, and an upper image IM_L2a is obtained as an image captured in the current frame for the flash L2. As shown in Fig. 15(e), the composite image of the current frame is an image obtained by combining the lower image IM_L2 and the upper image IM_L2a, but this composite image has a noticeable horizontal black line in the boundary image.

[0066] Referring to FIG. 16, similarly to FIG. 15, the CMOS image sensor 14 starts exposure in units of frames by the rolling shutter method in synchronization with the vertical synchronization signal Vsync. In FIG. 16, the vocal cord vibration frequency Fv is smaller than that in FIG. 15, and the timing of emitting the flashes L1 and L2 is different from that in FIG. 15. Therefore, as shown in FIG. 16(d), when the current frame is taken as a reference, the lower image IM_L1 is obtained as the captured image three frames before the flash L1, the upper image IM_L1a is obtained as the captured image two frames before the flash L1, the lower image IM_L2 is obtained as the captured image one frame before the flash L2, and the upper image IM_L2a is obtained as the captured image of the current frame for the flash L2. As shown in FIG. 16(e), the composite image of the current frame is an image obtained by combining the lower image IM_L2 and the upper image IM_L2a, but in this composite image, a horizontal black line is noticeable in the boundary image as in FIG. 15.

[0067] Unlike Fig. 15 and Fig. 16, Fig. 17 shows image processing in the case where no flash was emitted in any line of the CMOS image sensor 14 at the exposure start time of one frame before the current frame. In this example, as shown in Fig. 17(d), with the current frame as a reference, a lower image IM_L1 is obtained as an image captured three frames before the flash L1, an upper image IM_L1a is obtained as an image captured two frames before the flash L1, a lower image IM_L2 is obtained as an image captured two frames before the flash L2, and an upper image IM_L2a is obtained as an image captured one frame before the flash L2. As shown in Fig. 17(e), the composite image of the current frame is an image obtained by combining the lower image IM_L2 and the upper image IM_L2a, but in this composite image, a horizontal black line is noticeable in the boundary image as in Fig. 15 and Fig. 16.

[0068] The amplification processing unit 51 shown in FIG. 14 generates a corrected image after applying digital gain (FIGS. 15(f) to 17(f)) by amplifying the pixel values ​​(digital values) of each pixel of each line of the boundary image in the composite image of the current frame generated by the composition unit 24. This increases the luminance of each pixel of each line of the boundary image, making horizontal black lines less noticeable in the boundary image. This is the same as the first aspect in FIG. 9.

[0069] 14 performs an interpolation process between the current frame and a past frame two or three frames back, using as target lines each line of the boundary image in the composite image of the current frame, or each line of a plurality of lines including each line of the boundary image as well as a predetermined number of lines above the adjacent position where the boundary image and the upper image are adjacent and a predetermined number of lines below the adjacent position where the boundary image and the lower image are adjacent. Specifically, the interpolation process 53 reads out the captured images two and three frames back stored in the frame buffer 23, and performs frame interpolation process as follows.

[0070] As shown in Figures 15 and 16, when a flash of light is emitted on any line of the CMOS image sensor 14 at the exposure start time of the frame prior to the current frame, the interpolation processing unit 53 calculates the pixel values ​​of pixels included in each target line in the composite image of the current frame by taking a weighted average of the pixel values ​​of the corresponding pixels in the composite image of the current frame and the pixel values ​​of the corresponding pixels in the captured image two frames prior (the lower image IM_L1 in Figure 15 and the upper image IM_L1a in Figure 16).

[0071] 17, when no flash is emitted in any line of the CMOS image sensor 14 at the exposure start time of the frame before the current frame, the interpolation processing unit 53 calculates the pixel values ​​of the pixels included in each target line in the composite image of the current frame by taking a weighted average of the pixel values ​​of the corresponding pixels in the composite image of the current frame and the pixel values ​​of the corresponding pixels in the captured image three frames before. In the example shown in Fig. 17, the lower image IM_L1, which is the captured image three frames before, is used as the basis for frame interpolation for the composite image of the current frame. As described above, by performing frame interpolation processing, horizontal black lines in boundary images can be made less noticeable, and an awkward feeling can be prevented when viewed as a moving image.

[0072] Next, a specific setting example in the second aspect of post-processing on a composite image will be described with reference to FIGS. In each of Figures 18 to 20, (a) shows the digital gain setting for each pixel of each line of the boundary image, (b) shows the flash profile, and (c) shows the previous frame ratio and current frame ratio in the interpolation process. Note that (a) and (b) in Figures 18 to 20 are the same as (a) and (b) in Figures 11 to 13, respectively. 18 to 20(c), the "current frame ratio" is the weight (or ratio) (%) of the pixel value of a pixel of interest included in a composite image of a current frame on which frame interpolation is based, relative to the total pixel values. The "previous frame ratio" is the weight (or ratio) (%) of the pixel value of the pixel of interest in a captured image of a previous frame on which frame interpolation is based, relative to the total pixel values. Here, the "immediately preceding frame" refers to the frame two frames before the current frame if a flash was emitted on any line of CMOS image sensor 14 at the exposure start time one frame before the current frame, and refers to the frame three frames before the current frame if a flash was not emitted on any line of CMOS image sensor 14 at the exposure start time one frame before the current frame.

[0073] In one embodiment, the interpolation processing unit 53 sets the weight of the weighted average so that the weight for the pixel value of the corresponding pixel of the captured image of the past frame (i.e., two frames or three frames before) is maximum at the center of the line direction of the boundary image, and the weight for the pixel value of the corresponding pixel of the captured image of the past frame decreases as it moves away from the center vertically. For example, Fig. 18(c) shows an example in which the previous frame ratio is set to be maximum (80%) at the center of the line direction of the boundary image, and to decrease as it moves away from the center vertically. The reason for setting the previous frame ratio in this manner is as follows: Since a flash profile generally has characteristics in which the emission intensity gradually rises from the start of emission and gradually falls towards the end of emission, the digital gain that is set is greatest for the line corresponding to the center of the flash profile (i.e., the central line of the border image), for example, as shown in Fig. 18(a), in order to compensate for the decrease in exposure amount at the center where the emission intensity is highest in the flash profile. At that time, since noise is also amplified in the central line of the border image due to the increased digital gain, the previous frame ratio is set to be greater toward the center of the border image in order to reduce the discomfort that this amplified noise gives to the viewer.

[0074] In one embodiment, the interpolation processing unit 53 may set the weights of the weighted average so that the weighting for the pixel values ​​of pixels included in a line in an adjacent position where the boundary image and the upper image are adjacent and / or a line in an adjacent position where the boundary image and the lower image are adjacent is maximum, and the weighting for the pixel values ​​of the corresponding pixels in the captured image of the previous frame (i.e., two frames ago or three frames ago) decreases as the distance from the adjacent position increases vertically.

[0075] In one embodiment, the interpolation processing unit 53 sets the weight of the weighted average applied to the pixels included in the line that is the target of the interpolation processing in accordance with the flash profile. 18 to 20 each show a different flash profile, and the previous frame ratio and current frame ratio shown in (c) of each figure differ depending on this flash profile. By adjusting the ratio according to the flash profile, a more natural looking composite image can be generated.

[0076] As described above, the electronic endoscope system 1 includes the electronic scope 10 having the CMOS image sensor 14 configured to capture an image of a subject by a rolling shutter method, and the processor 20 having the light source device 28 that emits a flash to capture a subject with a strobe. The light source device 28 emits a flash so that the period from the flash end time of one flash to the flash start time of the next flash is longer than a flash prohibition period that prohibits emission of a flash for at least one frame period. The processor 20 processes the captured images obtained by the CMOS image sensor 14 in frame units based on the timing of the emission of the flash by the light source device 28 to generate a composite image, and generates screen data for monitor display based on the composite image. That is, in the electronic endoscope system 1, a flash prohibition period is set in which emission of a flash is prohibited for at least one frame period. The length of the flash emission period, which is the period during which one flash is emitted, can be freely set as long as the length of the flash prohibition period, which is the interval between flashes, is one frame period or longer. Therefore, stroboscopy can be achieved with a sufficient amount of light.

[0077] 3 to 7, the electronic endoscope system 1 allows a flash of light to be emitted on any line of the exposure start time of the CMOS image sensor 14 depending on the timing of the flash of light emission. This makes it possible to suppress a decrease in the frame rate, and eliminates the need for the light source device 28 to emit light in synchronization with a synchronization signal (such as Vsync) of the CMOS image sensor 14. On the other hand, when generating a composite image, a horizontal black line may appear corresponding to a boundary image that is the boundary between the upper image and the lower image, since the upper image and the lower image obtained by the CMOS image sensor 14 are composited. This horizontal black line can be made less noticeable by performing at least one of digital gain application processing, spatial filter processing, and frame interpolation processing on the boundary image. Unlike this embodiment, when the CMOS image sensor is operated using a pseudo-global shutter method in which a common exposure period is set for all lines within one frame period, image synthesis processing is not required, but the readout speed of the CMOS image sensor is inevitably reduced, making it impossible to suppress a decrease in the frame rate.

[0078] In the electronic endoscope system 1 shown in Fig. 1, the light source device 28 is built into the processor 20, which has the advantage of facilitating the process of transmitting a light emission timing signal from the timing control circuit 27 to the synthesis unit 24, but other configurations can also be adopted as shown in Fig. 21 and Fig. 22. In each of Fig. 21 and Fig. 22, the same reference numerals are used to designate the same parts as those included in the system in Fig. 1.

[0079] An electronic endoscope system 1A in Fig. 21 shows an embodiment in which a light source device is built into an electronic scope. As shown in Fig. 21, the electronic endoscope system 1A includes an electronic scope 10A and a processor 20B, and compared to the electronic scope 10 shown in Fig. 1, the electronic scope 10A differs from the electronic scope 10 in that it includes a timing control circuit 27 and a light source device 28. In this configuration, the electronic scope 10A can set the length of the flash prohibition period based on the frame period that it sets itself.

[0080] An electronic endoscope system 1B in Fig. 22 shows an embodiment in which an electronic scope, a processor, and a light source device are separate devices. As shown in Fig. 22, the electronic endoscope system 1B includes an electronic scope 10, a processor 20B, and a light source system 50. In this system, the processor 20B has a CPU 29 that notifies the timing control circuit 27 of the light source system 50 of data related to the length of the light emission prohibition period. The CPU 29 needs to notify the data only once when the electronic endoscope system 1B is started up.

[0081] The imaging system and electronic endoscope system of the present invention have been described in detail above. However, the imaging system and electronic endoscope system of the present invention are not limited to the above-described embodiments, and various improvements and modifications may be made without departing from the spirit and scope of the present invention. [Explanation of symbols]

[0082] 1, 1A, 1B...Electronic endoscope system 10,10A…Electronic scope 11...LCB 12...Light distribution lens 13...Objective lens 14…CMOS image sensor 15…CMOS driver 20, 20A, 20B…Processor 21...System controller 22...Image input processing unit 23...Frame buffer 24…Synthetic section 25...Image output processing section 26...Operation panel 27...Timing control circuit 28...Light source device 29…CPU 30…Monitor 31...Audio processing circuit 32...Frequency detection circuit 40…Microphone 50…Light source system 51...Amplification processing section 52...Filter processing section 53...Interpolation processing unit

Claims

1. An imaging element configured to capture an image of a subject using a rolling shutter method; a light source unit that emits a flash of light to photograph the subject with a flash, the light source unit emitting the flash of light such that a period from a flash end time of one flash to a flash start time of the next flash is longer than a flash prohibition period that prohibits emission of a flash of light for at least one frame period; an image processing unit that processes the captured image obtained by the image sensor on a frame-by-frame basis based on the timing of emitting a flash of light from the light source unit to generate an image for display; Equipped with The image processing unit includes: If a flash is emitted on any line of the image sensor at the exposure start time of the frame one frame before the current frame, Among the display images of the current frame, an upper image corresponding to a line above the flash start line of the previous frame is set as a captured image obtained by the corresponding line of the current frame; Among the display images of the current frame, a lower image corresponding to a line below the flash end line of the previous frame is set as a captured image obtained by the corresponding line of the previous frame; a boundary image from the flash start line to the flash end line of the display image of the current frame is defined as an image obtained by adding together a captured image obtained by the corresponding line of the current frame and a captured image obtained by the corresponding line of the previous frame; thereby generating the display image of the current frame; If no flash was emitted on any line of the image sensor at the exposure start time of the frame preceding the current frame, Among the display images of the current frame, an upper image corresponding to a line above the flash start line of two frames before is set as a captured image obtained by the corresponding line of one frame before; Among the display images of the current frame, a lower image corresponding to a line below the flash end line of two frames before is set as a captured image obtained by the corresponding line of two frames before; Among the display images of the current frame, a boundary image from the flash start line to the flash end line is defined as an image obtained by adding together an image obtained by a corresponding line in the previous frame and an image obtained by a corresponding line in the previous frame. and generating the display image of the current frame by Imaging system.

2. the image processing unit has an amplification processing unit that amplifies a pixel value of each pixel of the display image of a current frame, the amplification processing unit, in accordance with a profile of the emission intensity according to the passage of time of the flash and a readout period of the image sensor, makes a gain applied to pixels of each line of the boundary image in the display image of the current frame larger than a gain applied to pixels of each line other than the boundary image. The imaging system according to claim 1 .

3. the image processing unit has a filter processing unit which performs a spatial filter on pixel values ​​of pixels included in a predetermined number of lines above and below an adjacent position where the boundary image and the upper image are adjacent and a predetermined number of lines above and below an adjacent position where the boundary image and the lower image are adjacent, for the display image of the current frame; 3. An imaging system according to claim 1 or 2.

4. the image processing unit has a filter processing unit that applies a spatial filter to pixel values ​​of pixels included in a predetermined number of lines above and below a center position in a line direction of the boundary image, for the display image of the current frame; 3. An imaging system according to claim 1 or 2.

5. the filter processing unit sets a filter coefficient of a spatial filter applied to pixels included in each of the predetermined number of lines in accordance with a profile of a light emission intensity according to the passage of time of the flash; 4. The imaging system according to claim 3.

6. the image processing unit has an interpolation processing unit that performs an interpolation process between the current frame and a previous frame two frames or three frames earlier, with each line of the boundary image in the display image of the current frame, or each line of a plurality of lines including, in addition to each line of the boundary image, a predetermined number of lines above an adjacent position where the boundary image and the upper image are adjacent and a predetermined number of lines below an adjacent position where the boundary image and the lower image are adjacent as target lines; The interpolation processing unit: when a flash is emitted in any one of the lines of the image pickup element at an exposure start time of one frame before the current frame, a pixel value of a pixel included in each target line in the image for display of the current frame is calculated by taking a weighted average of a pixel value of a corresponding pixel in the image for display of the current frame and a pixel value of a corresponding pixel in the captured image two frames before; if no flash is emitted in any line of the imaging element at the exposure start time of the frame preceding the current frame, the pixel values ​​of pixels included in each target line of the image for display of the current frame are calculated by taking a weighted average of the pixel values ​​of the corresponding pixels in the image for display of the current frame and the pixel values ​​of the corresponding pixels in the captured image three frames preceding; 3. An imaging system according to claim 1 or 2.

7. the interpolation processing unit sets weights of the weighted average such that a weight for a pixel value of a corresponding pixel in a captured image of a past frame is maximum at a center in a line direction of the boundary image, and a weight for a pixel value of a corresponding pixel in a captured image of a past frame decreases as the pixel moves away from the center in the vertical direction.

7. The imaging system according to claim 6.

8. the interpolation processing unit sets weights for the weighted average so that a weight for a pixel value of a pixel included in a line at an adjacent position where the boundary image and the upper image are adjacent and / or a line at an adjacent position where the boundary image and the lower image are adjacent is maximized, and a weight for a pixel value of a corresponding pixel of an image captured in a previous frame decreases with increasing distance from the adjacent position in the vertical direction.

7. The imaging system according to claim 6.

9. the interpolation processing unit sets weights of the weighted average applied to pixels included in the target line in accordance with a profile of a light emission intensity according to the passage of time of the flash; 7. The imaging system according to claim 6.

10. A microphone and a voice detection unit that detects a voice frequency from a voice signal acquired from the microphone; The imaging system according to claim 1 or 2, The light source unit emits the flash of light in a cycle synchronized with the audio frequency detected by the audio detection unit. Electronic endoscope system.

Images