Image pickup system and electronic endoscope system

Abstract

One embodiment of the present invention is an electronic endoscope system (1) provided with a CMOS image sensor (14) that captures an image of a subject in a rolling shutter manner, and a light source device (28) that emits flash light in order to carry out stroboscopic imaging on the subject. In this system, the light source device (28) emits the flash light in such a manner that the period from the flash light end time of a certain flash light to the flash light start time of the next flash light is longer than a flash light prohibition period in which the emission of the flash light is prohibited, which lasts for at least one frame period. The electronic endoscope system (1) processes captured images obtained by the CMOS image sensor (14) in units of frames according to the light emission timing of the flash light of the light source device (28), and generates a display image.

Description

Technical Field

[0001] The present invention relates to a dynamic mirror inspection using a rolling shutter type imaging element. Background Art

[0002] In the prior art, there is known an electronic endoscope system for performing dynamic laryngoscopy.

[0003] For example, in the electronic endoscope system described in Japanese Patent Gazette No. 6196105, the pixel signal is continuously read from the CMOS image sensor by the rolling shutter method, and the LED is strobed in accordance with the subject's voice. The pixel signal of the row read after the stroboscopic (flash) irradiation in the frame that the CMOS image sensor is reading during the stroboscopic (flash) irradiation is stored in the frame buffer as the lower image, and in the next frame, the pixel signal of the row above the lower image of the previous frame is directly output to the image signal processing unit, and then the lower image of the previous frame stored in the frame buffer is output to the image signal processing unit. As a result, in stroboscopic photography using a rolling shutter, it is not necessary to set the period for stroboscopic exposure, and an image of a complete screen shot by the same stroboscopic light can be obtained. Summary of the invention

[0004] Technical problem to be solved by the invention

[0005] However, there is a problem in the electronic endoscope system that the irradiation period (flash irradiation period) of the stroboscopic light (flash) must be shorter than the read time of one line of the CMOS image sensor. If the flash irradiation period is shorter than the read time of one line of the CMOS image sensor, the light intensity of the light source such as LED must be increased to obtain sufficient brightness to observe the vocal cords, which will have negative effects such as the need to strengthen the heat dissipation measures of the light source and the decrease in the durability of the light source. In addition, if the gain of the CMOS image sensor is increased without increasing the light source intensity, the S / N ratio will decrease.

[0006] Therefore, an object of the present invention is to increase the degree of freedom in setting the length of a flash irradiation period when a subject is imaged by irradiating the subject with flash using a rolling shutter method.

[0007] Technical solutions to solve problems

[0008] One embodiment of the present disclosure is a camera system having:

[0009] An imaging element configured to capture an image of a subject using a rolling shutter method;

[0010] a light source unit that emits a flash for stroboscopic photography of the subject, the light source unit being configured to emit the flash in such a manner that a period from a flash end time of a certain flash to a flash start time of a next flash is longer than a flash prohibition period during which the emission of the flash is prohibited for at least one frame period;

[0011] An image processing unit processes the captured image in units of frames obtained by the image pickup element according to the light emission timing of the flash of the light source unit to generate an image for display.

[0012] When a flash is emitted from any row of the image sensor at the exposure start time of one frame before the current frame, the image processing unit performs the following operations:

[0013] The upper image corresponding to the row above the flash start row one frame ago in the display image of the current frame is used as the captured image obtained from the corresponding row of the current frame,

[0014] The lower image corresponding to the row below the flash end row one frame ago in the display image of the current frame is used as the captured image obtained from the corresponding row one frame ago,

[0015] The display image of the current frame is generated by adding a camera image obtained from a corresponding row of the current frame and a camera image obtained from a corresponding row one frame ago as the boundary image between the flash start row and the flash end row in the display image of the current frame.

[0016] When no flash is emitted in any row of the image sensor at the exposure start time one frame before the current frame, the image processing unit performs the following operations:

[0017] The upper image corresponding to the row above the flash start row two frames ago in the display image of the current frame is used as the captured image obtained from the corresponding row one frame ago,

[0018] The lower image corresponding to the row below the flash end row two frames ago in the display image of the current frame is used as the captured image obtained from the corresponding row two frames ago.

[0019] The boundary image between the flash start row and the flash end row in the display image of the current frame is generated as an image obtained by adding a camera image obtained from a corresponding row one frame ago and a camera image obtained from a corresponding row two frames ago.

[0020] In the imaging system, the image processing section includes an amplification processing section for amplifying a pixel value of each pixel of the display image of a current frame.

[0021] The amplification processing unit can increase the gain applicable to each row of pixels of the boundary image in the display image of the current frame based on the luminous intensity curve of the flash changing with time and the reading period of the camera element, so that it is greater than the gain applicable to each row of pixels outside the boundary image.

[0022] The image processing unit may also have a filtering processing unit, which takes the display image of the current frame as an object and performs spatial filtering on the pixel values ​​of pixels contained in a predetermined number of rows above and below the adjacent position where the boundary image is adjacent to the upper image, and a predetermined number of rows above and below the adjacent position where the boundary image is adjacent to the lower image.

[0023] The image processing unit may further include a filter processing unit that performs spatial filtering on pixel values ​​of pixels included in a predetermined number of lines above and below a center position in a line direction of the boundary image, with the display image of a current frame as an object.

[0024] The filter processing unit may further set a filter coefficient of a spatial filter applicable to pixels included in each of the predetermined number of rows according to a luminous intensity curve of the flash that changes with time.

[0025] The image processing unit may also include an interpolation processing unit, which performs interpolation processing between the current frame and a past frame two or three frames ago, using as object rows each row of the boundary image in the display image of the current frame, or each row of the boundary image plus a plurality of rows including a predetermined number of rows above an adjacent position where the boundary image is adjacent to the upper image and a predetermined number of rows below an adjacent position where the boundary image is adjacent to the lower image.

[0026] At this time, the interpolation processing unit performs the following operations:

[0027] When a flash is emitted in any row of the imaging element at the exposure start time one frame before the current frame, the pixel values ​​of the pixels included in each object row in the display image of the current frame are calculated by weighted averaging the pixel values ​​of the corresponding pixels of the display image of the current frame and the pixel values ​​of the corresponding pixels of the imaging image two frames before.

[0028] When no flash is emitted in any row of the imaging element at the start of exposure one frame before the current frame, the pixel values ​​of the pixels contained in each object row in the display image of the current frame are calculated by taking a weighted average of the pixel values ​​of the corresponding pixels of the display image of the current frame and the pixel values ​​of the corresponding pixels of the imaging image three frames ago.

[0029] The interpolation processing unit can also set the weight of the weighted average so that the weight corresponding to the pixel value of the corresponding pixel of the camera image of the past frame is the largest in the center of the row direction of the boundary image, and the weight of the pixel value of the corresponding pixel of the camera image of the past frame decreases as it moves away from the center.

[0030] The interpolation processing unit can also set the weight of the weighted average so that the weight of the pixel values ​​contained in the row at the adjacent position where the boundary image is adjacent to the upper image, and / or the row at the adjacent position where the boundary image is adjacent to the lower image is the largest, and the weight of the pixel value of the corresponding pixel of the camera image of the past frame decreases as it moves away from the adjacent position upward or downward.

[0031] The interpolation processing unit may further set the weight of the weighted average applied to the pixels included in the target row according to a luminous intensity curve of the flash that changes with time.

[0032] One embodiment of the present disclosure is an electronic endoscope system having:

[0033] microphone;

[0034] a sound detection unit, configured to detect a sound frequency from a sound signal obtained by the microphone;

[0035] And the camera system according to any one of claims 1 to 9.

[0036] The light source unit emits the flash at a period synchronized with the frequency of the sound detected by the sound detection unit.

[0037] Effects of the Invention

[0038] According to the imaging system and the electronic endoscope system, when the subject is imaged by irradiating the subject with flash light using the rolling shutter method, the degree of freedom in setting the length of the flash irradiation period can be increased. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a block diagram showing an example of the configuration of an electronic endoscope system according to an embodiment.

[0040] Figure 2 This is a diagram showing the relationship between the operation of the CMOS image sensor and the light emission timing of the flash.

[0041] Figure 3 This is an operation timing chart showing the generation of a synthetic image when flash light is emitted at a vocal cord vibration frequency of 1000 Hz in an electronic endoscope system according to one embodiment.

[0042] Figure 4This is an operation timing chart showing the generation of a synthetic image when flash light is emitted at a vocal cord vibration frequency of 250 Hz in an electronic endoscope system according to one embodiment.

[0043] Figure 5 This is an operation timing chart showing the generation of a synthetic image when flash light is emitted at a vocal cord vibration frequency of 125 Hz in an electronic endoscope system according to one embodiment.

[0044] Figure 6 This is an operation timing chart showing the generation of a synthetic image when flash light is emitted at a vocal cord vibration frequency of 115 Hz in an electronic endoscope system according to one embodiment.

[0045] Figure 7 This is an operation timing chart showing the generation of a synthetic image when flash light is emitted at a vocal cord vibration frequency of 63 Hz in an electronic endoscope system according to one embodiment.

[0046] Figure 8 This is a diagram showing the relationship between the vocal cord vibration frequency and the frame rate in an electronic endoscope system according to one embodiment.

[0047] Fig. 9 Yes Figure 1 The illustrated system is an explanatory diagram of an embodiment in which an amplification processing unit and a filtering processing unit are added.

[0048] Fig.10 is Fig. 9 This is a diagram for explaining an example of a method for improving the quality of a synthetic image in the electronic endoscope system according to the embodiment shown.

[0049] Fig.11 FIG. 1 is a diagram showing an example of spatial filtering processing applied to a synthesized image.

[0050] Fig.12 FIG. 1 is a diagram showing an example of spatial filtering processing applied to a synthesized image.

[0051] Fig.13 FIG. 1 is a diagram showing an example of spatial filtering processing applied to a synthesized image.

[0052] Fig.14 Yes Figure 1 The illustrated system is an explanatory diagram of an embodiment in which an enlargement processing unit and an interpolation processing unit are added.

[0053] Fig.15 is Fig.14 This is a diagram for explaining an example of a method for improving the quality of a synthetic image in the electronic endoscope system according to the embodiment shown.

[0054] Fig.16 is Fig.14This is a diagram for explaining an example of a method for improving the quality of a synthetic image in the electronic endoscope system according to the embodiment shown.

[0055] Fig.17 is Fig.14 This is a diagram for explaining an example of a method for improving the quality of a synthetic image in the electronic endoscope system according to the embodiment shown.

[0056] Fig.18 is a diagram showing an example of frame interpolation processing applied to a synthesized image.

[0057] Fig.19 is a diagram showing an example of frame interpolation processing applied to a synthesized image.

[0058] Fig. 20 is a diagram showing an example of frame interpolation processing applied to a synthesized image.

[0059] Fig.21 It means a Figure 1 A block diagram showing a configuration example of an electronic endoscope system according to a different embodiment.

[0060] Fig. 22 It means a Figure 1 A block diagram showing a configuration example of an electronic endoscope system according to a different embodiment. DETAILED DESCRIPTION

[0061] Next, an electronic endoscope system as one embodiment of the imaging system of the present invention will be described in detail with reference to the drawings.

[0062] Figure 1 1 is a block diagram showing a configuration example of an electronic endoscope system 1 according to an embodiment. The electronic endoscope system 1 is a system specifically used for medical treatment, and is particularly suitable for dynamic laryngoscopy. In dynamic laryngoscopy, an operator uses a light source device that can intermittently emit flashes and an electronic mirror to perform stroboscopic photography and display images, thereby observing the vocal cords of a subject.

[0063] like Figure 1 As shown, an electronic endoscope system 1 according to one embodiment includes an electronic scope (endoscope) 10, a processor 20 having a built-in light source device, a monitor 30, and a microphone 40. The processor 20 is connected to the monitor 30 and the microphone 40.

[0064] The processor 20 includes a sound processing circuit 31 and a frequency detection circuit 32 .

[0065] When performing stroboscopic photography of the vocal cords, a flash is intermittently lit (emitted) according to the vocal cord vibration frequency generated by the sound emitted by the patient. The sound emitted by the patient is collected by the microphone 40. The sound processing circuit 31 can eliminate noise and the like in the sound collected by the microphone 40, and make the sound waveform suitable for detecting the vocal cord vibration frequency. The frequency detection circuit 32 detects the vocal cord vibration frequency from the sound waveform obtained by the sound processing circuit 31.

[0066] The processor 20 includes a system controller 21 , an operation panel 26 , a timing control circuit 27 , and a light source device 28 (an example of a light source unit).

[0067] The system controller 21 executes various programs to comprehensively control the entire electronic endoscope system 1. The system controller 21 changes each action of the electronic endoscope system 1 and the parameters used for each action according to the instructions of the operator (observer) input to the operation panel 26. The system controller 21 supplies clock pulses for adjusting the timing of the actions of each part to each circuit in the electronic endoscope system 1.

[0068] The timing control circuit 27, under the control of the system controller 21, determines the light emission timing of the flash according to the vocal cord vibration frequency detected by the frequency detection circuit 32, and sequentially sends a signal (light emission timing signal) indicating the determined light emission timing of the flash to the light source device 28. In addition, the timing control circuit 27 receives data related to the length of the light emission prohibition period from the system controller 21 when the system is started, and determines the light emission timing under the condition of maintaining the length of the light emission prohibition period. The timing control circuit 27 sequentially sends the light emission timing signal to the synthesis unit 24 in real time.

[0069] The flash prohibition period refers to the shortest period from the flash end time of a certain flash to the flash start time of the next flash. That is, from the flash end time of a certain flash to the end of the flash prohibition period, the next flash is prohibited from being emitted. The flash prohibition period is at least one frame period (1 frame cycle), or a longer period. The reason why the flash prohibition period is set to at least one frame period is to avoid exposing the same row of the CMOS image sensor 14 more than twice within one frame period.

[0070] The light source device 28 emits a flash L for illuminating a subject such as a vocal cord in synchronization with a light emission timing signal sent from the timing control circuit 27. The flash L may be white light, pseudo-white light, or special light of a specific wavelength band. According to one embodiment, the flash L emitted from the light source device 28 is focused by a focusing lens onto the incident end face of the LCB (Light Carrying Bundle) 11 and is incident into the LCB 11.

[0071] The light source of the light source device 28 is not limited to a specific type, and may be, for example, an LED, a laser diode, a high-intensity lamp (such as a xenon lamp, a metal halide lamp, a mercury lamp, or a halogen lamp), etc. In the following description, the light source of the light source device 28 is an LED as an example.

[0072] The flash light L incident on the LCB 11 propagates in the LCB 11. The flash light L propagating in the LCB 11 is emitted from the emission end surface of the LCB 11 disposed at the front end of the electronic scope 10, and irradiates the subject via the light distribution lens 12. The return light from the subject illuminated by the flash light L from the light distribution lens 12 forms an optical image on the light receiving surface of the CMOS image sensor 14 via the objective lens 13.

[0073] The CMOS image sensor 14 is an example of an imaging element, and is configured to capture an image of a subject by a rolling shutter method. The CMOS image sensor 14 has, for example, a Bayer-type pixel configuration, and accumulates an optical image formed in each pixel on the light-receiving surface as an electric charge corresponding to the amount of light, generates and outputs image signals of R (red), G (green), and B (blue). In addition, a CCD (Charge-Coupled Device) image sensor or other types of imaging devices may be used instead of the CMOS image sensor 14. The CMOS image sensor 14 may also be equipped with a complementary color filter.

[0074] The CMOS driver 15 provided in the electronic mirror 10 drives and controls the CMOS image sensor 14 at a timing synchronized with the frame rate of the image processed by the processor 20 according to the clock pulse supplied by the system controller 21. The CMOS driver 15 performs predetermined processing on the captured image input from the CMOS image sensor 14 and outputs the image to the image input processing unit 22 of the processor 20, wherein the predetermined processing includes A / D conversion.

[0075] The CMOS image sensor 14 inputs an imaging signal for imaging a subject into the CMOS driver 15 at a predetermined frame cycle.

[0076] The frame period (one frame period) is, for example, 1 / 120 second, 1 / 60 second, or 1 / 30 second. Next, the case where the frame period is 1 / 60 second is described as an example. That is, the length of the flash prohibition period is at least 1 / 60 second.

[0077] The processor 20 includes an image input processing unit 22 , a frame buffer 23 , a synthesis unit 24 , and an image output processing unit 25 , and generates an image to be displayed on the monitor 30 .

[0078] The image input processing unit 22 performs predetermined signal processing on the captured images in frames sent from the CMOS driver 15 , such as noise reduction processing, demosaic processing, matrix operation, etc., and sends the captured images in frames (the captured images of the current frame) to the frame buffer 23 and the synthesis unit 24 .

[0079] The frame buffer 23 is a memory for buffering the camera images sent by the image input processing unit 22 in units of frames. In one embodiment, the frame buffer 23 temporarily stores the camera images of the frame one frame before the current frame, the frame two frames before the current frame, and the frame three frames before the current frame. In addition, when the frame interpolation processing described below is not performed, the frame buffer 23 only temporarily stores the camera images of the frame one frame before the current frame and the frame two frames before the current frame.

[0080] The synthesis unit 24 synthesizes the captured images of a plurality of frames stored in the frame buffer 23 to generate a synthesized image according to the emission timing indicated by the emission timing signal received from the timing control circuit 27. A specific example of generating a synthesized image will be described in detail later.

[0081] The image output processing unit 25 processes the synthesized image generated by the synthesizing unit 24 in units of frames, generates screen data for monitor display, and converts the generated screen data for monitor display into a specific video format signal. The converted video format signal is output to the monitor 30. As a result, the image of the subject (vocal cords) is displayed on the display screen of the monitor 30.

[0082] The frame buffer 23 , the synthesis section 24 , and the image output processing section 25 constitute an example of an image processing section of the present invention.

[0083] Next, refer to Figure 2 The operation of the CMOS image sensor 14 based on the flash light will be described.

[0084] exist Figure 2 In the figure, (a) shows the operation of the CMOS image sensor 14 during two consecutive frames (1 / 60 sec+1 / 60 sec), (b) shows the flash L, and (c) shows a part of (a) in an enlarged manner.

[0085] In the CMOS image sensor 14, in the effective pixel region except the invalid pixel region, a time difference is set for each of a plurality of rows, and exposure in each frame is started. In addition, the invalid pixel region is composed of one or more rows.

[0086] like Figure 2As shown in (a) of FIG. 1 , in the CMOS image sensor 14, the time difference is set in the order of the rows LN1, LN2, ..., LNn (the last row of the effective pixel area) included in the effective pixel area excluding the invalid pixel area (the order from top to bottom), and the exposure of the Nth frame is started. Figure 2 As shown in the enlarged view of (c), each frame period includes a charge accumulation period T INT and signal reading period T RO The charge accumulation period is used to accumulate charges in each pixel area of ​​the effective pixel area from the beginning of exposure, and the signal reading period is used to read the signal corresponding to the accumulated charge after the charge accumulation during one frame is completed. RO , charge accumulation is not possible.

[0087] like Figure 2 As shown in (a) and (b), the flash L is the light emitted in a short time from the flash start time Ts to the flash end time Te, which is used to accumulate charge for a part of the row group of the effective pixel area in the Nth frame and a part of the row group of the effective pixel area in the N+1th frame. The period from the flash start time Ts to the flash end time Te is the flash irradiation period. Figure 2 In the enlarged view of (c), the row corresponding to the flash start time Ts of the flash L is defined as the flash start row Ls, and the row corresponding to the flash end time Te of the flash L is defined as the flash end row Le.

[0088] For each line from line LN1 to the flash start line Ls, the charge generated by the flash L is accumulated during the charge accumulation period T of the N+1th frame. INT On the other hand, for each row from the flash end row Le to the last row LNn of the effective pixel area, the charge generated by the flash L is accumulated during the charge accumulation period T of the Nth frame. INT For each line from the flash start line Ls to the flash end line Le, the charge generated by a part of the flash L is accumulated during the charge accumulation period T of the Nth frame. INT , and the charge generated by the remaining part of the flash L is accumulated during the charge accumulation period T of the N+1th frame. INT .

[0089] from Figure 2 As can be seen from the figure, a display image obtained by the flash L can be obtained based on a composite image obtained by appropriately synthesizing the following upper image, lower image, and boundary image.

[0090] The upper image obtained during the N+1th frame by each line from line LN1 to the flash start line Ls

[0091] The lower image obtained during the Nth frame by each row from the flash end row Le to the last row LNn of the effective pixel area

[0092] The boundary image between the upper image and the lower image obtained by each line from the flash start line Ls to the flash end line Le during the Nth frame and the N+1th frame

[0093] In the electronic endoscope system 1 of the present embodiment, as long as the light emission prohibition period is ensured, the light emission timing of the flash L can be set arbitrarily, and the flash irradiation period can also be set arbitrarily. Therefore, in a certain frame, during the charge accumulation period T of any row of the effective pixel area of ​​the CMOS image sensor 14, INT At the start time of the flash L, there is a possibility that no flash L is generated. In consideration of this situation, more specifically, the synthesis unit 24 of the processor 20 generates a synthesized image as a basis of the display image in the following processing Pa and processing Pb according to different situations.

[0094] [Processing Pa] When any row of the CMOS image sensor 14 emits a flash at the exposure start time one frame before the current frame, the synthesis unit 24 generates a synthetic image of the current frame as follows:

[0095] (a-1) An upper image corresponding to a line above the flash start line one frame ago in the composite image of the current frame is used as a captured image obtained from the corresponding line of the current frame.

[0096] (a-2) The lower image corresponding to the row below the flash end row one frame ago in the composite image of the current frame is regarded as the captured image obtained from the corresponding row one frame ago.

[0097] (a-3) The boundary image from the flash start line to the flash end line in the composite image of the current frame is set to an image obtained by adding the captured image obtained from the corresponding line of the current frame and the captured image obtained from the corresponding line one frame ago.

[0098] [Process Pb] At the exposure start time one frame before the current frame, when no flash is emitted from any row of the CMOS image sensor 14, the synthesis unit 24 generates a synthetic image of the current frame in the following manner:

[0099] (b-1) The upper image corresponding to the row above the flash start row two frames ago in the composite image of the current frame is regarded as the captured image obtained from the corresponding row one frame ago.

[0100] (b-2) The lower image corresponding to the row below the flash end row two frames ago in the composite image of the current frame is regarded as the captured image obtained from the corresponding row two frames ago.

[0101] (b-3) The boundary image from the flash start line to the flash end line in the composite image of the current frame is taken as an image obtained by adding the image obtained from the corresponding line one frame ago and the image obtained from the corresponding line two frames ago.

[0102] Next, refer to Figure 3 to Figure 7 The timing diagrams of FIG. 1 illustrate the specific operation of generating a composite image according to the processing Pa and processing Pb in the electronic endoscope system 1 when the vocal cord vibration frequencies obtained from the microphone 40 are different. Figure 3 to Figure 7 In the description, a case where the light emission prohibition period is set to approximately 1 frame period (1 / 60 second) is taken as an example.

[0103] exist Figure 3 to Figure 7 In each figure, the (a) operation of the CMOS image sensor 14 (CMOS operation), (b) the light emission timing of the flash, (c) the captured image obtained by the CMOS image sensor 14, and (d) the synthesized image synthesized by the synthesizing unit 24 are respectively shown in units of the Mth frame, the M+1th frame, ..., etc. Figure 3 to Figure 7 In (a), the vertical line indicates the light emission period of the flash shown in (b). Figure 3 to Figure 7 The waveform of the vocal cord vibration is also shown in (b). The lighting timing of the flashes L1, L2, L3, ... is adjusted according to the vocal cord vibration frequency Fv so as to satisfy the conditions of the lighting prohibition period.

[0104] (I) When the vocal cord vibration frequency Fv = 1000Hz ( Figure 3 )

[0105] exist Figure 3 In the figure, since the flash L1 is emitted during the Mth frame, the lower image IM_L1 obtained by the flash L1 can be obtained as the camera image of the Mth frame, and the upper image IM_L1a obtained by the flash L1 can be obtained as the camera image of the M+1th frame. Similarly, the lower image IM_L2 obtained by the flash L2 can be obtained as the camera image of the M+1th frame, and the upper image IM_L2a obtained by the flash L2 can be obtained as the camera image of the M+2th frame. The lower image IM_L3 obtained by the flash L3 can be obtained as the camera image of the M+2th frame, and the upper image IM_L3a obtained by the flash L3 can be obtained as the camera image of the M+3th frame.

[0106] exist Figure 3In the figure, since the flash L4 is emitted during the M+4th frame, as the camera image of the M+4th frame, the lower image IM_L4 obtained by the flash L4 can be obtained, and as the camera image of the M+5th frame, the upper image IM_L4a obtained by the flash L4 can be obtained. Similarly, as the camera image of the M+5th frame, the lower image IM_L5 obtained by the flash L5 can be obtained, and as the camera image of the M+6th frame, the upper image IM_L5a obtained by the flash L5 can be obtained.

[0107] When creating the composite images of the M+1th to M+3th frames, since a flash is emitted at any row of the CMOS image sensor 14 at the exposure start time of the previous frame, the composite images are generated as follows according to the process Pa.

[0108] For example, when generating a composite image of the M+1th frame as the current frame, the upper image corresponding to the row above the flash start row of the Mth frame in the composite image is used as the upper image IM_L1a obtained from the corresponding row of the M+1th frame (current frame), and the lower image corresponding to the row below the flash end row of the Mth frame in the composite image is used as the lower image IM_L1 obtained from the corresponding row of the Mth frame (one frame ago). Figure 3 Although not visible in the image, the boundary image from the flash start row to the flash end row in the composite image of the M+1th frame is taken as an image obtained by adding the camera image obtained by the corresponding row of the M+1th frame (current frame) and the camera image obtained by the corresponding row one frame ago.

[0109] The same applies to the case where a synthetic image is generated for each of the M+2th and M+3th frames as the current frame.

[0110] When creating the composite image of the M+4th frame as the current frame, since no flash was emitted in any row of the CMOS image sensor 14 at the exposure start time of the M+3th frame one frame ago, the composite image is generated as follows.

[0111] The upper image of the composite image of the M+4th frame, which corresponds to the row above the flash start row of the M+2th frame two frames ago, is taken as the camera image IM_L3a obtained from the corresponding row of the M+3th frame one frame ago, and the lower image of the composite image, which corresponds to the row below the flash end row of the M+2th frame two frames ago, is taken as the camera image IM_L3 obtained from the corresponding row of the M+2th frame two frames ago. Figure 3 Although not visible in the image, the boundary image from the flash start row to the flash end row of the composite image of the M+4th frame is taken as an image obtained by adding the camera image obtained from the corresponding row of the M+3th frame one frame ago and the camera image obtained from the corresponding row of the M+2th frame two frames ago.

[0112] In short, when creating the composite image of the M+4th frame, since the camera image IM_L4a is not obtained, it is impossible to create a composite image based on the camera images IM_L4 and IM_L4a. Figure 3 As shown, the synthesized image of the M+3th frame will be reused (ie, it is the same as the synthesized image of the M+3th frame).

[0113] For each of the M+5th frame and the M+6th frame as the current frame, a synthetic image is generated according to the process Pa, in the same way as when creating the synthetic image of each of the M+1th to M+3th frames.

[0114] (II) When the vocal cord vibration frequency Fv = 250Hz ( Figure 4 )

[0115] like Figure 4 As shown in (b), when the vocal cord vibration frequency Fv is 250 Hz, the period from the end of the light emission prohibition period after a certain flash to the emission of the next flash is longer than when the vocal cord vibration frequency Fv is 1000 Hz.

[0116] exist Figure 4 In the figure, since the flash L1 is emitted during the Mth frame, the lower image IM_L1 obtained by the flash L1 can be obtained as the camera image of the Mth frame, and the upper image IM_L1a obtained by the flash L1 can be obtained as the camera image of the M+1th frame. Similarly, the lower image IM_L2 obtained by the flash L2 can be obtained as the camera image of the M+1th frame, and the upper image IM_L2a obtained by the flash L2 can be obtained as the camera image of the M+2th frame. The lower image IM_L3 obtained by the flash L3 can be obtained as the camera image of the M+2th frame, and the upper image IM_L3a obtained by the flash L3 can be obtained as the camera image of the M+3th frame.

[0117] exist Figure 4 In the figure, the flash L4 is emitted during the M+4th frame, and as the captured image of the M+4th frame, an image IM_L4 of the entire effective pixel area obtained by the flash L4 can be obtained.

[0118] When creating a composite image of each of the M+1 to M+3 frames as the current frame, since a flash is emitted at any row of the CMOS image sensor 14 at the exposure start time of the previous frame of each frame, a composite image is generated according to the above-mentioned process Pa. The generation process of this composite image is the same as the case of (I) creating a composite image of each of the M+1 to M+3 frames when the vocal cord vibration frequency Fv=1000 Hz.

[0119] For the M+4th frame, which is the current frame, although a composite image is generated according to the processing Pa, since a flash is emitted in the invalid pixel area of ​​the CMOS image sensor 14 at the start of exposure of the M+3th frame one frame ago, no substantial synthesis is performed, and the composite image of the M+4th frame is the camera image IM_L4.

[0120] For the M+5th frame as the current frame, since no flash is emitted at any row of the CMOS image sensor 14 at the exposure start time equivalent to the M+4th frame one frame ago, a synthetic image is generated according to the processing Pb. Therefore, the synthetic image of the M+5th frame is the same as the synthetic image of the M+4th frame.

[0121] (III) When the vocal cord vibration frequency Fv = 125Hz ( Figure 5 )

[0122] like Figure 5 As shown in (b), when the vocal cord vibration frequency Fv is 125 Hz, the period from the end of the light emission prohibition period after a certain flash to the emission of the next flash is longer than when the vocal cord vibration frequency Fv is 250 Hz.

[0123] exist Figure 5 In the figure, since the flash L1 is emitted during the Mth frame, the lower image IM_L1 obtained by the flash L1 can be obtained as the camera image of the Mth frame, and the upper image IM_L1a obtained by the flash L1 can be obtained as the camera image of the M+1th frame. Similarly, the lower image IM_L2 obtained by the flash L2 can be obtained as the camera image of the M+1th frame, and the upper image IM_L2a obtained by the flash L2 can be obtained as the camera image of the M+2th frame. The lower image IM_L3 obtained by the flash L3 can be obtained as the camera image of the M+3th frame, and the upper image IM_L3a obtained by the flash L3 can be obtained as the camera image of the M+4th frame. The lower image IM_L4 obtained by the flash L4 can be obtained as the camera image of the M+4th frame.

[0124] When creating the composite image of each of the M+1 to M+2 frames as the current frame, since a flash is emitted at any row of the CMOS image sensor 14 at the exposure start time of the previous frame of each frame, the composite image is generated according to the above-mentioned process Pa. The generation process of this composite image is the same as the case of (I) creating the composite image of each of the M+1 to M+3 frames when the vocal cord vibration frequency Fv=1000 Hz.

[0125] For the M+3th frame as the current frame, since no flash is emitted at any row of the CMOS image sensor 14 at the exposure start time equivalent to the M+2th frame one frame ago, a synthetic image is generated according to the above-mentioned process Pb. Therefore, the synthetic image of the M+3th frame is the same as the synthetic image of the M+2th frame.

[0126] When creating the composite image of each of the M+4th to M+5th frames as the current frame, since a flash is emitted at any row of the CMOS image sensor 14 at the exposure start time of the previous frame of each frame, the composite image is generated according to the above-described process Pa. That is, the composite image of each of the M+4th to M+5th frames is created in the same manner as the composite image of each of the M+1th to M+2th frames.

[0127] (IV) When the vocal cord vibration frequency Fv = 115Hz ( Figure 6 )

[0128] like Figure 6 As shown in (b), when the vocal cord vibration frequency Fv is 115 Hz, the period from the end of the light emission prohibition period after a certain flash to the next flash emission becomes very short.

[0129] exist Figure 6 In the figure, since the flash L1 is emitted during the Mth frame, the lower image IM_L1 obtained by the flash L1 can be obtained as the camera image of the Mth frame, and the upper image IM_L1a obtained by the flash L1 can be obtained as the camera image of the M+1th frame. Similarly, the lower image IM_L2 obtained by the flash L2 can be obtained as the camera image of the M+1th frame, and the upper image IM_L2a obtained by the flash L2 can be obtained as the camera image of the M+2th frame. The lower image IM_L3 obtained by the flash L3 can be obtained as the camera image of the M+2th frame, and the upper image IM_L3a obtained by the flash L3 can be obtained as the camera image of the M+3th frame. The lower image IM_L4 obtained by the flash L4 can be obtained as the camera image of the M+3th frame, and the upper image IM_L4a obtained by the flash L4 can be obtained as the camera image of the M+4th frame. As the captured image of the M+4th frame, a lower image IM_L5 obtained by the flash L5 can be obtained.

[0130] When creating a composite image of each of the M+1 to M+5 frames as the current frame, since a flash is emitted at any row of the CMOS image sensor 14 at the exposure start time of the previous frame of each frame, a composite image is generated according to the above-mentioned process Pa. The generation process of this composite image is the same as the case of (I) creating a composite image of each of the M+1 to M+3 frames in the case of the vocal cord vibration frequency Fv=1000 Hz.

[0131] (V) When the vocal cord vibration frequency Fv = 63Hz ( Figure 7 )

[0132] like Figure 7 As shown in (b), when the vocal cord vibration frequency Fv is 63 Hz, the period from the end of the light emission prohibition period after a certain flash to the next flash emission becomes very long.

[0133] exist Figure 7 In the figure, since the flash L1 is emitted during the Mth frame, the lower image IM_L1 obtained by the flash L1 can be obtained as the camera image of the Mth frame, and the upper image IM_L1a obtained by the flash L1 can be obtained as the camera image of the M+1th frame. Similarly, the lower image IM_L2 obtained by the flash L2 can be obtained as the camera image of the M+2th frame, and the upper image IM_L2a obtained by the flash L2 can be obtained as the camera image of the M+3th frame. The lower image IM_L3 obtained by the flash L3 can be obtained as the camera image of the M+4th frame, and the upper image IM_L3a obtained by the flash L3 can be obtained as the camera image of the M+5th frame.

[0134] When creating a composite image of the M+1th frame as the current frame, since a flash is emitted at any row of the CMOS image sensor 14 at the exposure start timing of the frame before this frame (Mth frame), a composite image is generated according to the process Pa.

[0135] When creating the composite image of the M+2th frame as the current frame, since no flash was emitted in any row of the CMOS image sensor 14 at the exposure start time of the previous frame (M+1th frame), the composite image is generated according to the processing Pb.

[0136] When creating a composite image of the M+3th frame as the current frame, since a flash is emitted at any row of the CMOS image sensor 14 at the exposure start timing of the frame before this frame (M+2th frame), a composite image is generated according to the process Pa.

[0137] When creating the composite image of the M+4th frame as the current frame, since no flash was emitted at any row of the CMOS image sensor 14 at the exposure start time of the previous frame (M+3th frame), the composite image is generated according to the process Pb.

[0138] When creating a composite image of the M+5th frame as the current frame, since a flash is emitted at any row of the CMOS image sensor 14 at the exposure start timing of the frame before this frame (the M+4th frame), a composite image is generated according to the process Pa.

[0139] Therefore, if Figure 7As shown, a composite image of each frame is generated thereby.

[0140] Reference Figure 3 to Figure 7 Specifically, in the electronic endoscope system 1 , the light emission timing of the flashes is determined so that the light emission interval of the continuous flashes is longer than a predetermined light emission prohibition period and is synchronized with the vocal cord vibration frequency.

[0141] At this time, when the exposure of the CMOS image sensor 14 operating in a rolling shutter mode starts, when no flash is emitted from any row of the CMOS image sensor 14, the composite images of two consecutive frames are the same, and the frame rate is low. Figure 6 As shown in FIG. 1 , when the vocal cord vibration frequency Fv is 115 Hz, for any frame, a flash is emitted from any row of the CMOS image sensor 14 at the start of exposure, so the synthetic images of each frame are different from each other, and the frame rate is high (about 60 fps). Figure 7 As shown, when the vocal cord vibration frequency Fv is 63 Hz, no flash is emitted in any row of the CMOS image sensor 14 at the start of exposure. This situation occurs every other frame, so the composite images of two consecutive frames are the same, and the frame rate is low (about 30fps).

[0142] exist Figure 8 , which shows the relationship between the vocal cord vibration frequency Fv and the frame rate of the synthesized image.

[0143] In the electronic endoscope system 1 of the present embodiment, the frame rate fluctuates discontinuously according to the vocal cord vibration frequency Fv and the flash emission timing, wherein the flash emission timing is set according to a predetermined light emission prohibition period. This is because the period from the end of the light emission prohibition period after a certain flash to the next flash emission varies discontinuously with the vocal cord vibration frequency.

[0144] Reference Figure 8 It can be seen that for any vocal cord vibration frequency, at least 30fps is ensured. As the vocal cord vibration frequency increases, the frame rate gradually approaches 60fps, and a high frame rate can be obtained. If the total length of the flash irradiation period and the flash prohibition period is consistent with the period corresponding to the vocal cord vibration frequency Fv, the frame rate becomes higher. Therefore, if Figure 8 As shown, the frame rate periodically becomes higher relative to the vocal cord vibration frequency Fv.

[0145] In addition, if Figure 3 to Figure 7 As shown, Figure 8 The results are shown when the light emission prohibition period is set to about 1 frame period (1 / 60 second). If the light emission prohibition period is further extended, the results are similar to those of FIG. Figure 8 different.

[0146] Next, a description will be given of a preferred post-processing performed on the composite image obtained through the processes Pa and Pb.

[0147] As described above, in the process Pa, the boundary image from the flash start line to the flash end line in the composite image of the current frame is taken as an image obtained by adding the camera image obtained from the corresponding line of the current frame and the camera image obtained from the corresponding line one frame ago. On the other hand, in the process Pb, the boundary image from the flash start line to the flash end line in the composite image of the current frame is taken as an image obtained by adding the camera image obtained from the corresponding line one frame ago and the camera image obtained from the corresponding line two frames ago.

[0148] The boundary image in the composite image is the area between the upper image and the lower image of the composite image. Figure 2 As shown in the figure, the flash irradiation period (Ts to Te) of the flash L and the signal reading period T RO The image obtained during the at least partially repeated and underexposed flash L. Therefore, if no processing is performed on this boundary image, such as Figure 3 to Figure 7 As shown in (d), the black line highlighted in the composite image is horizontal, which is visually recognized by the observer as a vertically moving black line.

[0149] Therefore, in order to prevent the black line in the horizontal direction of the boundary image from being too conspicuous in the synthesized image, it is preferable to perform the following image processing on the boundary image and the image of the adjacent lines as post-processing on the synthesized image.

[0150] In one embodiment, the post-processing of the synthesized image is any one of the following three processes Pc1 to Pc3 or a combination of two or more processes.

[0151] [Processing Pc1] Digital gain imparting processing

[0152] The digital gain imparting process is a process of amplifying the pixel value of each pixel in each row of the boundary image. By imparting the digital gain, the brightness of the boundary image obtained in a state where the flash exposure is insufficient can be corrected.

[0153] [Processing Pc2] Spatial filtering processing

[0154] The spatial filtering process is a process for performing spatial filtering on the pixel values ​​of the boundary image or the boundary image and a predetermined number of rows of pixels in the vicinity thereof contained in the synthesized image of the current frame to achieve the purpose of smoothing (blurring) the image.

[0155] [Processing Pc3] Frame interpolation processing

[0156] The frame interpolation process refers to a process of interpolating between the current frame and a past frame two or three frames ago, using a boundary image included in the current frame, or a boundary image and a predetermined number of behaviors in the vicinity thereof as objects.

[0157] Although only the process Pc1 may be performed, it is preferable to use the processes Pc1 and Pc2 in combination, or to use the processes Pc1 and Pc3 in combination, so that the horizontal black line of the boundary image in the synthesized image can be made less noticeable more effectively.

[0158] (i) The first method of post-processing the composite image

[0159] The first method of post-processing for the synthesized image is a combined process of the process Pc1 and the process Pc2.

[0160] The system structure for implementing the first method is as follows: Fig. 9 As shown. Fig. 9 As shown, in the first method, Figure 1 The processor 20 shown in the figure further includes an enlargement processing section 51 and a filter processing section 52 between the synthesis section 24 and the image output processing section 25 .

[0161] Fig.10 This is an explanatory diagram of image processing when the first method of post-processing of a composite image is applied. Fig.10 In the figure, (a) represents the vertical synchronization signal Vsync of the CMOS image sensor 14, (b) represents the operation of the CMOS image sensor 14 (CMOS operation), (c) represents the flash L, (d) represents the captured image obtained by the CMOS image sensor 14, (e) represents the synthesized image, (f) represents the corrected image after digital gain is given, and (g) represents the corrected image after spatial filtering processing.

[0162] refer to Fig.10 ,like Fig.10 As shown in (a) and (b) of FIG10 , the CMOS image sensor 14 starts exposure in units of frames by means of a rolling shutter in synchronization with the vertical synchronization signal Vsync. Fig.10 As shown in (d), with the current frame as the reference, the flash L emitted one frame ago can obtain the camera image one frame ago, that is, the lower image IM_L, and obtain the camera image of the current frame, that is, the upper image IM_La. Fig.10 As shown in (e), the synthesized image of the current frame is an image synthesized by the lower image IM_L and the upper image IM_La. However, in this synthesized image, the black line in the horizontal direction of the boundary image is very obvious.

[0163] Fig. 9The enlargement processing unit 51 shown generates a corrected image (a digital value) after digital gain is given) by enlarging the pixel value (digital value) of each pixel in each row of the boundary image in the synthesized image of the current frame generated by the synthesizing unit 24. Fig.10 As a result, the brightness of each pixel in each row of the boundary image increases, and the black line in the horizontal direction of the boundary image becomes less noticeable.

[0164] Fig. 9 The filter processing unit 52 shown in FIG. 1 performs spatial filtering on the pixel values ​​of the pixels included in a predetermined number of rows above and below the adjacent position where the boundary image is adjacent to the upper image IM_La, and a predetermined number of rows above and below the adjacent position where the boundary image is adjacent to the lower image IM_L, with the synthetic image of the current frame as the object, to generate a corrected image after the spatial filtering process ( Fig.10 In this way, the boundary image and the images near it are smoothed, and the black lines in the horizontal direction of the boundary image are less obvious.

[0165] The spatial filter used here may be any known spatial filter, as long as it is a filter that can perform smoothing between the boundary image and the upper image IM_La, and between the boundary image and the lower image IM_L, such as a Gaussian filter or an average filter.

[0166] Next, refer to Figure 11 to Figure 13 , a specific example of setting the spatial filter in the first method of post-processing the synthetic image is described.

[0167] exist Figure 11 to Figure 13 In each figure, (a) represents the setting of the digital gain of each pixel in each row of the boundary image, (b) represents the flash curve, (c) represents the pixel to be processed by the spatial filter, and (d) represents the filter coefficient (kernel) of the spatial filter. Figure 2 (c) Similarly, the charge accumulation period T is shown for a plurality of rows (row X to row X+9) near the flash start row and the flash end row. IN T and signal reading period T RO ,At the same time, the flash curve shown in (b) of each figure after compression in the vertical direction is also shown.

[0168] The flash curve refers to a curve of the luminous intensity of the flash that changes with time, in other words, it refers to the luminous characteristics of the flash when the horizontal axis is time and the vertical axis is the luminous intensity of the flash.

[0169] exist Figure 2 In (b), an ideal pulse-shaped flash curve is shown for the sake of convenience, but the actual flash curve may vary depending on the magnitude of the luminous intensity. For example, Figure 11 to Figure 13(b) is the flash of the same LED, which is based on Fig.11 (b) Fig.12 (b) Fig.13 The flash curve when the light intensity is reduced (i.e., the LED current is reduced) in the order of (b). In this way, if the light intensity of the LED is changed, the flash curve generated by the LED (especially at the rising and falling edges) changes according to the changes in the characteristics of the drive circuit and power supply circuit in the light source device 28.

[0170] In addition, refer to Figure 11 to Figure 13 (a) Based on the flash curve and the signal reading period T during which charge accumulation is not possible RO The exposure of each row of the boundary image changes depending on the degree of repetition.

[0171] Therefore, preferably, the amplification processing unit 51 makes the gain applied to the pixels of each row of the boundary image in the synthesized image of the current frame greater than the gain applied to the pixels of each row other than the boundary image according to the flash curve and the signal reading period of the CMOS image sensor 14.

[0172] For example, in Fig.11 In the example of setting the digital gain of (a), in the Xth row and the X+1th row, due to the flash curve and the signal reading period T RO There is almost no repetition, so the digital gain is set to "1.0", but in the X+3th row, the signal reading period T RO The peak part of the flash is equivalent to the signal reading period T RO , and there is no exposure in any of the previous and next frames, so the digital gain is set to "1.2". In the X+2th to X+7th rows, since there is an unexposed part in any of the previous and next frames in the flash curve, if the boundary image is not enlarged, the horizontal black line in the boundary image will be more obvious. Therefore, the reduction in exposure is compensated by setting a digital gain greater than 1 for these rows (that is, a gain that actually amplifies the pixel value). At this time, if Fig.11 As shown in (a), the smaller the charge accumulation amount (or the integrated amount of luminous intensity) of each line of the boundary image is, the larger the gain can be set, thereby effectively increasing the brightness of each pixel.

[0173] Figure 11 to Figure 13 (c) shows the pixel group P to be processed by the filtering process ADJ .

[0174] Pixel group to be processed P ADJThe pixel group P is a pixel group included in a predetermined number of rows above and below the adjacent position where the boundary image is adjacent to the upper image, and a predetermined number of rows above and below the adjacent position where the boundary image is adjacent to the lower image. ADJ Each pixel contained in is respectively processed as follows Figure 11 to Figure 13 The filtering process of the filter coefficients shown in (d).

[0175] Figure 11 to Figure 13 The 5×5 filter coefficients shown in (d) are filter coefficients when a color filter in a Bayer arrangement is arranged in each pixel, and are substantially 3×3 filter coefficients.

[0176] Preferably, the filter processing unit 52 sets filter coefficients of a spatial filter applicable to pixels included in each of a predetermined number of rows according to the flicker curve.

[0177] For example, Fig.13 The filter coefficients shown in (d) are Fig.11 (d) and Fig.12 Compared with the filter coefficients shown in (d), the weight of the pixel of interest is large, and the weight of the pixels in the vertical direction is low. By optimizing the filter coefficients according to the flash curve, the flash curve can be appropriately smoothed.

[0178] In one embodiment, the filter processing unit 52 performs spatial filtering on the pixel values ​​of pixels included in a predetermined number of rows from the center position in the row direction of the boundary image to the top and bottom, with the synthetic image of the current frame as the object. Figure 11 to Figure 13 The pixel group P to be processed shown in (c) ADJ Differently, the pixel group to be processed can also be a pixel group contained in a predetermined number of rows from the central position upward and downward in the row direction of the boundary image. In this case, the boundary image and the image near it can also be smoothed, thereby further making the black lines in the horizontal direction in the boundary image less obvious.

[0179] (ii) Second method of post-processing the composite image

[0180] The second method of post-processing the composite image is a combined process of the process Pc1 and the process Pc3.

[0181] The system structure for implementing the second method is as follows: Fig.14 As shown. Fig.14 As shown, in the second method, Figure 1 The processor 20 shown in the figure further includes an enlargement processing section 51 and an interpolation processing section 53 between the synthesis section 24 and the image output processing section 25 .

[0182] Figure 15 to Figure 17Each of them is an explanatory diagram of image processing when the second method of post-processing of the synthesized image is applied. Figure 15 to Figure 17 In each figure, (a) represents the vertical synchronization signal Vsync of the CMOS image sensor 14, (b) represents the operation of the CMOS image sensor 14 (CMOS operation), (c) represents the flash L, (d) represents the captured image obtained by the CMOS image sensor 14, (e) represents the synthesized image, (f) represents the corrected image after the digital gain is given, and (g) represents the corrected image after the frame interpolation processing. In addition, Fig.15 (c) and Fig.16 (c) shows the vocal cord vibration waveform which is the basis of the light emission timing of the flash. Fig.15 (c) shows the flash timing when the vocal cord vibration frequency Fv is high, while Fig.16 (c) shows the flash timing when the vocal cord vibration frequency Fv is low (for example, Fv = 63 Hz).

[0183] refer to Fig.15 ,like Fig.15 As shown in (a) and (b), the CMOS image sensor 14 starts exposure in units of frames through a rolling shutter method in synchronization with the vertical synchronization signal Vsync. Fig.15 As shown in (d), taking the current frame as a reference, as the camera image two frames ago, the lower image IM_L1 obtained under the flash L1 can be obtained, as the camera image one frame ago, the upper image IM_L1a obtained under the flash L1 can be obtained, as the camera image one frame ago, the lower image IM_L2 obtained under the flash L2 can be obtained, and as the camera image of the current frame, the upper image IM_L2a obtained under the flash L2 can be obtained. Fig.15 As shown in (e), the synthesized image of the current frame is an image synthesized by the lower image IM_L2 and the upper image IM_L2a. However, in this synthesized image, the black line in the horizontal direction of the boundary image is very obvious.

[0184] refer to Fig.16 ,and Fig.15 Similarly, the CMOS image sensor 14 starts to expose in units of frames through a rolling shutter method in synchronization with the vertical synchronization signal Vsync. Fig.16 The vocal cord vibration frequency Fv is Fig.15 Small, medium, the timing of flash L1 and L2 is also the same Fig.15 Therefore, if Fig.16As shown in (d), taking the current frame as a reference, as the camera image three frames ago, the lower image IM_L1 obtained under the flash L1 can be obtained, as the camera image two frames ago, the upper image IM_L1a obtained under the flash L1 can be obtained, as the camera image one frame ago, the lower image IM_L2 obtained under the flash L2 can be obtained, and as the camera image of the current frame, the upper image IM_L2a obtained under the flash L2 can be obtained. Fig.16 As shown in (e), the composite image of the current frame is an image composited by the lower image IM_L2 and the upper image IM_L2a. However, in this composite image, Fig.15 The same is true for the case in Figure 1, where the horizontal black lines in the boundary image are very obvious.

[0185] Fig.17 and Fig.15 and Fig.16 The difference is that the image processing is performed when no flash is emitted in any row of the CMOS image sensor 14 at the exposure start time of one frame before the current frame. Fig.17 As shown in (d), taking the current frame as a reference, as the camera image three frames ago, the lower image IM_L1 obtained under the flash L1 can be obtained, as the camera image two frames ago, the upper image IM_L1a obtained under the flash L1 can be obtained, as the camera image two frames ago, the lower image IM_L2 obtained under the flash L2 can be obtained, as the camera image one frame ago, the upper image IM_L2a obtained under the flash L2 can be obtained. Fig.17 As shown in (e), the composite image of the current frame is an image composited by the lower image IM_L2 and the upper image IM_L2a. However, in this composite image, Fig.15 and Fig.16 The same is true for the case in Figure 1, where the horizontal black lines in the boundary image are very obvious.

[0186] Fig.14 The enlargement processing unit 51 shown generates a corrected image (a digital value) after digital gain is given) by enlarging the pixel value (digital value) of each pixel of each line of the boundary image in the synthesized image of the current frame generated by the synthesizing unit 24. Fig.15 (f)~ Fig.17 As a result, the brightness of each pixel in each row of the boundary image increases, and the horizontal black line in the boundary image becomes less noticeable. Fig. 9 Same as the first way.

[0187] Fig.14The interpolation processing unit 53 shown performs interpolation processing between the current frame and the past frame two or three frames ago, with each line of the boundary image in the synthetic image of the current frame, or each line of the boundary image and each line of a plurality of lines including a predetermined number of lines above the adjacent position where the boundary image is adjacent to the upper image and a predetermined number of lines below the adjacent position where the boundary image is adjacent to the lower image as the target line. Specifically, the interpolation processing unit 53 reads the captured images two or three frames ago stored in the frame buffer 23, and performs frame interpolation processing in the following manner.

[0188] like Fig.15 and Fig.16 As shown, the interpolation processing unit 53 performs interpolation processing on the pixel value of the corresponding pixel of the composite image of the current frame and the captured image two frames ago (at the beginning of the exposure of the current frame) when a flash is emitted at any row of the CMOS image sensor 14 at the exposure start time of one frame before the current frame. Fig.15 The middle is the lower image IM_L1, Fig.16 The pixel values ​​of the pixels included in each target line in the synthesized image of the current frame are calculated by performing weighted averaging on the pixel values ​​of the corresponding pixels in the upper image IM_L1a (in the figure).

[0189] like Fig.17 As shown, the interpolation processing unit 53 calculates the pixel values ​​of the pixels included in each object row in the composite image of the current frame by weighted averaging the pixel values ​​of the corresponding pixels of the composite image of the current frame and the pixel values ​​of the corresponding pixels of the captured image three frames ago when no flash is emitted in any row of the CMOS image sensor 14 at the exposure start time one frame before the current frame. Fig.17 In the example shown, the composite image of the current frame uses the captured image three frames ago, that is, the lower image IM_L1, as a basis for frame interpolation.

[0190] As described above, by performing the frame interpolation process, the black line in the horizontal direction in the boundary image can be made less noticeable and no sense of discomfort is generated when the image is viewed as a moving image.

[0191] Next, refer to Figures 18 to 20 , a specific setting example in the second method of post-processing of the composite image is described.

[0192] exist Figures 18 to 20 In each figure, (a) shows the setting of the digital gain of each pixel in each row of the boundary image, (b) shows the flash curve, and (c) shows the previous frame ratio and the current frame ratio in the interpolation process. Figures 18 to 20 (a) and (b) are respectively Figure 11 to Figure 13 (a) and (b) are the same.

[0193] exist Figures 18 to 20In (c), the "current frame ratio" is the weight (or ratio) (%) of the pixel value of the pixel of interest contained in the composite image of the current frame as the basis for frame interpolation relative to the overall pixel value. The "previous frame ratio" is the weight (or ratio) (%) of the pixel value of the pixel of interest in the previous frame of the camera image as the basis for frame interpolation relative to the overall pixel value.

[0194] Among them, the "previous frame" refers to the frame two frames before the current frame when a flash is emitted in any row of the CMOS image sensor 14 at the start of exposure one frame before the current frame, and the frame three frames before the current frame when no flash is emitted in any row of the CMOS image sensor 14 at the start of exposure one frame before the current frame.

[0195] In one embodiment, the interpolation processing unit 53 sets the weight of the weighted average so that the weight of the pixel value of the corresponding pixel of the camera image of the past frame (i.e., two frames ago or three frames ago) is the largest in the center of the row direction of the boundary image, and the weight of the pixel value of the corresponding pixel of the camera image of the past frame decreases as it moves away from the center. Fig.18 (c) shows an example in which the previous frame ratio is the largest (80%) at the center of the boundary image in the row direction, and the previous frame ratio decreases as it moves away from the center to the top and bottom.

[0196] The reason for setting the previous frame ratio in this way is as follows: generally speaking, the flash curve has a characteristic that the luminous intensity gradually increases from the beginning of the flash and gradually decreases at the end of the flash. Therefore, the set digital gain should compensate for the decrease in exposure at the center of the flash curve where the luminous intensity is the highest, for example Fig.18 As shown in (a), the row corresponding to the center of the flash curve (i.e., the row in the center of the boundary image) is the largest. At this time, since the digital gain is increased in the row in the center of the boundary image, its noise is also amplified. In order to reduce the discomfort caused by noise amplification to the observer, it can be set so that the closer to the center of the boundary image, the larger the previous frame ratio.

[0197] In one embodiment, the interpolation processing unit 53 can also set the weight of the weighted average so that the weight of the pixel values ​​contained in the row of adjacent positions where the boundary image is adjacent to the upper image, and / or the row of adjacent positions where the boundary image is adjacent to the lower image is the largest, and as the distance from the adjacent position is upward or downward, the weight of the pixel value of the corresponding pixel of the camera image in the past frame (i.e., two frames ago or three frames ago) decreases.

[0198] In one embodiment, the interpolation processing unit 53 sets the weight of the weighted average applied to the pixels included in the target line of the interpolation processing according to the flash curve.

[0199] Figures 18 to 20Different flash curves are shown, and according to the flash curves, the previous frame ratio and the current frame ratio shown in (c) in each figure are different. By adjusting the ratio according to the flash curve, a more natural composite image can be generated.

[0200] As described above, the electronic endoscope system 1 includes an electronic scope 10 having a CMOS image sensor 14 configured to capture an image of a subject in a rolling shutter manner, and a processor 20 having a light source device 28 that emits a flash for stroboscopic photography of the subject. The light source device 28 emits the flash in such a manner that the period from the flash end time of a certain flash to the flash start time of the next flash is longer than the flash prohibition period during which the flash emission is prohibited during a period lasting at least one frame. The processor 20 processes the captured image obtained by the CMOS image sensor 14 in frames according to the light emission timing of the flash of the light source device 28 and generates a composite image, and generates screen data for monitor display based on the composite image.

[0201] That is, in the electronic endoscope system 1, a flash prohibition period is set to prohibit the emission of flash light during at least one frame. Moreover, as a flash irradiation period, the length of the flash irradiation period can be freely set as long as the interval between flashes, that is, the length of the flash prohibition period is longer than one frame period. Therefore, dynamic mirror inspection can be achieved with sufficient light quantity.

[0202] In the electronic endoscope system 1, as Figure 3 to Figure 7 As shown, according to the flash emission timing, the flash is allowed to be emitted at any row at the exposure start time of the CMOS image sensor 14. Therefore, not only can the frame rate be suppressed from being reduced, but also the light source device 28 does not need to be synchronized with the synchronization signal (Vsync, etc.) of the CMOS image sensor 14 to emit light.

[0203] On the other hand, when generating a composite image, since it is necessary to composite the upper image and the lower image acquired by the CMOS image sensor 14, a black line in the horizontal direction corresponding to the boundary between the upper image and the lower image, that is, the boundary image, may be generated. The black line in the horizontal direction can be made less noticeable by performing at least one of digital gain imparting processing, spatial filtering processing, and frame interpolation processing on the boundary image.

[0204] Furthermore, unlike the present embodiment, if the CMOS image sensor is operated using a pseudo-global shutter method in which a common exposure time is set for all rows during one frame, image synthesis processing is not required. However, this causes the reading speed of the CMOS image sensor to decrease, and thus the decrease in frame rate cannot be suppressed.

[0205] exist Figure 1In the electronic endoscope system 1 shown in FIG. 1 , since the light source device 28 is built into the processor 20, it is easy to process the light emission timing signal from the timing control circuit 27 to the synthesis unit 24. However, Fig.21 and Fig. 22 As shown, other structures can also be used. Fig.21 and Fig. 22 In each figure, Figure 1 The same parts included in the system are marked with the same symbols.

[0206] Fig.21 The electronic endoscope system 1A shows an embodiment in which the light source device is built into the electronic endoscope. Fig.21 As shown in FIG. 1 , the electronic endoscope system 1A includes an electronic scope 10A and a processor 20B. Figure 1 The electronic scope 10A shown in the figure is different from the electronic scope 10 in that it includes a timing control circuit 27 and a light source device 28. In this configuration, the electronic scope 10A can set the length of the flash prohibition period based on the frame period set by itself.

[0207] Fig. 22 The electronic endoscope system 1B shows an embodiment in which the electronic endoscope, the processor, and the light source device are respectively independent devices. Fig. 22 As shown, the electronic endoscope system 1B includes an electronic scope 10, a processor 20B, and a light source system 50. In this system, the processor 20B has a CPU 29 for notifying the timing control circuit 27 of the light source system 50 of data related to the length of the light emission prohibition period. The CPU 29 only needs to perform the data notification once when the electronic endoscope system 1B is started.

[0208] The imaging system and the electronic endoscope system of the present invention have been described in detail above, but the imaging system and the electronic endoscope system of the present invention are not limited to the above-described embodiments, and various improvements or changes can be made without departing from the scope of the present invention.

[0209] The present invention relates to a patent application of Japanese Patent Application No. 2023-4384 filed with the Japan Patent Office on January 16, 2023, the entire contents of which are incorporated into this specification by reference.

Claims

1. A camera system, It has: An imaging element configured to capture an image of a subject using a rolling shutter method; a light source unit that emits a flash for stroboscopic photography of the subject, the light source unit being configured to emit the flash in such a manner that a period from a flash end time of a certain flash to a flash start time of a next flash is longer than a flash prohibition period during which emission of the flash is prohibited during a period lasting at least one frame; an image processing unit that processes the captured image in frames obtained by the image pickup element according to the light emission timing of the flash of the light source unit to generate an image for display; The image processing unit performs the following operations: When a flash is emitted from any row of the image pickup element at the exposure start time of one frame before the current frame, The upper image corresponding to the row above the flash start row one frame ago in the display image of the current frame is used as the captured image obtained from the corresponding row of the current frame, The lower image corresponding to the row below the flash end row one frame ago in the display image of the current frame is used as the captured image obtained from the corresponding row one frame ago, The boundary image between the flash start line and the flash end line in the display image of the current frame is taken as an image obtained by adding the camera image obtained from the corresponding line of the current frame and the camera image obtained from the corresponding line of the previous frame, Thereby generating the display image of the current frame; When no flash is emitted from any row of the image sensor at the exposure start time one frame before the current frame, The upper image corresponding to the row above the flash start row two frames ago in the display image of the current frame is used as the captured image obtained from the corresponding row one frame ago, The lower image corresponding to the row below the flash end row two frames ago in the display image of the current frame is used as the captured image obtained from the corresponding row two frames ago. The display image of the current frame is generated by adding a camera image obtained from a corresponding row one frame ago and a camera image obtained from a corresponding row two frames ago using a boundary image between the flash start row and the flash end row in the display image of the current frame as an image obtained by adding a camera image obtained from a corresponding row two frames ago.

2. The camera system according to claim 1, wherein: The image processing unit includes an amplification processing unit for amplifying the pixel value of each pixel of the display image of the current frame. The amplification processing unit increases the gain applicable to each row of pixels of the boundary image in the display image of the current frame based on the luminous intensity curve of the flash changing with time and the reading period of the camera element, making it greater than the gain applicable to each row of pixels outside the boundary image.

3. The imaging system according to claim 1 or 2, wherein: The image processing unit includes a filtering processing unit, which is used to perform spatial filtering on the pixel values ​​of pixels contained in a predetermined number of rows above and below the adjacent position where the boundary image is adjacent to the upper image, and a predetermined number of rows above and below the adjacent position where the boundary image is adjacent to the lower image, taking the display image of the current frame as the object.

4. The imaging system according to claim 1 or 2, wherein: The image processing unit includes a filter processing unit for performing spatial filtering on pixel values ​​of pixels included in a predetermined number of lines above and below a center position in a line direction of the boundary image, with the display image of a current frame as an object.

5. The camera system according to claim 3, wherein: The filter processing unit sets a filter coefficient of a spatial filter applied to pixels included in each of the predetermined number of rows according to a light emission intensity curve of the flash that changes over time.

6. The imaging system according to claim 1 or 2, wherein: The image processing unit includes an interpolation processing unit for performing interpolation processing between the current frame and a past frame two or three frames ago, using each line of the boundary image in the display image of the current frame, or each line of the boundary image plus a plurality of lines including a predetermined number of lines above the adjacent position where the boundary image is adjacent to the upper image and a predetermined number of lines below the adjacent position where the boundary image is adjacent to the lower image as object lines; The interpolation processing unit may also perform the following operations: When a flash is emitted in any row of the imaging element at the exposure start time one frame before the current frame, the pixel values ​​of the pixels included in each target row in the display image of the current frame are calculated by weighted averaging the pixel values ​​of the corresponding pixels of the display image of the current frame and the pixel values ​​of the corresponding pixels of the imaging image two frames before; When no flash is emitted in any row of the imaging element at the start of exposure one frame before the current frame, the pixel values ​​of the pixels contained in each object row in the display image of the current frame are calculated by taking a weighted average of the pixel values ​​of the corresponding pixels of the display image of the current frame and the pixel values ​​of the corresponding pixels of the imaging image three frames ago.

7. The camera system according to claim 6, wherein: The interpolation processing unit sets the weight of the weighted average so that the weight corresponding to the pixel value of the corresponding pixel of the camera image of the past frame is the largest in the center of the row direction of the boundary image, and the weight of the pixel value of the corresponding pixel of the camera image of the past frame decreases as it moves away from the center.

8. The camera system according to claim 6, wherein: The interpolation processing unit sets the weight of the weighted average so that the weight of the pixel values ​​of the pixels contained in the rows at the adjacent positions where the boundary image and the upper image are adjacent, and / or the rows at the adjacent positions where the boundary image and the lower image are adjacent is the largest, and the weight of the pixel values ​​of the corresponding pixels of the camera image of the past frame decreases as the distance from the adjacent positions is upward or downward.

9. The camera system according to claim 6, wherein: The interpolation processing section sets the weight of the weighted average applied to the pixels included in the target row according to the luminous intensity curve of the flash that changes with time.

10. An electronic endoscope system comprising: microphone, a sound detection unit for detecting a sound frequency from a sound signal obtained by the microphone, And the camera system according to any one of claims 1 to 9; The light source section emits the flash light at a cycle synchronized with a frequency of the sound detected by the sound detection section.

Images