Image processing device and image processing method

Abstract

To provide an image processing device and a method thereof.SOLUTION: An image processing device includes: a first light source; a first imaging part for exposing light reflected from a subject to first light from the first light source and generating a first image signal; a first image processing part for generating a first image on the basis of the first image signal; a second light source; a second imaging part for exposing fluorescent light generated from the subject to second light from the second light source and generating a second image signal; a second image processing part for generating a second image on the basis of the second image signal; a superimposition image generation part for generating a superimposition image by superimposing the first image and the second image; and a control part for adjusting the brightness of the first image and the brightness of the second image by adjusting a first control parameter for controlling at least one of the first light source, the first imaging part, and the first image processing part, and a second control parameter for controlling at least one of the second light source, the second imaging part, and the second image processing part on the basis of brightness information of the superimposition image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an image processing device and an image processing method. [Background technology]

[0002] A medical image processing device is known that acquires two types of captured images from an observation object and superimposes corresponding pixels to generate a superimposed image for observation. One of the two types of captured images is an image (referred to as a white image) obtained by irradiating the observation object with white light and capturing the light reflected from the observation object using an imaging element. The other captured image is an image (referred to as an infrared image) obtained by irradiating the observation object with excitation light such as near-infrared light and capturing the fluorescence excited by the excitation light from phosphors contained in the observation object using an imaging element.

[0003] In a method for observing an object based on such a superimposed image, Patent Document 1 below describes a method for achieving a synthesis balance between a white image and an infrared image by changing the light intensity of a white light source (white light source) and a near-infrared light source (near-infrared light source) or the gain of image processing. However, in this method, the light intensity of the near-infrared light source or the gain of image processing is excessively increased before a phosphor is injected into the object. This causes a problem of the infrared image becoming too bright, increasing noise in the superimposed image.

[0004] Patent Document 1 also describes a method for adjusting the brightness of near-infrared light emitted from a near-infrared light source in accordance with the brightness of a white image. With this method, the light intensity of the near-infrared light source or the gain of the image processing is not excessively increased before the phosphor is injected, so the infrared image does not become too bright. However, if the amount of excited fluorescence increases, the brightness of the infrared image is not properly reflected in the superimposed image, resulting in a problem of the superimposed image becoming excessively bright. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent No. 5184016 Summary of the Invention [Problem to be solved by the invention]

[0006] The present disclosure is intended to solve the above-mentioned problems, and provides an image processing device and an image processing method that enable multiple images obtained by capturing images using multiple light sources to be superimposed in an optimal balance. [Means for solving the problem]

[0007] An image processing device according to the present disclosure includes a first light source that irradiates a subject with first light having a first wavelength band, a first imaging unit that exposes light reflected from the subject to the first light to generate a first image signal, a first image processing unit that performs first image processing based on the first image signal to generate a first image, a second light source that irradiates the subject with second light having a second wavelength band different from the first wavelength band and including an excitation wavelength of a fluorescent substance contained in the subject, and a second imaging unit that exposes fluorescence generated from the subject to the second light to generate a second image signal, The image processing device includes a second image processing unit that performs second image processing based on a second image signal to generate a second image, a superimposed image generating unit that generates a superimposed image by superimposing the first image and the second image, and a control unit that adjusts the luminance of the first image and the luminance of the second image by adjusting a first control parameter that controls at least one of the first light source, the first imaging unit, and the first image processing unit, and a second control parameter that controls at least one of the second light source, the second imaging unit, and the second image processing unit, based on luminance information of the superimposed image. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram showing a configuration of an endoscope apparatus according to a first embodiment of the present disclosure. [Figure 2] FIG. 2 is a diagram showing the internal configuration of an insertion member and a camera head. [Figure 3]FIG. 2 is a block diagram showing the configuration of a camera head and a control device. [Figure 4] FIG. 3 is a block diagram of a configuration for performing brightness control in a control unit. [Figure 5] 4 is a flowchart showing the operation of the control device. [Figure 6] 10A and 10B are diagrams showing examples of display of superimposed images with high sensitivity to white light; [Figure 7] FIG. 10 is a diagram showing an example of a superimposed image generated by a related technique. [Figure 8] FIG. 10 is a diagram showing a display example of a superimposed image in which the proportion of fluorescent light brightness is high. [Figure 9] FIG. 10 is a diagram showing an example of a superimposed image generated by a related technique. [Figure 10] 10A and 10B are diagrams showing examples of a target image and a peripheral image, and an example of calculating a weighted average value of brightness between the target image and a part of the peripheral image. [Figure 11] 10A and 10B are diagrams showing an image of a masked region and an image of a non-masked region in a superimposed image. [Figure 12] FIG. 10 is a diagram showing an image in which the luminance value is saturated in a superimposed image. [Figure 13] FIG. 10 is a diagram showing an example of calculating luminance information by increasing the weight for a region with high luminance. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In the drawings, identical or corresponding elements are designated by the same reference numerals, and detailed descriptions thereof will be omitted where appropriate.

[0010] (First embodiment) 1 is a diagram showing the configuration of an endoscopic device 100 according to a first embodiment of the present disclosure. The endoscopic device 100 is used in the medical field and is an apparatus for capturing or observing the inside of a living body (observation target) that is a subject. The endoscopic device 100 includes a light source device 10, an insertion member (endoscope body) 20, a light guide 30, a camera head 50, a display device 70, a transmission cable 80, and a control device 90.

[0011] The light source device 10 generates light to irradiate the inside of a living body. The light source device 10 is connected to the insertion member 20 by a light guide 30. The light source device 10 is connected to the insertion member 20 by the light guide 30. The light source device 10 is detachably connected to one end 31 of the light guide 30, and supplies the generated light to the one end 31 of the light guide 30. The other end 32 of the light guide 30 is detachably connected to the connection portion 22 of the insertion member 20.

[0012] The light source device 10 includes a first light source 11 and a second light source 12. The first light source 11 emits (emits) light in a first wavelength band. In this embodiment, the first light source 11 emits white light (normal light) as the first wavelength band. The first light source 11 is configured by, for example, an LED (Light Emitting Diode). The light emission luminance (light intensity) of the first light source 11 is adjustable by the control unit 94. The light intensity of the first light source 11 corresponds to an example of a first control parameter or a first light intensity adjustment parameter for adjusting the luminance of an image (normal light image) captured based on white light (normal light). The emitted color of the first light source 11 is not limited to white. For example, the first light source 11 can also emit light of another color, such as purple. In this case, the emitted color can be changed by, for example, changing the mixing ratio of RGB colors of the first light source 11.

[0013] The second light source 12 emits (lights) excitation light in a second wavelength band different from the first wavelength band. In this embodiment, the second light source 12 emits near-infrared excitation light (excitation light) in a near-infrared wavelength band as the second wavelength band. The second light source 12 is configured, for example, by a semiconductor laser. In this embodiment, the second light source 12 is configured by a semiconductor laser that emits near-infrared excitation light in the near-infrared wavelength band. The light emission brightness (light amount) of the second light source 12 can be adjusted by the control unit 94.

[0014] The near-infrared excitation light emitted by the second light source 12 is excitation light that excites a fluorescent substance (phosphor) such as indocyanine green. Furthermore, when excited by the near-infrared excitation light, the fluorescent substance such as indocyanine green emits fluorescence having a central wavelength longer than the central wavelength of the wavelength band of the near-infrared excitation light. The wavelength bands of the near-infrared excitation light and the fluorescence may be set to partially overlap or not overlap at all. The light intensity of the second light source 12 corresponds to an example of a second control parameter or second light intensity adjustment parameter for adjusting the brightness of an image (fluorescence image) captured based on the fluorescence.

[0015] Under the control of the control device 90, the light source device 10 causes the first light source 11 to emit light during the first period of alternately repeated first and second periods. That is, during the first period, the light source device 10 emits normal light (white light). Furthermore, under the control of the control device 90, the light source device 10 causes the second light source 12 to emit light during the second period. That is, during the second period, the light source device 10 emits near-infrared excitation light. The light source device 10 is configured separately from the control device 90, but the light source device 10 may also be provided inside the control device 90.

[0016] The insertion member 20 is connected to the light source device 10 via a light guide 30. The insertion member 20 is a rigid endoscope having a rigid, elongated shape, and can be inserted into a living body from its distal end portion 21. A proximal end portion 23 (eyepiece portion) of the insertion member 20 is detachably connected to a camera head 50.

[0017] Light (normal light or near-infrared excitation light) supplied from the light source device 10 via the light guide 30 is emitted from the tip 21 of the insertion member 20 toward the observation target inside the subject. The emitted illumination light is reflected by the observation target inside the living body and enters the insertion member 20 again from the tip 21. An observation optical system composed of one or more optical lenses is provided inside the insertion member 20. The reflected light from the observation target S is collected by the observation optical system and guided into the camera head 50 via the base end (eyepiece) 23 of the insertion member 20.

[0018] More specifically, when normal light (white light) is irradiated into a living body, the normal light passing through the living body (normal light reflected within the living body) is collected by the optical system in the insertion member 20 and guided into the camera head 50. Hereinafter, the normal light collected by the optical system in the insertion member 20 may be referred to as a first object image. Furthermore, when near-infrared excitation light is irradiated into a living body, the near-infrared excitation light reflected within the living body and fluorescence emitted by excitation of a fluorescent substance such as indocyanine green that accumulates in a lesion within the living body are collected by the optical system in the insertion member 20 and guided into the camera head 50. Hereinafter, the fluorescence (or near-infrared excitation light and fluorescence) collected by the optical system in the insertion member 20 may be referred to as a second object image.

[0019] FIG. 2 is a diagram showing the internal configuration of the insertion member 20 and the camera head 50. The tip 21 of the insertion member 20 is provided with an emission section 24A that emits light supplied from the light source device 10 via the light guide 30 toward a target region (observation target S) within a subject (living body). The emission section 24A may be composed of a lens or other member, or may be a space (hole). The light L1 (normal light or near-infrared excitation light) emitted from the emission section 24A is reflected by the target region (observation target S) within the living body. The tip 21 of the insertion member 20 is provided with an incidence section 24B onto which reflected waves (normal light or near-infrared excitation light) and fluorescence generated by excitation are incident. The light L2 incident on the incidence section 24B includes reflected light of the normal light, or includes reflected light of the near-infrared excitation light and fluorescence. The incidence section 24B may be composed of a lens or other member, or may be a space (hole). The light L2 enters the incident portion 24B and is guided again into the insert member 20. Although the exit portion 24A and the incident portion 24B are provided separately, the exit portion 24A and the incident portion 24B may be the same portion.

[0020] The insertion member 20 includes an observation optical system 25 composed of optical lenses 25a, 25b, 25c, 25d, 25e, 25f, 25g, and 25h. The final optical lens 25h is sometimes called an eyepiece. Light L2 from the observation target S is collected by the observation optical system 25 and guided to the camera head 50 via the proximal end 23 of the insertion member 20. A field of view mask 27 (rigid endoscope mask) is provided between the optical lens 25g and the optical lens 25h (eyepiece). The optical axis of the optical lens 25h (eyepiece), the central axis of the field of view mask 27, and the optical axes of the optical lenses 25a to 25g are aligned. The field of view mask 27 has, for example, a circular opening and passes light corresponding to a field of view image to be observed by a user such as a doctor. The light collected by the observation optical system 25 is partially masked by a field of view mask 27 after the optical lens 25g, and then collected by an optical lens 25h and directed into the camera head 50. Note that a configuration without a field of view mask is also possible.

[0021] The camera head 50 corresponds to an imaging device that captures an image of the subject or an observation target of the subject. The camera head 50 is detachably connected to the proximal end 23 (eyepiece) of the insertion member 20. Under the control of the control device 90, the camera head 50 captures a first subject image (normal light) and a second subject image (fluorescence and near-infrared excitation light) focused by the insertion member 20, and outputs an image signal resulting from the capture. The image signal is, for example, an image signal of 4K or higher.

[0022] The camera head 50 is provided with a lens unit 51, an imaging section 52 (first imaging section, second imaging section), and a communication section 53 inside. The optical axes of the lens unit 51 and the imaging section 52 coincide with the optical axis of the observation optical system 25 of the insertion member 20. Light L2 guided to the camera head 50 is collected by the lens unit 51 and output to an imaging element 522 of the imaging section 52. The imaging section 52 exposes the light output to the imaging element 522 and converts it into an electrical signal, thereby generating an image signal of the observation target S. The communication section 53 outputs the image signal to the transmission cable 80. The detailed configuration and operation of the camera head 50 will be described later.

[0023] One end of the transmission cable 80 is detachably connected to the control device 90, and the other end is detachably connected to the camera head 50. The transmission cable 80 transmits image signals and the like output from the camera head 50 to the control device 90, and also transmits control signals, synchronization signals, clocks, power, and the like output from the control device 90 to the camera head 50. The transmission of image signals and the like from the camera head 50 to the control device 90 via the transmission cable 80 may be by optical signals or electrical signals. Similarly, the transmission of control signals, synchronization signals, and clocks from the control device 90 to the camera head 50 via the transmission cable 80 may be by optical signals or electrical signals. Wireless transmission may be used instead of wired transmission via the transmission cable 80. Any method or standard may be used, such as a wireless LAN (Local Area Network), Bluetooth (registered trademark), or infrared communication.

[0024] The display device 70 is, for example, a display device such as a liquid crystal display or an organic electro-luminescence (EL) display. Under the control of the control device 90, the display device 70 receives an image signal processed by the control device 90 and displays an image (video) based on the image signal on a screen. The display device 70 may be connected to the control device 90 by a transmission cable and receive the image signal for display via a wired connection, or may receive the image signal for display via wireless transmission.

[0025] The control device 90 comprehensively controls the operations of the light source device 10, the camera head 50, and the display device 70. For example, the control device 90 generates an image signal (video signal) for display by performing various image processing on an image signal of an object to be observed received from the camera head 50 via a transmission cable 80. The control device 90 also transmits a control signal for controlling the light source device 10 to the light source device 10. The light source device 10 may be connected to the control device 90 via a transmission cable and receive the control signal for display via a wired connection, or may receive the control signal via wireless transmission.

[0026] The control device 90 may be realized by a general-purpose processor such as a CPU (Central Processing Unit), MPU, or GPU (Micro Processing Unit), or GPU (Graphics Processing Unit), or may be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array). When the control device 90 is configured by a general-purpose processor, the functions of the control device 90 are realized by causing the general-purpose processor to execute software or a program. The detailed configuration of the control device 90 will be described later.

[0027] FIG. 3 is a block diagram showing the configuration of the camera head 50 and the control device 90. As shown in FIG. [Configuration of camera head 50] The camera head 50 includes a lens unit 51 , an imaging unit 52 , and a communication unit 53 .

[0028] The lens unit 51 includes one or more lenses. The lens unit 51 collects the first object image (normal light) or the second object image (fluorescence and near-infrared excitation light) collected by the insertion member 20 and outputs the collected image to an imaging surface of the imaging section 52 (image sensor 522). The imaging section 52 images the inside of a living body under the control of the control device 90. The imaging section 52 includes a first imaging section that exposes the first object image (normal light) collected by the lens unit 51 to generate a first image signal, and a second imaging section that exposes the second object image (fluorescence) collected by the lens unit 51 to generate a second image signal.

[0029] The imaging unit 52 includes an excitation light cut filter 521 , an imaging element 522 , and a signal processing unit 523 .

[0030] The excitation light cut filter 521 is provided between the lens unit 51 and the image sensor 522 and includes a bandstop filter that removes a specific wavelength band. The wavelength band that is cut (removed) by the excitation light cut filter 521 is referred to as the cut band. A wavelength band that is shorter than the cut band and that passes through the excitation light cut filter 521 is referred to as the short-wave transmission band. A wavelength band that is longer than the cut band and that passes through the excitation light cut filter 521 is referred to as the long-wave transmission band. The cut band includes at least a portion of the wavelength band of near-infrared excitation light. The long-wave transmission band includes the wavelength band of fluorescence. The long-wave transmission band may include not only the wavelength band of fluorescence but also a portion of the wavelength band of near-infrared excitation light. In the following description, the long-wave transmission band will be described as including only the wavelength band of fluorescence, but it may also include a portion of the wavelength band of near-infrared excitation light. The short-wave transmission band includes the wavelength band of normal light (white light) (first wavelength band). That is, the excitation light cut filter 521 transmits a first object image (normal light (white light)) traveling from the lens unit 51 to the image sensor 522. The excitation light cut filter 521 also transmits a second object image (fluorescence) traveling from the lens unit 51 to the image sensor 522. Note that, although the excitation light cut filter 521 is disposed in the camera head 50, the present invention is not limited to this and may be disposed on the optical path of the optical system in the insertion member 20.

[0031] The image sensor 522 receives light transmitted through the excitation light cut filter 521 and converts it into an electrical signal (analog signal). The image sensor 522 includes, for example, a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). The image sensor 522 is shared by a first imaging unit that exposes a first subject image (normal light) to generate a first image signal, and a second imaging unit that exposes a second subject image (fluorescence) to generate a second image signal, but may be provided separately for each.

[0032] A color filter, which is an optical filter, is provided on the imaging surface (light receiving surface) of the image sensor 522. The color filter is a predetermined arrangement of three types of filters (R filter, G filter, B filter) classified according to the wavelength bands of light to be transmitted (R (red), G (green), B (blue)). An example of an arrangement format for the RGB filters is the Bayer array. The R filter is a filter that mainly transmits light in the R wavelength band, the B filter is a filter that mainly transmits light in the B wavelength band, and the G filter is a filter that mainly transmits light in the G wavelength band. Note that each of the R, G, and B filters also transmits fluorescent light (and near-infrared excitation light). The image sensor 522 is sensitive not only to light in the R, G, and B wavelength bands, but also to light in the fluorescent wavelength band (and near-infrared excitation light).

[0033] Under the control of the control device 90, the image sensor 522 captures images in synchronism with the light emission timing of the light source device 10, for each of the first and second periods that are alternately repeated.

[0034] Specifically, the image sensor 522 exposes a first object image (normal light) during a first period to generate a first image signal. The exposure amount or exposure time of the first object image (normal light) during the first period of the image sensor 522 can be adjusted by the control unit 94. An image generated by capturing the first object image (normal light) by the image sensor 522, i.e., an image based on the first image signal, is referred to as a normal light image (first image). The exposure amount or exposure time of the first object image corresponds to an example of a first control parameter or a first exposure adjustment parameter for adjusting the brightness of an image (normal light image) captured based on white light (normal light). Furthermore, the image sensor 522 captures a second object image (fluorescence) during a second period to generate a second image signal. The exposure amount or exposure time of the second object image (fluorescence) captured by the image sensor 522 during the second period can be adjusted by the control unit 94. An image generated by capturing the second object image, i.e., an image based on the second image signal, is referred to as a fluorescence image (second image). The exposure amount or exposure time of the second object image corresponds to an example of a second control parameter or second exposure adjustment parameter for adjusting the brightness of an image captured based on fluorescence (fluorescence image).

[0035] The signal processing unit 523 performs various signal processing on the analog image signals (analog first image signal and second image signal, respectively) generated by the image sensor 522, and outputs digital image signals (digital first image signal and second image signal, respectively). The first image signal represents a normal light image (first image), and the second image signal represents a fluorescent image (second image). The normal light image (first image) and the fluorescent image (second image) may be collectively referred to as captured images. The signal processing unit 523 outputs the captured image (digital signal). The signal processing unit 523 includes a gain adjustment unit (AGC (Auto Gain Control) circuit) 540 for performing analog gain adjustment. That is, the AGC circuit 540 performs AGC as signal processing. The AGC circuit 540 adjusts the analog gain of the first image signal (analog signal) captured based on normal light. The AGC circuit 540 adjusts the analog gain of the second image signal (analog signal) captured based on fluorescent light. By adjusting the analog gain, it is possible to adjust the amplitude (e.g., brightness) of the image signal (analog signal). The signal processing unit 523 performs AD conversion on the first image signal and the second image signal, each of which has passed through AGC or the like, to generate a digital first image signal and a digital second image signal, i.e., an imaging signal (digital signal). The signal processing unit 523 is included in the imaging unit 52, but may also be present as an independent functional unit between the imaging unit 52 and the communication unit 53.

[0036] The analog gain of the first image signal (analog signal) used in the gain adjustment unit 540 can be adjusted by the control unit 94. The analog gain of the first image signal (analog signal) corresponds to an example of a first control parameter or a first analog gain adjustment parameter for adjusting the luminance of an image (normal light image) captured based on white light (normal light).

[0037] The analog gain of the second image signal (analog signal) used in the gain adjustment unit 540 can be adjusted by the control unit 94. The analog gain of the second image signal (analog signal) corresponds to an example of a second control parameter or a second analog gain adjustment parameter for adjusting the brightness of an image captured based on fluorescence (fluorescence image).

[0038] The gain adjustment unit 540 is used in common by the first imaging unit that exposes a first subject image (normal light) to generate a first image signal, and the second imaging unit that exposes a second subject image (fluorescence) to generate a second image signal, but may be provided separately for each.

[0039] The communication unit 53 transmits the captured image (digital signal) output from the imaging unit 52 to the control device 90 via the transmission cable 80. The communication unit 53 is configured, for example, with a high-speed serial interface that communicates the captured image at a high-speed transmission rate.

[0040] [Configuration of the control device 90] The control device 90 includes a communication unit 91 , a memory 92 , an observation image generation unit 93 , a control unit 94 , an input unit 95 , an output unit 96 , and a storage unit 97 .

[0041] The communication unit 91 receives the captured image output from the camera head 50 via the transmission cable 80. The communication unit 91 is configured, for example, with a high-speed serial interface that communicates the captured image with the communication unit 53 of the camera head 50 at a high-speed transmission rate.

[0042] The memory 92 temporarily stores multiple frames of captured images sequentially output from the camera head 50. The memory 92 is configured, for example, by a volatile memory or a non-volatile memory. As an example, the memory 92 is configured by a DRAM (Dynamic Random Access Memory), which is a volatile memory.

[0043] Under the control of the control unit 94, the observation image generation unit 93 processes the captured images sequentially output from the camera head 50 and received by the communication unit 91. The observation image generation unit 93 includes a memory controller 931, a first image processing unit 932, a second image processing unit 933, a superimposed image generation unit 934, and a display control unit 935. The first image processing unit 932 and the second image processing unit 933 are provided separately, but may be integrated into a single image processing unit.

[0044] The memory controller 931 controls the writing and reading of captured images to the memory 92. More specifically, the memory controller 931 sequentially writes captured images (normal light images and fluorescent images) that are sequentially output from the camera head 50 and received by the communication unit 91 into the memory 92. The memory controller 931 also reads normal light images from the memory 92 at specific times and outputs the read normal light images to the first image processing unit 932. The memory controller 931 also reads fluorescent images from the memory 92 at specific times and outputs the read fluorescent images to the second image processing unit 933.

[0045] The first image processing unit 932 performs first image processing on the input normal light image signal (digital first image signal). The first image processing unit 932 includes a gain adjustment unit (AGC circuit) 940A that adjusts the digital gain of the normal light image signal using AGC (Auto Gain Control). As the first image processing, the first image processing unit 932 adjusts the digital gain of the normal light image signal. Specifically, the pixel values ​​(RGB luminance values) of the normal light image are adjusted by adjusting the digital gain of the normal light image signal (RGB signal).

[0046] Furthermore, the first image processing unit 932 may perform at least one of optical black subtraction processing, white balance adjustment processing, demosaic processing, color correction processing, and gamma correction processing as the first image processing.

[0047] Furthermore, as the first image processing, the first image processing unit 932 performs YC processing, which converts RGB signals (normal light image) after processing such as AGC into luminance signals and color difference signals (Y, Cb / Cr signals). That is, in the YC processing, a luminance signal (Y value (hereinafter referred to as WLI_Y)) and color difference signals (Cb value (hereinafter referred to as WLI_Cb) and Cr value (hereinafter referred to as WLI_Cr)) are calculated for each pixel based on the RGB signals (normal light image) after processing such as AGC.

[0048] For example, WLI_Y is expressed by the following formula: "+" indicates an addition symbol, and "x" indicates a multiplication symbol. WLI_Y=D R ×R+D G ×G+D B ×B···(1) R is the pixel value of red (R), G is the pixel value of green (G), and B is the pixel value of blue (B). D R , D G , D B are the coefficients for R, G, and B, respectively. For example, D R is 0.3, D G is 0.6, D B is 0.1. D R , D G , D B The total value is set to 1. D R , D G , D B may be a predetermined value or may be adjusted by the control unit 94 as described in the description of other embodiments below. R , D G , D B may be stored in the memory 92 or the storage unit 97.

[0049] For example, WLI_Cb is expressed by the following equation: WLI_Cb=ER ×R+E G ×G+E B ×B···(2) E R , E G , E B are the coefficients for R, G, and B, respectively. E R , E G , E B may be a predetermined value or may be adjusted by the control unit 94. R , E G , E B may be stored in the memory 92 or the storage unit 97.

[0050] For example, WLI_Cr is expressed by the following formula: WLI_Cr=F R ×R+F G ×G+F B ×B···(3) F R , F G , F B are the coefficients for R, G, and B, respectively. F R , F G , F B may be a predetermined value or may be adjusted by the control unit 94. R , F G , F B may be stored in the memory 92 or the storage unit 97.

[0051] The second image processing unit 933 performs second image processing on the input fluorescence image. As the second image processing, the second image processing unit 933 is provided with a gain adjustment unit (AGC circuit) 940B that adjusts the gain of the fluorescence image signal by AGC. As the second image processing, the second image processing unit 933 adjusts the gain of the fluorescence image signal by the AGC circuit 940B. Specifically, the pixel values ​​of the fluorescence image are adjusted by adjusting the digital gain of the fluorescence image signal.

[0052] Furthermore, the second image processing unit 933 performs a process of generating a luminance signal (Y value (hereinafter referred to as IR_Y)) from the fluorescent image after AGC as the second image processing. As an example, each pixel value of the fluorescent image is acquired as the luminance signal IR_Y.

[0053] The first image processing unit 932 and the second image processing unit 933 each have a separate gain adjustment unit (AGC circuit), but a common gain adjustment unit (AGC circuit) may be provided for the first image processing unit 932 and the second image processing unit 933. Hereinafter, the gain adjustment units (AGC circuits) 940A and 940B may be collectively referred to as gain adjustment unit (AGC circuit) 940.

[0054] The digital gain for normal light used in the AGC circuit 940 can be adjusted by the control unit 94. The digital gain for normal light corresponds to an example of a first control parameter or a first digital gain adjustment parameter for adjusting the brightness of an image (normal light image) captured based on white light (normal light).

[0055] The digital gain for fluorescence used in the AGC circuit 940 can be adjusted by the control unit 94. The digital gain for fluorescence corresponds to an example of a second control parameter or a second digital gain adjustment parameter for adjusting the brightness of an image captured based on fluorescence (fluorescence image).

[0056] The superimposed image generating unit 934 generates a superimposed image (composite image) by superimposing (combining) corresponding pixels for all regions of the normal light image after the first image processing has been performed by the first image processing unit 932 and the fluorescent image after the second image processing has been performed by the second image processing unit 933. That is, the superimposed image generating unit 934 generates a superimposed image by superimposing a first image (normal light image) represented by a first image signal generated by exposing a first object image (normal light) with a second image (fluorescence image) represented by a second image signal generated by capturing a second object image. The process of generating the superimposed image will be described in detail below.

[0057] The superimposed image generation unit 934 acquires, for each pixel, WLI_Y, WLI_Cb, and WLI_Cr in the normal light image after the first image processing has been performed by the first image processing unit 932. The superimposed image generation unit 934 also acquires, for each pixel, IR_Y in the fluorescence image after the second image processing has been performed by the second image processing unit 933. The superimposed image generation unit 934 also acquires a predetermined Cb value (hereinafter referred to as IR_Cb) and a predetermined Cr value (hereinafter referred to as IR_Cr) for the fluorescence image. IR_Cb and IR_Cr may be stored in the memory 92 or the storage unit 97.

[0058] The superimposed image generation unit 934 performs a plurality of calculations such as addition, multiplication, or subtraction based on WLI_Y, WLI_Cb, and WLI_Cr and IR_Y, IR_Cb, and IR_Cr, thereby calculating the Y value (hereinafter referred to as stacked_Y), Cb value (hereinafter referred to as stacked_Cb), and Cr value (hereinafter referred to as stacked_Cr) in the superimposed image for each pixel.

[0059] Specifically, "stacked_Y", "stacked_Cb", and "stacked_Cr" are respectively expressed by the following equations (4) to (6), where "-" indicates a subtraction symbol, "+" indicates an addition symbol, and "x" indicates a multiplication symbol. stacked_Y=(1-α)×WLI_Y+α×β×IR_Y···(4) stacked_Cb=(1-α)×WLI_Cb+α×IR_Cb···(5) stacked_Cr=(1-α)×WLI_Cr+α×IR_Cr···(6)

[0060] In equations (4) to (6), α is an alpha value. α may be a value determined in advance based on the knowledge of a user such as a doctor, or may be adjusted by the control unit 94. For example, α takes a value greater than or equal to 0 and less than or equal to 1. The alpha value may be stored in the memory 92 or a storage unit 97, which will be described later. When α is adjusted by the control unit 94, α may be determined based on at least one of IR_Y, gain (gain of the normal light image, gain of the fluorescence image), gamma setting of the normal light image, etc. For example, α may be a value according to IR_Y (for example, a value calculated by multiplying IR_Y by a coefficient).

[0061] β is an adjustment coefficient for IR_Y. β is a predetermined value, for example, a value greater than or equal to 0 and less than or equal to 1. As an example, β is 0.6. However, β may be adjusted by the control unit 94. Note that a coefficient equivalent to β, i.e., an adjustment coefficient for IR_Cb and an adjustment coefficient for IR_Cr, may also be introduced into equations (5) and (6).

[0062] As shown in equation (4), the Y value (stacked_Y) of each pixel (target pixel) in the superimposed image is generated by alpha blending the Y value (WLI_Y) of the pixel (first corresponding pixel) corresponding to the target pixel in the normal light image and the Y value (IR_Y) of the pixel (second corresponding pixel) corresponding to the target pixel in the fluorescent image using an alpha value (α).

[0063] As shown in equation (5), the Cb value (stacked_Cb) of the target pixel in the superimposed image is generated by alpha blending the Cb value (WLI_Cb) of the first corresponding pixel and a predetermined Cb value (IR_Cb) using an alpha value (α).

[0064] As shown in equation (6), the Cr value (stacked_Cr) of the target pixel in the superimposed image is generated by alpha blending the Cr value (WLI_Cr) of the first corresponding pixel and a predetermined Cr value (IR_Cr) using an alpha value (α).

[0065] Equations (4) to (6) are merely examples, and other equations may be used to calculate "stacked_Y", "stacked_Cb", and "stacked_Cr".

[0066] In the first image processing example, YC processing is performed to convert the normal light image (RGB signal) into a luminance signal and color difference signals (Y, Cb / Cr signals). However, alpha blending may be performed between each RGB signal and the fluorescence image signal without YC processing. For example, the pixel value (RGB luminance value) of a target pixel in the superimposed image may be calculated by superimposing the pixel value (RGB luminance value) of a first corresponding pixel in the normal light image with the RGB value of a specific pseudo color at a ratio using an alpha value. Specifically, the pixel value (RGB luminance value) of the target pixel in the superimposed image is calculated by weighting the pixel value (RGB luminance value) of the first corresponding pixel in the normal light image and the RGB value of the specific pseudo color by 1-α and α, respectively, and then summing them. α may be determined based on the luminance value of the second corresponding pixel in the fluorescence image. The luminance control of normal light images and fluorescence images disclosed in this embodiment and described in detail below can be used both with and without YC processing. In the following explanation, it is assumed that alpha blending is performed based on the luminance signal and color difference signals (Y, Cb / Cr signals) obtained by YC processing to calculate "stacked_Y", "stacked_Cb", and "stacked_Cr".

[0067] Under the control of the control unit 94, the display control unit 935 generates an image signal (video signal) for display based on the superimposed image (stacked_Y, stacked_Cb, stacked_Cr) generated by the superimposed image generation unit 934. The display control unit 935 outputs the generated video signal to the display device 70.

[0068] The control unit 94 controls the operations of the light source device 10, the camera head 50, and the display device 70 by outputting control signals, and also controls the operation of the control device 90 as a whole.

[0069] The control unit 94 controls at least one of the light source device 10 (light source unit), the image capture unit 52, the gain adjustment unit 540, and the gain adjustment unit 940 (940A, 940B) based on the luminance information (e.g., Y value (stacked_Y)) of the superimposed image generated by the superimposed image generation unit 934. As a result, the control unit 94 performs luminance control to adjust the luminance of the normal light image (image based on the first image signal) and the luminance of the fluorescent image (image based on the second image signal) by feedback control.

[0070] FIG. 4 is a block diagram of a configuration for controlling brightness in the control unit 94. The control unit 94 includes a brightness information calculation unit 941 and a parameter adjustment unit 942. The operation of the configuration shown in FIG. 4 will be described below using as an example a case where the light source device 10 (light source unit), the image capture unit 52 (image capture element 522), the gain adjustment unit 540, and the gain adjustment units 940A and 940B are controlled.

[0071] [Control of light source device 10] This shows an example in which the control unit 94 adjusts the brightness of the normal light image and the brightness of the fluorescent image by feedback controlling the light source device 10 based on the brightness information of the superimposed image and target brightness information representing a predetermined target brightness. The luminance information calculation unit 941 of the control unit 94 calculates luminance information of the superimposed image. The luminance information of the superimposed image is the average value (average luminance) of the luminance values ​​(stacked_Y) of all pixels of the superimposed image. However, the luminance information may also be a statistical value such as a median, maximum value, or minimum value. In the following description, the luminance information is assumed to be the average luminance.

[0072] The parameter adjustment unit 942 adjusts the light emission luminance (light intensity) of the first light source 11 as a first control parameter and the light emission luminance (light intensity) of the second light source 12 as a second control parameter based on the luminance information of the superimposed image and target luminance information representing a predetermined target luminance. Specifically, the parameter adjustment unit 942 adjusts the first and second control parameters so that the luminance information of the superimposed image matches or approaches the target luminance information, i.e., so that the difference between the two is zero or approaches zero. By adjusting the light intensity of the first light source 11, the luminance of the normal light image generated by the normal light image processing unit 932 can be adjusted. By adjusting the light intensity of the second light source 12, the luminance of the fluorescent image generated by the fluorescent light image processing unit 933 can be adjusted. The control unit 94 transmits the first and second control parameters adjusted by the parameter adjustment unit 942 to the first light source 11 and the second light source 12, respectively, as control signals.

[0073] When adjusting the light intensity of the first light source 11 and the light intensity of the second light source 12, the control unit 94 may satisfy a condition regarding the ratio between the light emission luminance (light intensity) of the first light source 11 and the light emission luminance (light intensity) of the second light source 12. The condition regarding the ratio may be to maintain the ratio the same before and after adjustment, or to suppress fluctuations in the ratio before and after adjustment. For example, the light intensity of these light sources may be adjusted so that the amount of fluctuation in the ratio of the light intensity of these light sources is suppressed to a threshold value or less. As a specific example of suppressing fluctuations in the ratio before and after adjustment, the difference in the ratio before and after adjustment may be set to a threshold value or less or to zero. For example, when the light intensity of the first light source 11 is increased by X times, the light intensity of the second light source 12 is also increased by X times.

[0074] In this way, the luminance information of the superimposed image is made to match or approach the target luminance information while suppressing fluctuations in the ratio between the emission luminance (light intensity) of the first light source 11 and the emission luminance (light intensity) of the second light source 12. Rather than individually making the luminance information of each normal light image and fluorescence image approach the target luminance, the luminance information of the superimposed image is made to match or approach the target luminance information. This makes it possible to generate a superimposed image in which the luminance of the normal light image and the luminance of the fluorescence image are balanced, regardless of the state of the observation subject (for example, before or after injection of a fluorescent substance).

[0075] A specific method for adjusting the first and second control parameters is, for example, to increase or decrease the parameter values ​​by a fixed percentage. For example, if the luminance information of the superimposed image is lower than the target luminance information, the current values ​​of the first and second control parameters are multiplied by the same coefficient K1. For example, if the light intensities of the first light source 11 and the second light source 12 are each increased by 1.1 times, K1=1.1. If the luminance information of the superimposed image is higher than the target luminance information, the current values ​​of the first and second control parameters are multiplied by the coefficient K2. For example, if the light intensities of the first light source 11 and the second light source 12 are each increased by 0.9 times, K2=0.9. The value of the multiplication coefficient may be changed depending on the magnitude of the difference between the luminance information of the superimposed image and the target luminance information. For example, the absolute value of the multiplication coefficient may be increased as the difference increases. In this case, for example, a table may be used that defines the value of the multiplication coefficient for each magnitude of difference. Alternatively, a function may be used that inputs the difference and outputs the value of the multiplication coefficient. The value of the coefficient may be the same regardless of the magnitude of the difference. By keeping the coefficient value constant regardless of the magnitude of the difference, the changes in the brightness of the normal image and the fluorescent image can be smoothed, and therefore the changes in the display brightness (brightness) of the superimposed image can also be smoothed.

[0076] [Control of the imaging unit 52 (imaging element 522)] This shows an example in which the control unit 94 adjusts the brightness of the normal light image and the brightness of the fluorescent image by feedback controlling the image capturing unit 52 (image capturing element 522) based on brightness information of the superimposed image and target brightness information representing a predetermined target brightness. Descriptions of content that is the same as the description of the control of the light source device 10 will be omitted or simplified as appropriate.

[0077] The luminance information calculation unit 941 of the control unit 94 calculates luminance information of the superimposed image. The parameter adjustment unit 942 of the control unit 94 adjusts the exposure amount (or exposure time) of normal light at the image sensor 522 as a first control parameter based on the luminance information of the superimposed image and the target luminance information. The parameter adjustment unit 942 of the control unit 94 adjusts the exposure amount (or exposure time) of fluorescent light as a second control parameter based on the luminance information of the superimposed image and the target luminance information. Specifically, the first and second control parameters are adjusted so that the luminance information matches or approaches the target luminance information, i.e., so that the difference between the two is zero or approaches zero. The exposure amount can be adjusted by, for example, controlling the electronic shutter of the image sensor 522 to adjust the amount of charge accumulated in the image sensor 522. Increasing the exposure time increases the amount of accumulated charge, and shortening the exposure time decreases the amount of accumulated charge. By adjusting the exposure amount of normal light, the luminance of the normal light image generated by the normal light image processing unit 932 can be adjusted. Adjusting the amount of exposure to the fluorescence makes it possible to adjust the brightness of the fluorescence image generated by the fluorescence image processing unit 933. The control unit 94 transmits the first control parameter and the second control parameter, each adjusted by the parameter adjustment unit 942, to the imaging unit 52 (image sensor 522) as control signals.

[0078] The control unit 94 may satisfy a condition regarding the ratio of the exposure amount of normal light and the exposure amount of fluorescent light in the image sensor 522. The condition regarding the ratio may be to maintain the same ratio before and after adjustment, or to suppress fluctuations in the ratio before and after adjustment. For example, these exposure amounts may be adjusted so that the amount of fluctuation in the ratio of these exposure amounts is kept below a threshold. As an example of suppressing fluctuations in the ratio of exposure amounts before and after adjustment, the difference in the ratio of exposure amounts before and after adjustment may be set below a threshold or to zero. For example, if the exposure amount of normal light is increased by X times, the exposure amount of fluorescent light is also increased by X times.

[0079] In this way, the luminance information of the superimposed image is made to match or approach the target luminance information while suppressing fluctuations in the ratio between the normal light exposure amount and the fluorescent light exposure amount at the image sensor 522. In other words, rather than individually making the luminance information of each normal light image and fluorescent light image match the target luminance information, the luminance information of the superimposed image is made to match or approach the target luminance information. This makes it possible to generate a superimposed image in which the luminance of the normal light image (white light luminance) and the luminance of the fluorescent image (fluorescence luminance) are balanced, regardless of the state of the observation subject (e.g., before or after injection of a fluorescent substance). The specific method for adjusting the normal light exposure amount (first control parameter) and the fluorescent light exposure amount (second control parameter) may be the same as in the case of controlling the light source device 10.

[0080] [Control of gain adjustment unit 540] In this example, the control unit 94 adjusts the luminance of the normal light image and the luminance of the fluorescent light image by feedback controlling the gain adjustment unit 540 based on the luminance information of the superimposed image and the target luminance information. Descriptions of the same content as those of the control of the light source device 10 will be omitted or simplified as appropriate.

[0081] The control unit 94 adjusts the analog gain of the normal light image in the gain adjustment unit 540 as a first control parameter. The control unit 94 adjusts the analog gain of the fluorescence image in the gain adjustment unit 540 as a second control parameter. Specifically, the analog gains are adjusted so that the luminance information of the superimposed image matches or approaches the target luminance information, that is, so that the difference between the two is zero or approaches zero. Examples of luminance information of the superimposed image are the same as those described above for controlling the light source device 10. The luminance of the normal light image is adjusted in an analog manner by adjusting the analog gain of the normal light image. The luminance of the fluorescence image is adjusted in an analog manner by adjusting the analog gain of the fluorescence image signal. The control unit 94 transmits the first control parameter and the second control parameter, each adjusted by the parameter adjustment unit 942, to the gain adjustment unit 540 as control signals.

[0082] The control unit 94 may satisfy a condition regarding the ratio between the analog gain of the normal light image and the analog gain of the fluorescence image. The condition regarding the ratio may be to maintain the ratio between the analog gain of the normal light image and the analog gain of the fluorescence image the same before and after adjustment, or to suppress fluctuations in the ratio before and after adjustment. For example, the control unit 94 adjusts these analog gains so that the amount of fluctuation in the ratio between the analog gain of the normal light image and the analog gain of the fluorescence image is kept below a threshold. As an example of suppressing fluctuations in the ratio before and after adjustment, the difference in the ratio before and after adjustment may be set below a threshold or to zero. For example, if one analog gain is increased by X times, the other analog gain is also increased by X times.

[0083] In this way, the luminance information of the superimposed image is made to match or approach the target luminance information while suppressing fluctuations in the ratio between the analog gain of the normal light image and the analog gain of the fluorescence image. In other words, rather than individually matching the luminance information of each normal light image and fluorescence image to the target luminance, the luminance information of the superimposed image is made to match or approach the target luminance information. This makes it possible to generate a superimposed image in which the luminance of the normal light image (the luminance of white light) and the luminance of the fluorescence image (the luminance of fluorescence) are balanced, regardless of the state of the observation subject (e.g., before or after injection of a fluorescent substance). The specific method for adjusting the analog gain of the normal light image (first control parameter) and the analog gain of the fluorescence image (second control parameter) may be the same as in the case of controlling the light source device 10.

[0084] [Control of gain adjustment unit 940] In this example, the control unit 94 adjusts the luminance of the normal light image and the luminance of the fluorescent light image by feedback controlling the gain adjustment unit 940 based on the luminance information of the superimposed image and the target luminance information. Descriptions of the same content as those of the control of the light source device 10 will be omitted or simplified as appropriate.

[0085] The control unit 94 adjusts the digital gain of the normal light image in the gain adjustment unit 940 as a first control parameter. The control unit 94 adjusts the digital gain of the fluorescence image in the gain adjustment unit 940 as a second control parameter. Specifically, the control unit 94 adjusts these digital gains so that the luminance information of the superimposed image matches or approaches the target luminance information, i.e., so that the difference between the two is zero or approaches zero. Examples of luminance information of the superimposed image are the same as those described above for controlling the light source device 10. The luminance of the normal light image is digitally adjusted by adjusting the digital gain of the normal light image signal. The luminance of the fluorescence image is digitally adjusted by adjusting the digital gain of the fluorescence image signal. The control unit 94 transmits the first control parameter and the second control parameter, each adjusted by the parameter adjustment unit 942, to the gain adjustment unit 940 as control signals.

[0086] The control unit 94 may satisfy a condition regarding the ratio between the digital gain of the normal light image and the digital gain of the fluorescence image. The condition regarding the ratio may be to maintain the ratio between the digital gain of the normal light image and the digital gain of the fluorescence image the same before and after adjustment, or to suppress fluctuations in the ratio before and after adjustment. For example, the control unit 94 adjusts these digital gains so that the amount of fluctuation in the ratio between the digital gain of the normal light image and the digital gain of the fluorescence image is kept below a threshold. As an example of suppressing fluctuations in the ratio before and after adjustment, the difference in the ratio before and after adjustment may be set below a threshold or to zero. For example, if one digital gain is increased by X times, the other digital gain is also increased by X times.

[0087] In this way, the luminance information of the superimposed image is made to match or approach the target luminance information while suppressing fluctuations in the ratio between the digital gain of the normal light image and the digital gain of the fluorescence image. In other words, rather than individually matching the luminance information of each normal light image and fluorescence image to the target luminance, the luminance information of the superimposed image is made to match or approach the target luminance information. This makes it possible to generate a superimposed image in which the luminance of the normal light image (the luminance of white light) and the luminance of the fluorescence image (the luminance of fluorescence) are balanced, regardless of the state of the observation subject (e.g., before or after injection of a fluorescent substance). The specific method for adjusting the digital gain of the normal light image (first control parameter) and the digital gain of the fluorescence image (second control parameter) may be the same as in the case of controlling light source device 10.

[0088] The input unit 95 is an operation device such as a mouse, a keyboard, or a touch panel. The input unit 95 accepts operations by a user such as a doctor. The input unit 95 generates an operation signal in response to the user operation and outputs the generated operation signal to the control unit 94.

[0089] The output unit 96 outputs various types of information and includes, for example, a speaker, a printer, a communication device, or an application.

[0090] The storage unit 97 stores the programs executed by the control unit 94, information necessary for the processing of the control unit 94, and the like.

[0091] The operation of the control device 90 will now be described. FIG. 5 is a flowchart showing an example of the operation of the control device 90. The control unit 94 causes the first light source 11 and the second light source 12 to emit light in a time-division manner (step S101). Specifically, the control unit 94 causes the first light source 11 to emit light during a first period and the second light source 12 to emit light during a second period of alternately repeated first and second periods based on a synchronization signal.

[0092] The control unit 94 synchronizes the light emission timing of the first light source 11 and the second light source 12 based on the synchronization signal. The control unit 94 causes the image sensor 522 to capture a first object image (normal light) and a second object image (fluorescence) during the first and second periods, respectively (steps S102 to S104). That is, if it is the first period (Yes in step S102), in other words, if normal light (white light) is irradiated into the living body, the image sensor 522 captures the first object image (normal light) to generate a normal light image (step S103). On the other hand, if it is the second period (No in step S102), in other words, if near-infrared excitation light is irradiated into the living body, the image sensor 522 captures the second object image (fluorescence) to generate a fluorescence image (step S104).

[0093] The memory controller 931 controls the writing and reading of the captured image to and from the memory 92 based on the synchronization signal (step S105).

[0094] The first image processing unit 932 and the second image processing unit 933 execute the following processing (step S106).

[0095] That is, the first image processing unit 932 performs first image processing on the normal light images sequentially read out from the memory 92 by the memory controller 931, and outputs the normal light images (WLI_Y, WLI_Cb, WLI_Cr) after the first image processing has been performed.

[0096] Furthermore, the second image processing unit 933 performs second image processing on the fluorescence image read out from the memory 92 by the memory controller 931, and outputs the fluorescence image (IR_Y) after the second image processing.

[0097] As shown in equations (4) to (6), the superimposed image generation unit 934 superimposes the normal light image (WLI_Y, WLI_Cb, WLI_Cr) output from the first image processing unit 932 and the fluorescent light image (IR_Y) output from the second image processing unit 933 at corresponding pixels, thereby generating a superimposed image (stacked_Y, stacked_Cb, stacked_Cr) (step S107).

[0098] The display control unit 935 generates an image signal (video signal) for displaying the superimposed image generated by the superimposed image generation unit 934, and outputs the video signal to the display device 70 (step S108). As a result, the superimposed image is displayed on the display device 70.

[0099] The control unit 94 calculates the luminance information of the superimposed image and controls at least one of the light source device 10, the image capturing unit 52, the gain adjustment unit 540, and the gain adjustment unit 940 so as to reduce the difference between the luminance information and the target luminance information (step S109). As a result, the luminance of the normal light image generated by the first image processing unit 932 and the luminance of the fluorescent light image generated by the second image processing unit 933 are adjusted by feedback control.

[0100] Specifically, when controlling the light source device 10, the light intensity (first control parameter) of the first light source 11 and the light intensity (second control parameter) of the second light source 12 are adjusted simultaneously. At this time, the light intensities of both light sources may be adjusted so as to maintain a constant ratio between these light intensities or suppress fluctuations in the ratio.

[0101] When controlling the imaging unit 52, the exposure amount when performing exposure with normal light (first control parameter) and the exposure amount when performing exposure with fluorescent light (second control parameter) are adjusted simultaneously. At this time, both exposure amounts may be adjusted so that the ratio between these exposure amounts is maintained at a constant ratio or fluctuations in the ratio are suppressed.

[0102] When controlling the gain adjustment unit 540, the analog gain of the normal light image (first control parameter) and the analog gain of the fluorescence image signal (second control parameter) are adjusted simultaneously. At this time, both gains may be adjusted so that the ratio between these analog gains is maintained at a constant ratio or fluctuations in the ratio are suppressed.

[0103] When controlling the gain adjustment unit 940, the digital gain of the normal light image signal (first control parameter) and the digital gain of the fluorescent light image signal (second control parameter) are simultaneously adjusted. At this time, both gains may be adjusted so that the ratio between these digital gains is maintained at a constant ratio or fluctuations in the ratio are suppressed.

[0104] After step S109, the control unit 94 determines whether or not to terminate the processing of the control device 90. For example, if an instruction to terminate observation is input from a user such as a doctor via the input unit 95, the control unit 94 determines to terminate the processing. If the control unit 94 determines to terminate the processing (No in step S110), the control unit 94 terminates the processing. If the control unit 94 determines to continue the processing, the process returns to step S101. Thereafter, steps S101 to S109 are repeated until it is determined to terminate the processing. When the processing is repeated, the first control parameter and the second control parameter have been changed from the previous processing due to the brightness control performed in step S108.

[0105] As described above, according to this embodiment, the first control parameter and the second control parameter are adjusted so that the luminance information of the superimposed image matches or approaches the target luminance information. This adjusts the luminance of the normal light image generated by the normal light image processing unit 932 and the luminance of the fluorescent image generated by the fluorescent image processing unit 933. In this case, for example, when adjusting the light source device 10, the ratio of the light intensities of the first light source 11 and the second light source 12 is maintained constant, and the respective light intensities of the first light source 11 and the second light source 12 are adjusted. When adjusting the image capturing unit 52, for example, the ratio of the exposure intensities of the normal light and the fluorescent light is maintained constant, and each exposure intensities is adjusted. When adjusting the gain adjustment unit 940, for example, the ratio of the digital gains of the normal light image and the fluorescent image is maintained constant, and the respective digital gains of the normal light image and the fluorescent image are adjusted. When adjusting the gain adjustment unit 540, for example, the ratio of the analog gains of the normal light image and the fluorescent image is maintained constant, and the respective analog gains of the normal light image and the fluorescent image are adjusted. This makes it possible to generate a superimposed image in which the brightness of the normal light image (brightness of white light) and the brightness of the fluorescent image (brightness of fluorescence) are balanced, regardless of the state of the object being observed (for example, before or after injection of a fluorescent substance).

[0106] For example, if fluorescent light has not yet been injected into the living body, the object of observation has no or very low sensitivity to infrared (IR) light. In this case, the brightness of the normal light image generated by the normal light image processing unit 932 is high, and the brightness of the fluorescent image generated by the fluorescent light image processing unit 933 is low or zero. As a result, the brightness of the fluorescent light image (α × β × IR_Y) is low in the brightness of the superimposed image, and the brightness of the normal light ((1-α) × WLI_Y) becomes dominant, and brightness control of the normal light image and fluorescent image is performed so that the brightness of the normal light is essentially brought closer to the target brightness. This allows the superimposed image, in which the brightness of normal light is dominant, to be observed at or close to the target brightness. In this way, the superimposed image, in which the brightness of normal light is dominant, can be observed at the target brightness.

[0107] 6 shows a display example of a superimposed image 210 that has high sensitivity to white light and is generated according to this embodiment. Image 211, which is a part of the superimposed image 210, is an image of a region observed by a user such as a doctor. The brightness of normal light is dominant. The superimposed image 210 is displayed with reduced noise.

[0108] On the other hand, FIG. 7 shows an example of generating a superimposed image D1 using related technology by adjusting the light intensity of a white light source (white light source) and a near-infrared light source (near-infrared light source). The numerous dots in the figure represent noise. Superimposed image D1 is an image acquired before a fluorescent substance is injected into the object of observation. The brightness control according to this embodiment, i.e., feedback control of brightness using brightness information of the superimposed image, is not performed. In the related technology, the light intensity of the near-infrared light source and the gain setting of the fluorescence image are increased, causing the fluorescence image to become too bright, resulting in increased noise in the superimposed image.

[0109] If a fluorescent substance is injected into the living body before the fluorescent substance is injected, the sensitivity of the observed subject to IR suddenly increases. In this case, the brightness of the fluorescent image increases, and the proportion of the fluorescent brightness in the brightness of the superimposed image increases. By controlling the brightness of the normal light image and the fluorescent image so that the brightness of the superimposed image approaches the target brightness, the brightness of the superimposed image is maintained at or near the target brightness. Furthermore, the ratio of the values ​​of the first control parameter for normal light and the second control parameter for fluorescent light to be controlled is maintained, so the fluorescent brightness in the superimposed image does not become too high or noise occurs. This allows the observation of a superimposed image with a good balance between the brightness of normal light and the brightness of fluorescent light. In other words, users such as doctors do not experience discomfort due to sudden large fluctuations in the brightness of the superimposed image and the screen appearing dazzling.

[0110] 8 shows a superimposed image 220 generated by this embodiment, and shows an example of the display of the superimposed image 220 generated when a fluorescent substance is injected into a living body and the proportion of fluorescent light luminance increases. Areas where the intensity of the fluorescent component is high (where IR_Y is large) are indicated by diagonal lines. The image displayed has a good balance between the luminance of normal light and the luminance of fluorescent light, without the luminance of fluorescent light becoming too high.

[0111] Figure 9 shows an example of a superimposed image D2 generated by a related art method for adjusting the brightness of the near-infrared light emitted from a near-infrared light source in accordance with the brightness of the white image. While this method prevents the fluorescent image from becoming too bright or generating a lot of noise, the brightness of the fluorescent image is not properly reflected in the superimposed image, resulting in an overly bright superimposed image. Portions of image 211 that were clearly visible in Figure 6 are not visible or are difficult to see in Figure 9.

[0112] After this, as time passes and the IR sensitivity of the object of observation decreases, the brightness of the superimposed image becomes dominated by the brightness of normal light again. In this case, brightness control is performed to bring the brightness of the superimposed image closer to the target brightness, resulting in the superimposed image being observed in which the brightness of normal light is dominant. In this case, too, the superimposed image is observed at or near the target brightness, so users such as doctors do not feel uncomfortable due to sudden large fluctuations in brightness.

[0113] (Variation) In the first embodiment, the set of first and second control parameters to be adjusted was at least one of a set of the light intensity of normal light and fluorescent light, a set of the exposure intensity of normal light and fluorescent light, a set of analog gains of an image of normal light and fluorescent light, and a set of digital gains of an image of normal light and fluorescent light. All of these sets are sets of the same type of parameters. As a modified example, sets of different types of parameters may be adjusted. For example, the set of first and second control parameters to be adjusted may be a set of the light intensity of normal light and the exposure intensity of fluorescent light, or a set of the analog gain of normal light and the digital gain of fluorescent light. In this case, correspondence data indicating the correspondence between the adjustment factor for the light intensity of normal light and the adjustment factor for the exposure intensity of fluorescent light may be stored in the memory 92 or the storage unit 97 in the form of a table or function. The control unit 94 may adjust the light intensity of normal light and the exposure intensity of fluorescent light based on the correspondence data.

[0114] (Second embodiment) In the first embodiment, brightness control of normal light images and fluorescence images was performed by executing at least one of four types of brightness control (light intensity control, exposure intensity control, analog gain control, and digital gain control). Specifically, in light intensity control, the normal light intensity and IR light intensity were adjusted as the first control parameter and the second control parameter, respectively. In exposure intensity control, the normal light exposure intensity and fluorescence exposure intensity were adjusted as the first control parameter and the second control parameter, respectively. In analog gain control, the analog gain of the normal light image and the analog gain of the fluorescence image were adjusted as the first control parameter and the second control parameter, respectively. In digital gain control, the digital gain of the normal light image and the digital gain of the fluorescence image were adjusted as the first control parameter and the second control parameter, respectively.

[0115] In the second embodiment, priorities are set for at least two of these four types of brightness control, and the at least two types of brightness control are executed in accordance with the priorities.

[0116] Specifically, the first control parameters include at least two of a first light intensity adjustment parameter that adjusts the light intensity of the first light source 11, a first exposure intensity adjustment parameter that adjusts the exposure intensity of normal light (reflected light) in the first imaging unit, a first analog gain adjustment parameter that adjusts the analog gain of a first image signal obtained by imaging in the first imaging unit, and a first digital gain adjustment parameter that adjusts the digital gain of the first image signal. The control unit 94 sets a first priority for the at least two parameters.

[0117] The second control parameters include at least two of a second light intensity adjustment parameter that adjusts the light intensity of the second light source 12, a second exposure intensity adjustment parameter that adjusts the exposure intensity of the fluorescence in the second image capturing unit, a second analog gain adjustment parameter that adjusts the analog gain of a second image signal obtained by capturing an image in the second image capturing unit, and a second digital gain adjustment parameter that adjusts the digital gain of the second image signal. The control unit 94 sets a second priority order for the at least two parameters.

[0118] For example, when the luminance information of the superimposed image is greater than the target luminance information and needs to be lowered, the control unit 94 prioritizes light intensity control as follows: first, exposure amount control as second. That is, the first priority is given to the first light intensity adjustment parameter and the first exposure amount adjustment parameter, and the second priority is given to the second light intensity adjustment parameter and the second exposure amount adjustment parameter.

[0119] For example, if the exposure amount (exposure time) is reduced (the electronic shutter is closed), the amount of information in each image is reduced by the amount of exposure time, which can cause a sense of discontinuity to the user viewing the displayed video (video). For this reason, the light source device's brightness control is first used to reduce the amount of light. After the amount of light falls below a threshold and the brightness of the emitted light is suppressed to a certain extent, the exposure amount is reduced (the electronic shutter is closed). This makes it less likely that a sense of discontinuity will occur in the image.

[0120] A condition may be that the first control parameter and the second control parameter adjust the same type of parameter. In this case, the first priority and the second priority may be combined into one, with the first order being (first light intensity adjustment parameter, second light intensity adjustment parameter), and the second order being (first exposure amount adjustment parameter, second exposure amount adjustment parameter).

[0121] In addition, when the brightness information of the superimposed image is smaller than the target brightness information and it is necessary to increase the brightness information, the control unit 94 prioritizes the following: first: light intensity control, second: gain control (analog gain control), third: gain control (digital gain control), and fourth: exposure control.

[0122] That is, the first priority is given to the first light intensity adjustment parameter, the first analog gain adjustment parameter, the first digital gain adjustment parameter, and the first exposure amount adjustment parameter, and the second priority is given to the second light intensity adjustment parameter, the second analog gain adjustment parameter, the second digital gain adjustment parameter, and the second exposure amount adjustment parameter, in that order.

[0123] Increasing the exposure amount (extending the exposure time) can increase the brightness of the captured image, but it also reduces the frame rate. This may affect the quality of the video (moving image). For this reason, exposure control is given the lowest priority. Gain control can increase the brightness of the signal, but it may also emphasize noise. Therefore, brightness control of the light source device is given the highest priority. In gain control, the digital gain used in image processing may emphasize noise compared to the analog gain used in the imaging unit, so analog gain control is given priority over digital gain control. The timing of switching the type of brightness control can be determined by any method that can achieve the purpose, such as based on the difference between the brightness information of the superimposed image and the target brightness information. For example, the larger the absolute value of the difference, the higher the priority of the brightness control to be performed. In this case, the type of brightness control to be used may be determined using a table that associates the range of the absolute value of the difference with the type of brightness control to be applied. Alternatively, these brightness control methods may be performed cyclically a predetermined number of times in order of priority. The type of brightness control to be used may also be switched by other methods.

[0124] A condition may be that the first control parameter and the second control parameter adjust the same type of parameter. In this case, the first priority and the second priority may be combined into one, with the first order being (first light intensity adjustment parameter, second light intensity adjustment parameter), the second order being (first analog gain adjustment parameter, second analog gain adjustment parameter), the third order being (first digital gain adjustment parameter, second digital gain adjustment parameter), and the fourth order being (first exposure amount adjustment parameter, second exposure amount adjustment parameter).

[0125] (Third embodiment) In the first or second embodiment, when controlling the luminance information of the superimposed image to match or approach the target luminance information, a statistical value such as the average value of the luminance values ​​(pixel values) of all pixels of the superimposed image is used as the luminance information of the superimposed image. In the second embodiment, another example of calculating the luminance information of the superimposed image is shown.

[0126] [Luminance information calculation example 1] The luminance information of the superimposed image may be an average of the luminance values ​​of a part of the target image of the superimposed image, rather than all pixels of the superimposed image. For example, the target image may be an image of a part of the superimposed image including the center.

[0127] FIG. 10A shows an example in which an image portion (first image portion) including the center of the superimposed image 301 is set as the target image 302 as the first position of the superimposed image 301. The first position is not limited to the center and may be determined arbitrarily depending on the area observed by the user. For example, eye tracking may be used to detect the position at which the user focuses in the superimposed image 301, and the detected position may be set as the first position. For example, an eye tracking device may be provided that irradiates the user's eyes with infrared light, captures an image of the area including the user's eyes, and analyzes the captured image to detect the first position at which the user focuses. The control unit 94 calculates an average brightness value using the target image 302 and sets the calculated average brightness information for the superimposed image. When the target image 302 is an area observed by a user such as a doctor, brightness adjustment (adjustment of the first control parameter for normal light and the second control parameter for fluorescence) is performed to match the observation target at the center. Therefore, even if the peripheral image outside the target image 302 includes an image of forceps or halation, the central image is not affected by the surrounding image. In other words, the central image can be prevented from becoming brighter due to the influence of the surrounding image.

[0128] Weighting may be applied to the target image (first image portion) and a part or all of the surrounding image (second image portion) of the target image. In this case, a weighted average value is calculated for the entire superimposed image, and this weighted average value is used as brightness information.

[0129] FIG. 10B shows an example of calculating a weighted average value. The control unit 94 sets the weight of the pixels of the target image 302 in the superimposed image 301 as a first weight W1, and the weight of the peripheral image 303, a portion of the image surrounding the target image 302, as a second weight W2. The control unit 94 multiplies the luminance value of each pixel of the target image 302 by W1, and multiplies the luminance value of each pixel of the peripheral image 303 by W2. The control unit 94 calculates the weighted average value by summing all the luminance values ​​multiplied by the weights W1 and W2 and dividing the sum by the total number of pixels in the superimposed image 301. The control unit 94 may adjust the values ​​of W1 and W2 depending on the average luminance of the target image 302 and the average luminance of the peripheral image 303, etc. The weight W1 of the target image and the weight W2 of the peripheral image may be fixed in advance.

[0130] According to the method of FIG. 10(B), the first and second control parameters can be adjusted taking into consideration not only the luminance (brightness) of the central portion of the superimposed image but also the luminance (brightness) of the peripheral image. For example, when a user such as a doctor observes a lumen in the central portion of the screen, the central portion of the image (superimposed image) displayed on the screen tends to be dark and the peripheral portion bright. In this case, by increasing the weight W2 of the peripheral portion of the superimposed image, the luminance control of this embodiment adjusts the central portion to be bright as well. On the other hand, when observing perpendicularly to the wall surface, the central portion of the image tends to be bright and the peripheral portion dark. In this case, by increasing the weight W1 of the central portion of the superimposed image, the luminance control of this embodiment adjusts the peripheral portion to be bright as well.

[0131] [Luminance information calculation example 2] The control unit 94 may calculate the average value of the luminance values ​​of the superimposed image by excluding specific pixels or specific regions from the superimposed image. For example, the image in the area masked by the field of view mask 27 (see FIG. 2) of the superimposed image may be excluded, and the average value of the luminance values ​​of the superimposed image may be calculated based on the luminance values ​​of the image in the unmasked area.

[0132] FIG. 11 shows an image 312 of a masked region from a superimposed image 311, and an image 313 of the remaining circular region (observation region). The control unit 94 can identify the image 312 of the masked region by, for example, detecting a region whose brightness value is less than a threshold (first threshold). That is, the control unit 94 can identify the image 313 of the observation region by detecting pixels whose brightness value is equal to or greater than the threshold (first threshold). The control unit 94 calculates the average brightness value using the image 313 of the observation region that includes pixels whose brightness value is equal to or greater than the threshold. By excluding the image 312 of the masked region, it is possible to calculate brightness information while excluding image portions with low brightness. For example, since the masked region is not observed by a user such as a doctor, by excluding the masked region from the brightness adjustment, only the region effective for observation can be subject to brightness control.

[0133] [Luminance information calculation example 3] The control unit 94 may fix (clip) the luminance values ​​of pixels in the superimposed image whose luminance values ​​are saturated at a specific value, and calculate the average luminance value of the entire superimposed image after clipping as luminance information.

[0134] 12 shows images 322 and 323 in which the brightness values ​​are saturated in the superimposed image 321. That is, the brightness values ​​(brightness values) of the pixels included in the images 322 and 323 are saturated. The brightness values ​​being saturated means, for example, that the brightness values ​​are equal to or greater than a threshold value (second threshold value) or are at the maximum value. The control unit 94 fixes (clips) the brightness values ​​of each pixel in the images 322 and 323 in which the brightness values ​​are saturated at a specific value. The average brightness values ​​are calculated using the clipped brightness values ​​for the images 322 and 323 and the original pixel brightness values ​​for the other regions.

[0135] The saturated brightness areas in an image are mainly caused by specular reflection of light. Pixels corresponding to specularly reflected light have fairly large brightness values. Therefore, when calculating the average brightness value of the superimposed image, even if the brightness control of the first or second embodiment is performed, in an image with many specular reflection saturation points, the areas other than the saturation points may become quite dark. To alleviate this problem, brightness values ​​exceeding a threshold are clipped (for example, the brightness values ​​of pixels with brightness values ​​greater than a certain value A are replaced with value A). This allows the areas other than the saturation points to be displayed brightly, even in an image with a lot of specular reflection.

[0136] [Luminance information calculation example 4] The weights of each of the RGB are changed to calculate the luminance value (pixel value) of the pixel of the superimposed image. That is, the normal light image (first image) contains luminance values ​​of R (red), G (green), and B (blue) for each pixel. The pixel value (luminance value) of the normal light image is calculated for each pixel based on the luminance values ​​of R, G, and B for each pixel and the weights of R, G, and B, as shown in equation (1). The luminance value for each pixel is then combined with the luminance value for each pixel of the fluorescent image to calculate the luminance value for each pixel of the superimposed image. After setting the weights for each of the RGB colors, the control unit 94 may calculate the luminance information by averaging the luminance values ​​of all pixels in the superimposed image, as in the first embodiment. Alternatively, the luminance information may be calculated by combining at least one of Calculation Examples 1, 2, 3, 5, and 6 with Calculation Example 4. Examples of setting the weights for each of the RGB colors are shown below as (Example 1) to (Example 4).

[0137] (Example 1) The control unit 94 calculates the Y luminance (WLI_Y) by subtracting the coefficient D from the R, G, and B values ​​in the equation (1) for calculating the Y luminance (WLI_Y). R , D G , D B For example, since the inside of a body cavity observed with an endoscope basically contains a lot of red components, in order to increase the sensitivity to red, R may be weighted more heavily than is commonly used in calculating Y luminance. R , D G , D Bis approximately 3:6:1, but D R , D G , D B may be set to a ratio of 4.5:4.5:1. R , D G , D B The values ​​may be 0.45, 0.45, and 0.1.

[0138] (Example 2) When generating a fluorescence image, the color of the fluorescence image (i.e., the color in which the fluorescence is displayed) can be changed. Because the fluorescence emitted from the observation target S does not itself have a specific color (a color that can be distinguished by humans), the color components of the fluorescence image must be determined when generating a fluorescence image. G (green) is generally used as the color component of fluorescence, but it is possible to specify another color, such as B (blue), as the color of the fluorescence image. When superimposing the fluorescence image on the normal light image, the control unit 94 may change the RGB weights according to the color of the fluorescence image.

[0139] For example, if you want to display fluorescent color as G (green), enter D R , D G , D B The ratio is 3:6:1. If the fluorescent display color is B (blue), R , D G , D B The ratio is set to 3:4:3. By changing the weighting of RGB according to the display color of the fluorescence in this way, it is possible to increase the tracking sensitivity to the brightness of the fluorescence.

[0140] (Example 3) The control unit 94 may change the weight of RGB according to the emission color (background color) of the first light source 11 of normal light. For example, when the emission of normal light is white light, R , D G , D B The ratio for red is 3:6:1, and for purple it is 3:4:3. By changing the weight of RGB according to the emitted color in this way, it becomes possible to superimpose images according to the spectral characteristics, and the visibility of the superimposed images can be improved.

[0141] (Example 4) The control unit 94 may change the weights of each RGB color when displaying a normal light image and when displaying a superimposed image in which a normal light image and a fluorescence image are superimposed. That is, the control unit 94 can execute a display mode (first display mode) in which a superimposed image is displayed and a display mode (second display mode) in which a normal light image is displayed. The control unit 94 may select the mode to be executed based on instruction data input by a user such as a doctor via the input unit 95. The control unit 94 changes the weights of R, G, and B depending on whether the first display mode or the second display mode is executed. In the second display mode, the normal light image may be displayed without being superimposed on the fluorescence image, or the superimposed image may be displayed with α set to zero.

[0142] When the second display mode for displaying a normal light image is executed, the normal light image contains little or almost no green (G) signal component, so the weight of the red (R) signal component is increased to calculate the average luminance. For example, R , D G , D B The ratio is 4.5:4.5:1. When the first display mode, which displays the superimposed image, is executed, the weighting is changed according to the color of the fluorescent image. For example, when the fluorescent display color (color component) is set to green (G), the sensitivity of G is required more than that of the normal light image. For this reason, R , D G , D B The ratio of is set to 3:6:1, the same as the commonly used ratio. Also, if you want to change the fluorescent display color (color component) to a color other than green, you can simply increase the ratio corresponding to that color, as described in (Example 2). For example, if the fluorescent display color is blue, D R , D G , D B By changing the ratio to 3:4:3, it becomes possible to increase the sensitivity to the blue.

[0143] [Luminance information calculation example 5] When calculating the luminance information of the superimposed image, pixels are sampled from the superimposed image at a certain interval in the vertical and horizontal directions, and the luminance information is calculated using only the luminance values ​​of the sampled pixels. The luminance information may be calculated by averaging the luminance values ​​of all the sampled pixels, or may be calculated by combining at least one of the other calculation examples 1 to 4 and 6 with this calculation example 5.

[0144] When all pixels of the superimposed images are used, the processing load increases in proportion to the number of pixels, and the circuit size or memory capacity also increases. Therefore, by thinning out the pixels to be processed, the processing load can be reduced and the circuit size or memory capacity can also be reduced. For example, when the number of pixels of the superimposed images (or the number of pixels of the imaging unit 52) ​​is low, all pixels are used in the vertical direction of the superimposed images (full pixel readout) and every other pixel is used in the horizontal direction (1 / 2 pixel readout). When the number of pixels of the superimposed images (or the number of pixels of the imaging unit 52) ​​is medium, every other pixel is used in the vertical direction (1 / 2 pixel readout) and every third pixel is used in the horizontal direction (1 / 4 pixel readout). When the number of pixels of the superimposed images (or the number of pixels of the imaging unit 52) ​​is high, every third pixel is used in the vertical direction (1 / 4 pixel readout) and every seventh pixel is used in the horizontal direction (1 / 8 pixel readout).

[0145] [Luminance information calculation example 6] The area of ​​the observation object displayed on the display device 70 that a user (observer), such as a doctor, is observing varies depending on the purpose, the situation, the procedure used by the doctor, etc. The area that the user is observing may be, for example, the center of the screen or the edge of the screen. If halation occurs in the observed area, observation becomes difficult. In this calculation example 6, a weight is assigned to areas with high brightness to calculate brightness information, thereby preventing halation from occurring in any area on the screen.

[0146] FIG. 13 illustrates an example of calculating brightness information by assigning a higher weight to regions with higher brightness. The control unit 94 divides the superimposed image into multiple regions. In the example of FIG. 13, the image is divided into nine regions, including a central region R5 and eight surrounding regions R1 to R4 and R6 to R9. The control unit 94 calculates the average brightness value (average brightness value) for each of the nine regions. A higher weight is assigned to each region in descending order of average brightness value. That is, a higher weight is assigned to a region with a higher average brightness value. The weighted average is calculated by multiplying the average brightness value of each region by the weight. The weighted average may be calculated by multiplying the brightness value of each pixel in each region by the region weight and calculating the weighted average for all pixels in all regions. The calculated weighted average is used as brightness information for the superimposed region. By performing brightness control using this brightness information, the control parameters for normal light and fluorescence (first control parameter, second control parameter) are adjusted according to regions with high brightness or regions where halation occurs. This prevents brightness saturation in the region being observed by a user such as a doctor. 13, the superimposed image is divided into nine regions, but the number of regions may be any number as long as it is equal to or greater than 2. Furthermore, the sizes of the divided regions may be the same or different.

[0147] Although several embodiments of the present disclosure have been described, these embodiments are presented as examples and are not intended to limit the scope of the disclosure. These embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the disclosure. These embodiments and their modifications are included within the scope and spirit of the disclosure, as well as the disclosure described in the claims and their equivalents.

[0148] Furthermore, the effects of the present disclosure described in this specification are merely examples, and other effects may also be present.

[0149] The present disclosure can also be configured as follows. [Item 1] a first light source that irradiates a first light having a first wavelength band onto an object; a first imaging unit that exposes light reflected from the object to the first light and generates a first image signal; a first image processing unit that performs first image processing based on the first image signal to generate a first image; a second light source that irradiates the subject with second light having a second wavelength band different from the first wavelength band and including an excitation wavelength of a fluorescent substance contained in the subject; a second imaging unit that exposes fluorescence generated from the subject to the second light and generates a second image signal; a second image processing unit that performs second image processing based on the second image signal to generate a second image; a superimposed image generating unit that generates a superimposed image by superimposing the first image and the second image; a control unit that adjusts the luminance of the first image and the luminance of the second image by adjusting a first control parameter that controls at least one of the first light source, the first imaging unit, and the first image processing unit, and a second control parameter that controls at least one of the second light source, the second imaging unit, and the second image processing unit, based on luminance information of the superimposed image; An image processing device comprising: [Item 2] The control unit adjusts the first control parameter and the second control parameter based on a difference between luminance information of the superimposed image and target luminance information of the superimposed image. Item 1. An image processing device according to item 1. [Item 3] The control unit adjusts the first control parameter and the second control parameter to make the difference zero or equal to or less than a threshold. Item 3. The image processing device according to item 2. [Item 4] The control unit adjusts the first control parameter and the second control parameter based on a condition regarding a ratio between the first control parameter and the second control parameter. Item 3. The image processing device according to any one of items 1 to 3. [Item 5] The control unit sets a difference between a ratio of the first control parameter to the second control parameter before adjustment and a ratio of the first control parameter to the second control parameter after adjustment to zero or a threshold value or less. Item 5. The image processing device according to item 4. [Item 6] the first control parameter is a parameter for adjusting the light amount of the first light source, The second control parameter is a parameter for adjusting the light amount of the second light source. 6. The image processing device according to any one of items 1 to 5. [Item 7] the first control parameter is a parameter for adjusting an exposure amount of reflected light of the first light in the first imaging unit, The second control parameter is a parameter for adjusting the amount of exposure of the fluorescent light in the second image capturing unit. 7. The image processing device according to any one of items 1 to 6. [Item 8] the first control parameter is a parameter for adjusting an analog gain of the first image signal in the first imaging unit, the second control parameter is a parameter for adjusting an analog gain of a second image signal in the second imaging unit. 8. The image processing device according to any one of items 1 to 7. [Item 9] the first control parameter is a parameter for adjusting a digital gain of the first image signal in the first image processing unit, the second control parameter is a parameter for adjusting a digital gain of the second image signal in the second image processing unit. 9. The image processing device according to any one of items 1 to 8. [Item 10] the first control parameters include at least two of a first light intensity adjustment parameter that adjusts the light intensity of the first light source, a first exposure intensity adjustment parameter that adjusts the exposure intensity of reflected light of the first light in the first imaging unit, a first analog gain adjustment parameter that adjusts an analog gain of a first image signal in the first imaging unit, and a first digital gain adjustment parameter that adjusts a digital gain of the first image signal in the first image processing unit, the second control parameters include at least two of a second light intensity adjustment parameter that adjusts the light intensity of the second light source, a second exposure intensity adjustment parameter that adjusts the exposure intensity of the fluorescence in the second image capturing unit, a second analog gain adjustment parameter that adjusts an analog gain of the second image signal in the second image capturing unit, and a second digital gain adjustment parameter that adjusts a digital gain of the second image signal in the second image processing unit, the control unit adjusts the at least two of the first light amount adjustment parameter, the first exposure amount adjustment parameter, the first analog gain adjustment parameter, and the first digital gain adjustment parameter in a first priority order; The control unit adjusts at least two of the second light amount adjustment parameter, the second exposure amount adjustment parameter, the second analog gain adjustment parameter, and the second digital gain adjustment parameter in a second order of priority. 10. The image processing device according to any one of items 1 to 9. [Item 11] the first priority is the first light amount adjustment parameter and the first exposure amount adjustment parameter, in that order; The second priority order is the second light amount adjustment parameter and the second exposure amount adjustment parameter. Item 11. An image processing device according to item 10. [Item 12] the first priority order is the first light amount adjustment parameter, the first analog gain adjustment parameter, the first digital gain adjustment parameter, and the first exposure amount adjustment parameter, in that order; The second priority order is the second light amount adjustment parameter, the second analog gain adjustment parameter, the second digital gain adjustment parameter, and the second exposure amount adjustment parameter. Item 11. An image processing device according to item 10. [Item 13] The luminance information of the superimposed image is the average luminance of the superimposed image. 13. The image processing device according to any one of items 1 to 12. [Item 14] the superimposed image includes a first image portion including a first position of the superimposed image and a second image portion surrounding the first image portion; the control unit sets a first weight to the first image portion and a second weight to the second image portion; calculating a weighted average of the luminance of the first image portion and the luminance of the second image portion based on the first weight and the second weight; Item 14. The image processing device according to item 13. [Item 15] The control unit calculates the average luminance based on only an image portion in the superimposed image that has a luminance equal to or greater than a first threshold value. Item 15. The image processing device according to item 13 or 14. [Item 16] The control unit calculates the average brightness by fixing the brightness of pixels in the superimposed image that have a brightness equal to or greater than a second threshold value to the second threshold value. 16. The image processing device according to any one of items 13 to 15. [Item 17] the first image includes red, green and blue intensity values ​​for each pixel; the superimposed image generation unit calculates a luminance value for each pixel of the first image based on the luminance values ​​of the red, the green, and the blue for each pixel of the first image and weights of the red, the green, and the blue; combining the luminance value of each pixel of the first image and the luminance value of each pixel of the second image to calculate the luminance value of each pixel of the superimposed image; The control unit changes the weights of the red, the green, and the blue according to the color of the second image. 17. The image processing device according to any one of items 1 to 16. [Item 18] the first image includes red, green and blue intensity values ​​for each pixel; the superimposed image generation unit calculates a luminance value for each pixel of the first image based on the luminance values ​​of the red, the green, and the blue for each pixel of the first image and weights of the red, the green, and the blue; combining the luminance value of each pixel of the first image and the luminance value of each pixel of the second image to calculate the luminance value of each pixel of the superimposed image; The control unit changes weights of the red, the green, and the blue according to the color of the first light. 18. The image processing device according to any one of items 1 to 17. [Item 19] the control unit selectively executes a first display mode in which the superimposed image is displayed and a second display mode in which the first image is displayed; the first image includes red, green and blue intensity values ​​for each pixel; the superimposed image generation unit calculates a luminance value for each pixel of the first image based on the luminance values ​​of the red, the green, and the blue for each pixel of the first image and weights of the red, the green, and the blue; combining the luminance value of each pixel of the first image and the luminance value of each pixel of the second image to calculate the luminance value of each pixel of the superimposed image; The control unit changes the weights of the red, the green, and the blue depending on whether the first display mode or the second display mode is to be executed. 19. The image processing device according to any one of items 1 to 18. [Item 20] the control unit calculates an average luminance for each of a plurality of regions in the superimposed image, and sets a higher weight to pixels included in a region where the average luminance is higher; Calculating a weighted average of the luminance of each pixel of the superimposed image based on the weight of the pixel for each region. 20. The image processing device according to any one of items 13 to 19. [Item 21] The control unit simultaneously adjusts the first control parameter and the second control parameter. 21. The image processing device according to any one of items 1 to 20. [Item 22] a first step of irradiating a first light having a first wavelength band onto an object; a second step of exposing light reflected from the object to the first light to generate a first image signal; a third step of performing first image processing based on the first image signal to generate a first image; a fourth step of irradiating the subject with second light having a second wavelength band different from the first wavelength band and including an excitation wavelength of a fluorescent substance contained in the subject; a fifth step of exposing the fluorescence generated from the subject to the second light to generate a second image signal; a sixth step of performing second image processing based on the second image signal to generate a second image; a seventh step of generating a superimposed image by superimposing the first image and the second image; an eighth step of adjusting the luminance of the first image and the luminance of the second image by adjusting a first control parameter that controls at least one of the first step, the second step, and the third step and a second control parameter that controls at least one of the fourth step, the fifth step, and the sixth step based on luminance information of the superimposed image; An image processing method comprising: [Explanation of symbols]

[0150] 100 endoscope device, 10 light source device, 11 first light source, 12 second light source, 20 insertion member (endoscope body), 21 tip portion, 22 connection portion, 23 base end portion (eyepiece portion), 23 base end portion, 24A exit portion, 24B entrance portion, 25 observation optical system, 25a to 25h optical lens, 27 field of view mask, 30 light guide, 31 one end, 32 other end, 50 camera head, 51 lens unit, 52 imaging unit, 53 communication unit, 70 display device, 80 transmission cable, 90 control device, 91 communication unit, 92 memory, 93 observation image generation unit, 94 control unit, 95 input unit, 96 output unit, 97 storage unit, 210 superimposed image, 521 excitation light cut filter, 522 imaging element, 523 signal processing unit, 540 Gain adjustment unit (AGC circuit), 931 memory controller, 932 normal light image processing unit, 932 first image processing unit, 933 fluorescence image processing unit, 933 second image processing unit, 934 superimposed image generation unit, 935 display control unit, 940, 940A, 940B gain adjustment unit (AGC circuit), 941 luminance information calculation unit, 942 parameter adjustment unit

Claims

1. a first light source that irradiates a first light having a first wavelength band onto an object; a first imaging unit that exposes light reflected from the subject to the first light and generates a first image signal; a first image processing unit that performs first image processing based on the first image signal to generate a first image; a second light source that irradiates the subject with second light having a second wavelength band different from the first wavelength band and including an excitation wavelength of a fluorescent substance contained in the subject; a second imaging unit that exposes fluorescence generated from the subject to the second light and generates a second image signal; a second image processing unit that performs second image processing based on the second image signal to generate a second image; a superimposed image generating unit that generates a superimposed image by superimposing the first image and the second image; a control unit that adjusts the luminance of the first image and the luminance of the second image by adjusting a first control parameter that controls at least one of the first light source, the first imaging unit, and the first image processing unit, and a second control parameter that controls at least one of the second light source, the second imaging unit, and the second image processing unit, based on luminance information of the superimposed image; An image processing device comprising:

2. The control unit adjusts the first control parameter and the second control parameter based on a difference between luminance information of the superimposed image and target luminance information of the superimposed image. The image processing device according to claim 1 .

3. The control unit adjusts the first control parameter and the second control parameter to make the difference zero or equal to or less than a threshold. The image processing device according to claim 2 .

4. The control unit adjusts the first control parameter and the second control parameter based on a condition regarding a ratio between the first control parameter and the second control parameter. The image processing device according to claim 1 .

5. The control unit sets a difference between a ratio of the first control parameter to the second control parameter before adjustment and a ratio of the first control parameter to the second control parameter after adjustment to zero or a threshold value or less. The image processing device according to claim 4 .

6. the first control parameter is a parameter for adjusting the light amount of the first light source, The second control parameter is a parameter for adjusting the light amount of the second light source. The image processing device according to claim 1 .

7. the first control parameter is a parameter for adjusting an exposure amount of reflected light of the first light in the first imaging unit, The second control parameter is a parameter for adjusting the amount of exposure of the fluorescent light in the second image capturing unit. The image processing device according to claim 1 .

8. the first control parameter is a parameter for adjusting an analog gain of the first image signal in the first imaging unit, the second control parameter is a parameter for adjusting an analog gain of a second image signal in the second imaging unit; The image processing device according to claim 1 .

9. the first control parameter is a parameter for adjusting a digital gain of the first image signal in the first image processing unit, the second control parameter is a parameter for adjusting a digital gain of the second image signal in the second image processing unit; The image processing device according to claim 1 .

10. the first control parameters include at least two of a first light amount adjustment parameter that adjusts the light amount of the first light source, a first exposure amount adjustment parameter that adjusts the exposure amount of reflected light of the first light in the first imaging unit, a first analog gain adjustment parameter that adjusts an analog gain of a first image signal in the first imaging unit, and a first digital gain adjustment parameter that adjusts a digital gain of the first image signal in the first image processing unit, the second control parameters include at least two of a second light amount adjustment parameter that adjusts a light amount of the second light source, a second exposure amount adjustment parameter that adjusts an exposure amount of the fluorescence in the second image capturing unit, a second analog gain adjustment parameter that adjusts an analog gain of the second image signal in the second image capturing unit, and a second digital gain adjustment parameter that adjusts a digital gain of the second image signal in the second image processing unit, the control unit adjusts the at least two of the first light amount adjustment parameter, the first exposure amount adjustment parameter, the first analog gain adjustment parameter, and the first digital gain adjustment parameter in a first priority order; The control unit adjusts at least two of the second light amount adjustment parameter, the second exposure amount adjustment parameter, the second analog gain adjustment parameter, and the second digital gain adjustment parameter in a second order of priority. The image processing device according to claim 1 .

11. the first priority is the first light amount adjustment parameter and the first exposure amount adjustment parameter, in that order; The second priority order is the second light amount adjustment parameter and the second exposure amount adjustment parameter. The image processing device according to claim 10.

12. the first priority order is the first light amount adjustment parameter, the first analog gain adjustment parameter, the first digital gain adjustment parameter, and the first exposure amount adjustment parameter, in that order; The second priority order is the second light amount adjustment parameter, the second analog gain adjustment parameter, the second digital gain adjustment parameter, and the second exposure amount adjustment parameter. The image processing device according to claim 10.

13. The luminance information of the superimposed image is information representing the average luminance of the superimposed image. The image processing device according to claim 1 .

14. the superimposed image includes a first image portion including a first position of the superimposed image and a second image portion surrounding the first image portion; the control unit sets a first weight to the first image portion and a second weight to the second image portion; Calculating a weighted average of the luminance of the first image portion and the luminance of the second image portion based on the first weight and the second weight. The image processing device according to claim 13 .

15. The control unit calculates the average luminance based on only pixels in the superimposed image that have a luminance equal to or greater than a first threshold value. The image processing device according to claim 13 .

16. The control unit calculates the average brightness by fixing the brightness of pixels in the superimposed image that have a brightness equal to or greater than a second threshold value to the second threshold value. The image processing device according to claim 13 .

17. the first image includes red, green and blue intensity values ​​for each pixel; the superimposed image generation unit calculates a luminance value for each pixel of the first image based on the luminance values ​​of the red, the green, and the blue for each pixel of the first image and weights of the red, the green, and the blue; calculating a luminance value for each pixel of the superimposed image by combining a luminance value for each pixel of the first image and a luminance value for each pixel of the second image; The control unit changes the weights of the red, the green, and the blue according to the color of the second image. The image processing device according to claim 1 .

18. the first image includes red, green and blue intensity values ​​for each pixel; the superimposed image generation unit calculates a luminance value for each pixel of the first image based on the luminance values ​​of the red, the green, and the blue for each pixel of the first image and weights of the red, the green, and the blue; calculating a luminance value for each pixel of the superimposed image by combining a luminance value for each pixel of the first image and a luminance value for each pixel of the second image; The control unit changes weights of the red, the green, and the blue according to the color of the first light. The image processing device according to claim 1 .

19. the control unit selectively executes a first display mode in which the superimposed image is displayed and a second display mode in which the first image is displayed; the first image includes red, green and blue intensity values ​​for each pixel; the superimposed image generation unit calculates a luminance value for each pixel of the first image based on the luminance values ​​of the red, the green, and the blue for each pixel of the first image and weights of the red, the green, and the blue; calculating a luminance value for each pixel of the superimposed image by combining a luminance value for each pixel of the first image and a luminance value for each pixel of the second image; The control unit changes the weights of the red, the green, and the blue depending on whether the first display mode or the second display mode is to be executed. The image processing device according to claim 1 .

20. the control unit calculates an average luminance for each of a plurality of regions in the superimposed image, and sets a higher weight to pixels included in a region where the average luminance is higher; Calculating a weighted average of the luminance of each pixel of the superimposed image based on the weight of the pixel for each region. The image processing device according to claim 13 .

21. The control unit simultaneously adjusts the first control parameter and the second control parameter. The image processing device according to claim 1 .

22. An image processing method executed by a control device capable of controlling a first light source that irradiates a subject with first light having a first wavelength band, a first imaging unit that exposes the subject to reflected light of the first light to generate a first image signal, a second light source that irradiates the subject with second light having a second wavelength band different from the first wavelength band and including an excitation wavelength of a fluorescent substance contained in the subject, and a second imaging unit that exposes fluorescence generated from the subject to the second light to generate a second image signal, a communication unit of the control device acquiring a first image signal from the first imaging unit and acquiring a second image signal from the second imaging unit; a first image processing unit of the control device performing first image processing based on the first image signal to generate a first image; a second image processing unit of the control device performing second image processing based on the second image signal to generate a second image; a superimposed image generating unit of the control device superimposing the first image and the second image to generate a superimposed image; a control unit of the control device adjusts a first control parameter for controlling at least one of the first light source, the first imaging unit, and the first image processing unit, and a second control parameter for controlling at least one of the second light source, the second imaging unit, and the second image processing unit, based on luminance information of the superimposed image, thereby adjusting the luminance of the first image and the luminance of the second image. Image processing methods.

Images