Processing apparatus, processing system, processing method, and program

Abstract

To automatically adjust brightness while suppressing variation in white balance of a background image even when reflected light of excitation light is included in the background image in fluorescence observation.SOLUTION: A processing apparatus includes: a first obtaining unit configured to obtain first image information relating to brightness adjustment of a first image obtained from light including light of a first light source and light of a second light source different from the first light source; a second obtaining unit configured to obtain second image information relating to brightness adjustment of a second image obtained from light of the second light source; and a control unit configured to control each of intensity of first irradiation light emitted by the first light source and sensitivity of the first image and intensity of second irradiation light emitted by the second light source and sensitivity of the second image. The control unit controls at least one of the intensity of the first irradiation light emitted by the first light source, the sensitivity of the first image, the intensity of the second irradiation light emitted by the second light source, and the sensitivity of the second image, on the basis of the first image information and the second image information so that a ratio of the intensity of the first irradiation light and the intensity of the second irradiation light may fall within a predetermined range.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The disclosed technology relates to a processing device, a processing system, a processing method, and a program. [Background technology]

[0002] When observing an object using a white light source that generates visible light (normal observation), there is a process that automatically controls the light intensity, exposure time, gain applied to the electrical signal generated by receiving light, etc. to keep the brightness of the image obtained by normal observation constant. In addition, when observing fluorescence generated by irradiating an object with X-rays, ultraviolet rays, infrared rays, or visible light as excitation light (fluorescence observation), a similar automatic brightness control is also used.

[0003] On the other hand, for example, in the medical field, the above-mentioned fluorescent observation is sometimes used to superimpose an image obtained by fluorescent observation (fluorescent image) on an image obtained by normal observation (background image).

[0004] Patent Document 1 discloses a technique for automatically adjusting the brightness of a fluorescent image in accordance with a control parameter determined from a background image, as a method for controlling brightness in fluorescent observation.

[0005] The technology of Patent Document 1 proposes a control method in which light intensity, exposure time (shutter), and gain are control parameters as brightness control information, but does not restrict the priority of each control method. For example, when increasing the gain, noise components also increase and image quality may deteriorate, so when brightening an image, it is common to preferentially use a control method using a control parameter other than gain. In a fluorescent image, if sufficient brightness cannot be obtained by simply increasing the intensity of the excitation light, the gain may also be increased to adjust the brightness. Also, when darkening an image with an increased gain, it is common to decrease the gain. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP2018-182741A Summary of the Invention [Problem to be solved by the invention]

[0007] However, when brightness adjustment is performed using different control methods for a fluorescent image and a background image, the white balance of the background image may be shifted. When reflected light of excitation light is mixed into the background image, if the brightness of the fluorescent image is adjusted by gain without changing the intensity of the excitation light, and the brightness of the background image is adjusted by the intensity of white light (visible light) in the light source, the white balance of the background image will be shifted. For example, if the reflected light of excitation light is blue, lowering the intensity of the white light without changing the intensity of the excitation light will make the background image relatively blue.

[0008] In view of the above-mentioned conventional techniques, the disclosed technique aims to automatically adjust brightness while suppressing fluctuations in the white balance of a background image even when reflected light of excitation light is mixed into the background image. [Means for solving the problem]

[0009] A processing device according to one aspect of the disclosed technology includes: a first acquisition unit that acquires first image information related to brightness adjustment of a first image obtained from light including a first light source and a second light source different from the first light source; a second acquisition unit that acquires second image information related to brightness adjustment of a second image obtained from the light of the second light source; a control unit that controls the intensity of a first illumination light emitted by the first light source and the sensitivity of the first image, and the intensity of a second illumination light emitted by the second light source and the sensitivity of the second image, The control unit controls at least one of the intensity of the first irradiation light irradiated by the first light source, the sensitivity of the first image, the intensity of the second irradiation light irradiated by the second light source, and the sensitivity of the second image based on the first image information and the second image information so that the ratio between the intensity of the first irradiation light and the intensity of the second irradiation light is within a predetermined range. Effect of the Invention

[0010] According to the disclosed technology, even when reflected light of the excitation light is mixed into the background image, it is possible to automatically adjust the brightness while suppressing fluctuations in the white balance of the background image. [Brief description of the drawings]

[0011] [Figure 1] 1 is a block diagram showing the configuration of an imaging apparatus according to a first embodiment. [Diagram 2] 5 is a flowchart showing a brightness adjustment process of the imaging device according to the first embodiment. [Diagram 3] 3A and 3B are diagrams showing an example of time-division control of the imaging apparatus according to the first embodiment. [Figure 4] FIG. 11 is a block diagram showing the configuration of an imaging apparatus according to a second embodiment. [Diagram 5] FIG. 11 is a block diagram showing the configuration of an imaging apparatus according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0013] [First embodiment] An imaging device according to a first embodiment will be described with reference to the block diagram of Fig. 1. The imaging device 1 according to the first embodiment includes an imaging unit 11, an image processing unit 13, and an image generating unit 18. The processing system STM includes the imaging device 1, a light source 23, and a display device 30.

[0014] The imaging unit 11 includes a prism 111, shutters 112, 113, 114, and 115, and image sensors 116, 117, 118, and 119.

[0015] Image processing section 13 includes gain adjustment sections 131, 132, 133, a background gain adjustment section 134 (first image gain adjustment section), a background evaluation information acquisition section 135, a background total gain acquisition section 136 (first acquisition section), a fluorescence total gain acquisition section 138 (second acquisition section), and a control section 139. Here, "gain" refers to the amount of amplification of an electrical signal (imaging signal).

[0016] The image generating section 18 includes a background image generating section 15 , a fluorescent light image generating section 16 , and an image combining section 17 .

[0017] Furthermore, the light source 23 includes a white light source 20 (first light source) and an excitation light source 22 (second light source) whose light intensity can be adjusted based on a control signal output from the processing device 1. The white light source 20 (first light source) is a light source for observing reflected light from the subject P. The white light source 20 outputs white light, which is visible light in which light rays of various wavelengths are mixed. The white light output from the white light source 20 is irradiated onto the subject P. The white light irradiated onto the subject P is reflected, generating reflected light.

[0018] Based on the white light intensity control signal, the control unit 139 in the image processing unit 13 can control the intensity of the white light output from the white light source 20. The intensity of the white light can be adjusted, for example, by the lighting time of the white light.

[0019] When controlling the white light source 20 (first light source), the control unit 139 is capable of controlling the intensity of the first irradiation light by using at least one of the amplitude of the first irradiation light irradiated from the white light source 20 (first light source), the lighting time of the first irradiation light, the pulse width of the first irradiation light, and the pulse density of the first irradiation light.

[0020] The excitation light source 22 (second light source) is a light source for observing fluorescence emitted from a fluorescent substance present in the subject P. The excitation light source 22 outputs, for example, blue excitation light. The excitation light output from the excitation light source 22 is irradiated onto the subject P. Here, it is assumed that the subject P contains a fluorescent substance that is excited by blue excitation light, and red fluorescence is generated from the fluorescent substance. Note that the wavelengths of the excitation light and fluorescence are not limited to these, and the excitation light source 22 may emit green excitation light or infrared fluorescence, for example.

[0021] Based on the excitation light intensity control signal, the control unit 139 can control the intensity of the excitation light output from the excitation light source 22. When controlling the excitation light source 22 (second light source), the control unit 139 can control the intensity of the second irradiation light using at least one of the amplitude of the second irradiation light irradiated from the excitation light source 22 (second light source), the illumination time of the second irradiation light, the pulse width of the second irradiation light, and the pulse density of the second irradiation light.

[0022] (Configuration of imaging unit 11) The imaging unit 11 captures a first image and a second image. Hereinafter, the first image is also referred to as a background image, and the second image is also referred to as a fluorescent image. The first image (background image) is an image obtained from light including light from a white light source 20 (first light source) and light from an excitation light source 22 (second light source) different from the white light source 20. That is, the first image is an image in which light from the white light source 20 (first light source) is mixed with light from the excitation light source 22 (second light source). Moreover, the second image (fluorescent image) is an image obtained from light from the excitation light source 22 (second light source).

[0023] Prism 111 is, for example, a three-plate prism, which splits (disperses) reflected light and fluorescence from subject P into the three primary colors of red light, green light, and blue light. Hereinafter, red light will be referred to as R (red) light, green light as G (green), and blue light as B (blue). Note that since red fluorescence is split into the same path as reflected R (red) light, in reality, shutter 112 and image sensor 116 are used in common, and the reflected R (red) light and red fluorescence are separated by time-division control. For ease of explanation, the illustration shows reflected R (red) light and red fluorescence being virtually processed along separate paths.

[0024] That is, during a period in which reflected light for a background image is being received, R (red) light passes through shutter 112 and is received by image sensor 116. Also, during a period in which red fluorescence for a fluorescence image is being received, the light receiving paths are virtually divided so that the fluorescence passes through shutter 115 and is received by image sensor 119. An example of time-division control will be described later with reference to FIG.

[0025] Shutter 112 adjusts the amount of incident light (hereinafter referred to as exposure time) when R (red) light separated by prism 111 is incident on downstream image sensor 116. Shutter 113 adjusts the exposure time when G (green) light separated by prism 111 is incident on downstream image sensor 117. Shutter 114 adjusts the exposure time when B (blue) light separated by prism 111 is incident on downstream image sensor 118.

[0026] Moreover, the shutter 115 adjusts the exposure time when the red fluorescence separated by the prism 111 enters the image sensor 119 at the subsequent stage.

[0027] As for the specific operation of each shutter, for example, if each shutter is configured as a so-called mechanical shutter, the amount of light incident on the image sensor is adjusted by physically opening and closing the shutter. In other words, when using a mechanical shutter, the "exposure time" is the time during which the image sensor receives light.

[0028] On the other hand, when each shutter is configured as a so-called electronic shutter, the amount of light incident on the image sensor is adjusted by adjusting the amount of charge charged from an empty capacitor. In other words, the "exposure time" in the case of an electronic shutter is the time it takes to recharge the capacitor after the accumulated charge has been discarded.

[0029] The image sensor 116 photoelectrically converts R (red) light to generate an R imaging signal, which is an electrical signal. The image sensor 117 photoelectrically converts G (green) light to generate a G imaging signal, which is an electrical signal. The image sensor 118 photoelectrically converts B (blue) light to generate a B imaging signal, which is an electrical signal. Hereinafter, the R imaging signal, G imaging signal, and B imaging signal are collectively referred to as RGB imaging signals, which are electrical signals obtained by receiving light for a background image.

[0030] The image sensor 119 photoelectrically converts the fluorescence to generate a fluorescence imaging signal, which is an electrical signal.

[0031] Although it is assumed that each of the image sensors described above is, for example, a complementary metal oxide semiconductor (CMOS), other imaging elements such as a charge coupled device (CCD) may also be used.

[0032] (Configuration of image processing unit 13) (Gain adjustment unit 131 and gain adjustment unit 132) The gain adjustment unit 131 acquires an R imaging signal from the image sensor 116, and adjusts the gain of the R imaging signal. Then, the gain adjustment unit 131 outputs the R imaging signal whose gain has been adjusted to the background gain adjustment unit 134. Also, the gain adjustment unit 132 acquires a B imaging signal from the image sensor 118, and adjusts the gain of the B imaging signal. Then, the gain adjustment unit 132 outputs the B imaging signal whose gain has been adjusted to the background gain adjustment unit 134.

[0033] The gain adjustment by the gain adjustment section 131 and the gain adjustment section 132 is so-called white balance adjustment, and the gain adjustment section 131 and the gain adjustment section 132 function as a white balance adjustment section 130 (hereinafter also simply referred to as an adjustment section). The gain adjustment section 131 and the gain adjustment section 132 adjust the gain by adjusting the amplitude of the R imaging signal and the B imaging signal as necessary through general processing. The white balance adjustment section 130 adjusts the white balance of the multiple imaging signals (R imaging signal, G imaging signal, B imaging signal) obtained by the imaging section 11 based on the light separated from the light of the white light source 20 (first light source).

[0034] The gain adjustment section 133 (second image gain adjustment section) adjusts the gain of the imaging signal (fluorescence imaging signal) obtained by the imaging section 11 based on the light of the excitation light source 22 (second light source). The gain adjustment section 133 (second image gain adjustment section) acquires the fluorescence imaging signal from the image sensor 119. The gain adjustment section 133 also acquires a control signal related to gain adjustment of the fluorescence imaging signal from the control section 139, which will be described later. The gain adjustment section 133 then adjusts the gain of the fluorescence imaging signal (hereinafter also referred to as fluorescence gain) based on the control signal related to the gain adjustment. The gain adjustment section 133 outputs the fluorescence imaging signal with the adjusted gain to the fluorescence image generation section 16. The control section 139 controls each of the intensity and background total gain (first image information) of the first irradiation light irradiated by the white light source 20 (first light source) and the intensity and fluorescence total gain (second image information) of the second irradiation light irradiated by the excitation light source 22 (second light source).

[0035] The control unit 139 controls at least one of the intensity of the first irradiation light irradiated by the first light source, the first image information of the first image, the intensity of the second irradiation light irradiated by the second light source, and the second image information of the second image so that the ratio between the intensity of the first irradiation light and the intensity of the second irradiation light is within a predetermined range.

[0036] The control unit 139 adjusts the background total gain (first image information) and the fluorescence total gain (second image information) using at least one of the exposure time of the imaging unit 11, which generates an electrical signal by receiving light, the gain applied to the electrical signal generated by the imaging unit 11, and the gain applied to the electrical signal used by the image generation unit 18 to generate an image (first image, second image).

[0037] When controlling the ratio between the intensity of the first irradiation light and the intensity of the second irradiation light to be a predetermined intensity ratio within the adjustment range of the white light source 20 (first light source) and the adjustment range of the excitation light source 22 (second light source), the control unit 139 controls the intensity of the second irradiation light to be maximum when the intensity of the first irradiation light is maximum.

[0038] In addition, based on the set intensity ratio of the irradiation lights, the adjustment range of the first light source, and the adjustment range of the second light source, the control unit 139 controls the intensity of the first irradiation light so that it does not exceed the adjustment range of the intensity of the second irradiation light.

[0039] Furthermore, when controlling the intensity of the first irradiation light, the control unit 139 controls the intensity of the second irradiation light with priority over the sensitivity of the second image, and when controlling the sensitivity of the first image, the control unit 139 controls the sensitivity of the second image with priority over the intensity of the second irradiation light. Furthermore, when controlling the intensity of the second irradiation light, the control unit 139 controls the intensity of the first irradiation light with priority over the sensitivity of the first image, and when controlling the sensitivity of the second image, the control unit 139 controls the sensitivity of the first image with priority over the intensity of the first irradiation light.

[0040] (Background gain adjustment unit 134) The background gain adjustment section 134 (first image gain adjustment section) adjusts the gains of the multiple imaging signals in the first image, using the multiple imaging signals (R imaging signal, G imaging signal, B imaging signal) whose white balance has been adjusted, and a control signal obtained from the control section 139. The background gain adjustment section 134 obtains the R imaging signal whose gain has been adjusted from the gain adjustment section 131, the G imaging signal from the image sensor 117, and the B imaging signal whose gain has been adjusted from the gain adjustment section 132. The background gain adjustment section 134 also obtains a control signal related to the gain adjustment from the control section 139, which will be described later.

[0041] Then, the background gain adjustment unit 134 adjusts the total gain of the brightness of the background image (hereinafter also referred to as the background image gain or the master gain), which is the overall gain of the R, G and B imaging signals, based on the control signal related to gain adjustment.

[0042] The background gain adjustment unit 134 outputs the R imaging signal, the G imaging signal, and the B imaging signal, in which the gain (master gain) of the background image has been adjusted based on a control signal related to gain adjustment, to the background evaluation information acquisition unit 135 and the background image generation unit 15.

[0043] (Background evaluation information acquisition unit 135) The background evaluation information acquisition unit 135 (first evaluation information acquisition unit) acquires evaluation information on the brightness of the background image (first image) using a plurality of imaging signals (R imaging signal, G imaging signal, B imaging signal) in which the gains of the background image (first image) have been adjusted. The background evaluation information acquisition unit 135 acquires the R imaging signal, G imaging signal, and B imaging signal in which the gains (master gains) of the background image have been adjusted from the background gain adjustment unit 134. Then, the background evaluation information acquisition unit 135 acquires evaluation information on the brightness of the background image, which is a color image (hereinafter also referred to as background evaluation information), based on the R imaging signal, G imaging signal, and B imaging signal in which the master gains have been adjusted. The background evaluation information is, for example, an average value of the luminance values ​​of each pixel of the background image. Alternatively, an integral of the maximum values ​​of the pixel values ​​of RGB may be used. Note that the background evaluation information is not limited to this example, and the background evaluation information acquisition unit 135 can acquire various pieces of information for each pixel of the background image as the background evaluation information. The background evaluation information acquisition unit 135 outputs the acquired evaluation information (background evaluation information) to the background total gain acquisition unit 136.

[0044] (Background total gain acquisition unit 136) The background total gain acquisition unit 136 (first acquisition unit) acquires a total gain (hereinafter referred to as background total gain) as a brightness adjustment related to a background image (first image). The background total gain acquisition unit 136 acquires a background total gain (first image information) as a brightness adjustment related to the background image (first image) by comparing evaluation information (background evaluation information) with preset target information.

[0045] The background total gain acquisition unit 136 (first acquisition unit) acquires (sets) the background total gain (first image information) so as to brighten the background image (first image) when the evaluation information is smaller than the target information, for example. Also, the background total gain acquisition unit 136 (first acquisition unit) sets the background total gain (first image information) so as to darken the background image (first image) when the evaluation information is larger than the target information.

[0046] The background total gain acquisition unit 136 (first acquisition unit) can change the setting of the background total gain (first image information) by adjusting at least one parameter among the intensity of the white light source 20 (first light source), the exposure time in the imaging unit 11, and the gain of multiple imaging signals.

[0047] For example, the background total gain acquired by the background total gain acquisition unit 136 can be expressed by the following formula (1).

[0048] Background total gain = white light intensity x exposure time x master gain (1) The white light intensity, exposure time, and master gain are parameters that adjust the background total gain, and the setting of the background total gain can be changed by adjusting each parameter. By changing the setting of the background total gain, the brightness of the background image can be controlled to be brighter or darker. Here, the exposure time x master gain indicates the sensitivity of the background image (first image). The sensitivity of the background image (first image) can be adjusted by adjusting at least one of the two parameters. For example, when formula (1) is expressed using the sensitivity of the background image (first image), the background total gain is "the white light intensity x the sensitivity of the background image (first image)."

[0049] The control unit 139 can adjust the sensitivity of the background image (first image) using at least one of the exposure time of the imaging unit 11, which generates an electrical signal by receiving light, the gain (amount of amplification of the electrical signal) applied to the electrical signal generated by the imaging unit 11, and the gain (amount of amplification of the electrical signal) applied to the electrical signal used by the image generation unit 18 to generate the background image (first image).

[0050] When the background image (first image) is to be brightened, the background total gain acquisition section 136 (first acquisition section) changes the setting of the background total gain (first image information) by adjusting the intensity of the white light source 20 (first light source) as a priority. Also, when the background image (first image) is to be darkened, the background total gain acquisition section 136 (first acquisition section) can change the setting of the background total gain (first image information) by adjusting the gain of the multiple imaging signals (R imaging signal, G imaging signal, B imaging signal) as a priority.

[0051] (Fluorescence total gain acquisition unit 138) The fluorescence total gain acquisition section 138 (second acquisition section) acquires a total gain (hereinafter referred to as fluorescence total gain) as a measure for adjusting the brightness of a fluorescence image (second image). The fluorescence total gain acquisition section 138 acquires a ratio between a background total gain (first image information) as a measure for adjusting the brightness of a background image and a fluorescence total gain (second image information) as a measure for adjusting the brightness of a fluorescence image, as a ratio (hereinafter referred to as background fluorescence ratio, hereinafter also referred to as second ratio) set externally by a user. The fluorescence total gain acquisition section 138 (second acquisition section) acquires the fluorescence total gain (second image information) as a measure for adjusting the brightness of a fluorescence image (second image) using a predetermined ratio (background fluorescence ratio) and the background total gain (first image information).

[0052] Specifically, the background fluorescence ratio Ro can be expressed by the following formula (2): The background fluorescence ratio may be stored in the storage unit 145, and the control unit 139 may obtain the data from the storage unit 145.

[0053] Background fluorescence ratio Ro = Fluorescence total gain Sf / Background total gain Sw (2) Furthermore, the fluorescence total gain acquisition section 138 acquires the background total gain output from the background total gain acquisition section 136, and acquires the fluorescence total gain using the background total gain and the background fluorescence ratio.

[0054] Here, the fluorescence total gain acquired by the fluorescence total gain acquisition section 138 can be expressed by the following formula (3).

[0055] Fluorescence Total Gain = excitation light intensity × exposure time × fluorescence imaging signal gain (3) The intensity of the excitation light, the exposure time, and the gain of the fluorescence imaging signal are parameters for adjusting the fluorescence total gain, and the setting of the fluorescence total gain can be changed by adjusting each parameter. By changing the setting of the fluorescence total gain, the brightness of the fluorescence image can be controlled to be brighter or darker. That is, the fluorescence total gain acquisition section 138 (second acquisition section) can change the setting of the fluorescence total gain (second image information) by adjusting at least one parameter among the intensity of the excitation light source 22 (second light source), the exposure time in the imaging section 11, and the gain of the imaging signal (fluorescence imaging signal). Here, the exposure time × the gain of the fluorescence imaging signal indicates the sensitivity of the fluorescence image (second image). The sensitivity of the fluorescence image (second image) can be adjusted by adjusting at least one of the two parameters. For example, when the formula (3) is expressed using the sensitivity of the fluorescence image (second image), the fluorescence total gain is "the intensity of the excitation light × the sensitivity of the fluorescence image (second image)".

[0056] The control unit 139 can adjust the sensitivity of the fluorescent image (second image) using at least one of the exposure time of the imaging unit 11, which generates an electrical signal by receiving light, the gain (amount of amplification of the electrical signal) applied to the electrical signal generated by the imaging unit 11, and the gain (amount of amplification of the electrical signal) applied to the electrical signal used by the image generation unit 18 to generate the fluorescent image (second image).

[0057] When the brightness of the fluorescent image (second image) is to be increased, the fluorescent total gain acquisition section 138 (second acquisition section) changes the setting of the fluorescent total gain (second image information) by adjusting the intensity of the excitation light source 22 (second light source) as a priority. Also, when the brightness of the fluorescent image (second image) is to be decreased, the fluorescent total gain acquisition section 138 (second acquisition section) can change the setting of the fluorescent total gain (second image information) by adjusting the gain of the imaging signal (fluorescence imaging signal) as a priority.

[0058] (Control unit 139) The control unit 139 controls the intensity of the first irradiation light irradiated by the white light source 20 (first light source) and the sensitivity (product of the exposure time and the master gain) of the background image (first image), and the intensity of the second irradiation light irradiated by the excitation light source 22 (second light source) and the sensitivity (product of the exposure time and the gain of the fluorescence imaging signal) of the fluorescent image (second image). The control unit 139 can control at least one of the intensity of the first irradiation light irradiated by the first light source, the sensitivity of the first image, the intensity of the second irradiation light irradiated by the second light source, and the sensitivity of the second image, based on the background total gain (first image information) and the fluorescent total gain (second image information), so that the ratio between the intensity of the first irradiation light and the intensity of the second irradiation light falls within a predetermined range.

[0059] The control unit 139 receives the background total gain (first image information) and determines which parameter to control among the master gain, the exposure time, and the intensity of the white light source 20. The control unit 139 generates a control signal for controlling the determined parameter.

[0060] Furthermore, the control unit 139 receives the fluorescence total gain (second image information) and determines which parameter to control among the gain of the fluorescence imaging signal, the exposure time, and the intensity of the excitation light source 22. The control unit 139 generates a control signal for controlling the determined parameters. When controlling the intensity of the first irradiation light, the control unit 139 controls the intensity of the second irradiation light with priority over the sensitivity of the second image. When controlling the sensitivity of the first image, the control unit 139 controls the sensitivity of the second image with priority over the intensity of the second irradiation light. When controlling the intensity of the second irradiation light, the control unit 139 controls the intensity of the first irradiation light with priority over the sensitivity of the first image. When controlling the sensitivity of the second image, the control unit 139 controls the sensitivity of the first image with priority over the intensity of the first irradiation light.

[0061] The control unit 139 controls at least one of the white light source 20 and the excitation light source 22 so that the ratio between the intensity of the first irradiation light irradiated from the white light source 20 and the intensity of the second irradiation light irradiated from the excitation light source 22 becomes the set intensity ratio of the irradiation light within the adjustment range of the white light source 20 (first light source) and the adjustment range of the excitation light source 22. When the brightness required to maintain the background fluorescence ratio setting is insufficient by only controlling to maintain the intensity ratio of the irradiation light within the adjustment range of the white light source 20 and the excitation light source 22, the exposure time and the gain may be used in combination.

[0062] -Control background image brightness The control unit 139 acquires the background total gain from the background total gain acquisition unit 136, and determines which of the parameters related to the background total gain (master gain, exposure time, and white light intensity) to use to control the brightness of the background image. Then, the control unit 139 generates a control signal for controlling the brightness of the background image using the determined parameters, and outputs the generated control signal.

[0063] When controlling the gain (master gain) of the background image, the control unit 139 outputs a control signal for controlling the brightness of the background image to the background gain adjustment unit 134. The background gain adjustment unit 134 outputs the R imaging signal, the G imaging signal, and the B imaging signal in which the gain (master gain) of the background image is adjusted based on the control signal related to the gain adjustment. When controlling the exposure time, the control unit 139 outputs a control signal for controlling the brightness of the background image to the imaging unit 11. The shutters 112, 113, and 114 of the imaging unit 11 control the exposure time according to the control signal. When controlling the intensity of the first irradiation light (intensity of the white light) of the white light source 20 (first light source) within the adjustment range of the white light source 20, the control unit 139 generates a control signal for controlling the brightness of the background image and outputs it to the white light source 20 of the light source 23. The white light source 20 controls the output of the light amount according to the control signal.

[0064] - Fluorescence image brightness control The control unit 139 acquires the fluorescence total gain from the fluorescence total gain acquisition unit 138. The control unit 139 also acquires a range setting of the intensity ratio of white light to excitation light (hereinafter referred to as the intensity ratio of irradiation light) set by a user from the outside. The control unit 139 may also separately acquire information indicating the adjustment range of the intensity of white light and information indicating the adjustment range of the intensity of excitation light. By using the adjustment range of the intensity of white light and the adjustment range of the intensity of excitation light in addition to the setting range of the intensity ratio of irradiation light, it is possible to improve the degree of freedom in adjusting the brightness of a fluorescence image, such as fine adjustment of the brightness of a fluorescence image. Note that information indicating the intensity ratio of irradiation light, the adjustment range of the intensity of white light, and the adjustment range of the intensity of excitation light may be stored in the storage unit 145, and the control unit 139 may acquire data from the storage unit 145.

[0065] Then, the control unit 139 uses the range settings of the fluorescence total gain and the intensity ratio of the irradiation light to determine which parameter of the parameters related to the fluorescence total gain (fluorescence gain, exposure time, and excitation light intensity) to use to control the brightness of the fluorescence image. Then, the control unit 139 generates a control signal for controlling the brightness of the fluorescence scene image using the determined parameters, and outputs the generated control signal.

[0066] For example, when controlling the fluorescence gain, the control unit 139 outputs the generated control signal to the gain adjustment unit 133. The gain adjustment unit 133 adjusts the gain of the fluorescence imaging signal based on the control signal. When controlling the exposure time, the control unit 139 outputs the generated control signal to the imaging unit 11. The shutter 115 of the imaging unit 11 controls the exposure time in accordance with the control signal. When controlling the intensity of the second irradiation light (intensity of the excitation light) of the excitation light source 22 within the adjustment range of the excitation light source 22 (second light source), the control unit 139 outputs the generated control signal to the excitation light source 22 of the light source 23. The excitation light source 22 controls the output of the light amount in accordance with the control signal.

[0067] (Image Generation Unit 18) Background image generator 15 The background image generating section 15 (first image generating section) generates a background image (first image) based on a plurality of imaging signals whose gains have been adjusted by the background gain adjusting section 134 (first image gain adjusting section). Every time the background gain adjusting section 134 generates an R imaging signal, a G imaging signal, and a B imaging signal whose master gains have been adjusted, the background image generating section 15 sequentially acquires the R imaging signal, the G imaging signal, and the B imaging signal generated by the background gain adjusting section 134. The background image generating section 15 sequentially generates a background image based on the R imaging signal, the G imaging signal, and the B imaging signal whose master gains have been adjusted.

[0068] Fluorescence image generator 16 The fluorescence image generating section 16 (second image generating section) generates a fluorescence image (second image) based on an imaging signal whose gain has been adjusted by the gain adjusting section 133 (second image gain adjusting section). Every time the gain adjusting section 133 generates a fluorescence imaging signal whose fluorescence gain has been adjusted, the fluorescence image generating section 16 sequentially generates a fluorescence image based on the fluorescence imaging signal generated by the gain adjusting section 133.

[0069] Image synthesis unit 17 Image synthesis unit 17 outputs an image signal of a synthetic image obtained by synthesizing a background image (first image) and a fluorescent image (second image). Image synthesis unit 17 acquires a background image from background image generation unit 15, acquires a fluorescent image from fluorescent image generation unit 16, and synthesizes the background image and the fluorescent image by alpha blending or the like to generate a synthetic image. As a result, an image is obtained in which the fluorescent image (fluorescent component) is superimposed as a marker on the background image, and is displayed on display unit 31 of display device 30.

[0070] (Configuration of display device 30) The imaging device 1 of this embodiment can be connected to the display device 30 via a communication unit (not shown). The communication unit functions as a connection interface between the imaging device 1 and the display device 30, and the communication unit transmits various information in the imaging unit 11, the image processing unit 13, and the image generating unit 18 to the display device 30. The various information includes, for example, a composite image, as well as various parameters (e.g., exposure time, white light intensity, master gain, excitation light intensity, fluorescence gain, etc.) set in the imaging unit 11 and the image processing unit 13 when generating a composite image (background image and fluorescent image).

[0071] The display device 30 has a display unit 31, a display control unit 32, and an operation unit 33. The display unit 31 is composed of, for example, a liquid crystal display or an organic EL display. The display control unit 32 performs display control to cause the display unit 31 to display various information acquired from the imaging device 1. The display control unit 32 performs display control to cause the display unit 31 to display an image in which a fluorescent image (second image) is superimposed on a background image (first image) based on an image signal output from the processing device 1. The operation unit 33 is composed of, for example, a dial, a mouse, an operation button, etc., and inputs various instructions from the user. The display unit 31 and the operation unit 33 may be realized as a touch panel in which they are integrated together.

[0072] The display control unit 32 acquires the composite image generated by the image composition unit 17 via the communication unit, and performs display control to display the acquired composite image on the display unit 31. Furthermore, when generating a composite image (background image and fluorescent image), the display control unit 32 can also perform display control to display the composite image and various parameters set by the imaging unit 11 and the image processing unit 13 on the display unit 31. Based on an input from the operation unit 33, the display control unit 32 can also change various parameters to be displayed together with the composite image, and display the changed parameters on the display unit 31.

[0073] The display control unit 32 can also perform display control to display a screen (user interface screen) for setting external setting parameters for the imaging device 1 on the display unit 31, based on a user's input via the operation unit 33. Here, the external setting parameters include, for example, a background fluorescence ratio, an intensity ratio of irradiation light, an adjustment range of the intensity of white light, an adjustment range of the intensity of excitation light, or a background fluorescence ratio 2 and additional fluorescence gain, which will be described later. For example, the image processing unit 13 of the imaging device 1 can also acquire the external setting parameters input by the user via the communication unit. The external setting parameters acquired via the communication unit may be stored in the storage unit 145, and the control unit 139 may acquire the data from the storage unit 145.

[0074] (Brightness control process flow) Next, the brightness control process performed by the imaging device 1 according to the first embodiment will be described with reference to the flowchart of FIG.

[0075] The operation of the imaging device 1 is assumed to be such that each imaging signal is sequentially acquired at a predetermined sampling timing so that a composite image based on each imaging signal can be displayed as a moving image of, for example, 60 fps. Still images may be acquired instead of moving images.

[0076] (Step S201) In step S201, the ranges of the background fluorescence ratio and the intensity ratio of the irradiated light are set. The ranges of the background fluorescence ratio and the intensity ratio of the irradiated light may be set by acquiring preset default values ​​by the image processing unit 13. The default value for the range of the intensity ratio of the irradiated light may be set so that the intensity of the excitation light is maximum when the intensity of the white light is maximum. Note that in step S201, in addition to the range of the intensity ratio of the irradiated light, information indicating the adjustment range of the intensity of the white light and information indicating the adjustment range of the intensity of the excitation light may also be set.

[0077] Alternatively, a composite image may be generated using a default value as the setting of the background fluorescence ratio, and the user may adjust the brightness of the composite image displayed on the display unit 31 of the display device 30 using, for example, an operation unit 33 such as a dial or a mouse so that the fluorescent site is easily visible against the background. After that, the ratio of the background total gain to the fluorescence total gain at the specific time phase in which the image confirmed by the user pressing a confirmation button or the like is obtained may be set as the background fluorescence ratio.

[0078] Similarly, for setting the range of the intensity ratio of the irradiation light, the user may check the degree of white balance shift and noise level of the composite image displayed on the display unit 31 of the display device 30, and adjust the range to a value that is set as the range of the intensity ratio.

[0079] (Step S202) In step S202, a photometric region is set. The photometric region may be a region manually set by a user and acquired by the image processing unit 13. Alternatively, the image processing unit 13 may use a photometric region having a preset shape and size for an area relative to an image.

[0080] (Step S203) In step S203, the imaging section 11 acquires an R imaging signal, a G imaging signal, a B imaging signal and a fluorescent imaging signal.

[0081] (Step S204) In step S204, the background evaluation information acquisition unit 135 calculates evaluation information on the brightness of the background image. The range for evaluating the brightness of the background image may be the entire background image, or the photometric area set in step S202. The brightness of the background image may be calculated using the average luminance value of each pixel, or the integral of the maximum RGB pixel values.

[0082] (Step S205) In step S205, the background total gain acquisition unit 136 sets a background total gain based on the background evaluation information calculated in step S204 and preset target information.

[0083] (Step S206) In step S206, the fluorescence total gain acquisition unit 138 acquires the fluorescence total gain using the background total gain Sw set in step S205 and the background fluorescence ratio set in step S201. If the background total gain is Sw, the fluorescence total gain is Sf, and the background fluorescence ratio is Ro, the fluorescence total gain acquisition unit 138 can acquire the fluorescence total gain Sf by calculation using the above formula (2).

[0084] (Step S207) In step S207, control unit 139 determines parameters for controlling the brightness of the background image. Control unit 139 acquires the background total gain from background total gain acquisition unit 136, and determines which of the parameters related to the background total gain (master gain, exposure time, and white light intensity) to use to control the brightness of the background image. Note that the parameter to be determined is not limited to one, and multiple parameters may be used in combination.

[0085] (Step S208) In step S208, the control unit 139 determines the intensity of the excitation light from the intensity of the white light determined in step S207, the intensity ratio of the irradiated light set in step S201, and the setting information Sf of the fluorescence total gain.

[0086] For example, if the range setting for the intensity ratio between white light and excitation light is 1 or more and 1.25 or less, when the intensity of the white light is at its maximum, the intensity of the excitation light is also set to its maximum. When the intensity of the white light is 80% of its maximum, the intensity of the excitation light is set in the range of 1 or more and 1.25 or less, based on the intensity of the white light. In other words, the intensity of the excitation light is determined so that the amount of light matches or is close to the setting information Sf of the fluorescence total gain, between 80% or more of the maximum (1 time the intensity of the white light) and 100% or less (1.25 times the intensity of the white light).

[0087] (Step S209) In step S209, the control unit 139 determines parameters (for example, exposure time and gain of the fluorescence imaging signal) that control the brightness of the fluorescence image in addition to the intensity of the excitation light. When the amount of light does not match or approximate the setting information Sf of the fluorescence total gain by adjusting only the intensity of the excitation light, the control unit 139 determines parameters to control the brightness in addition to the intensity of the excitation light.

[0088] When brightening a fluorescent image, for example, if the exposure time is adjusted to result in an amount of light that matches or is close to the setting information Sf of the fluorescent total gain, the control unit 139 determines the exposure time as a parameter. Also, if the amount of light does not match or is close to the setting information Sf of the fluorescent total gain even within the adjustment range of the exposure time, the control unit 139 determines the exposure time and the gain of the fluorescent imaging signal as parameters.

[0089] When the gain of the fluorescence imaging signal has been adjusted and the fluorescence image is to be darkened, if the gain of the fluorescence imaging signal is adjusted to result in an amount of light that matches or is close to the setting information Sf of the fluorescence total gain, the control unit 139 determines the gain of the fluorescence imaging signal as a parameter. Also, if the amount of light does not match or is close to the setting information Sf of the fluorescence total gain even within the adjustment range of the gain of the fluorescence imaging signal, the control unit 139 determines the exposure time and the gain of the fluorescence imaging signal as parameters.

[0090] (Step S210) In step S210, the control unit 139 adjusts the brightness of the background image in accordance with the determined parameters.

[0091] For example, when controlling the master gain, the control unit 139 outputs a control signal for controlling the brightness of the background image to the background gain adjustment unit 134. The background gain adjustment unit 134 adjusts the gain of the background image (master gain), which is the overall gain of the R, G and B imaging signals, based on the control signal.

[0092] When controlling the exposure time, the control unit 139 outputs a control signal for controlling the brightness of the background image to the imaging unit 11. The shutters 112, 113, and 114 of the imaging unit 11 control the exposure time in accordance with the control signal.

[0093] When controlling the intensity of the white light, the control unit 139 outputs a control signal for controlling the brightness of the background image to the white light source 20 of the light source 23. The white light source 20 controls the output of the amount of light in accordance with the control signal.

[0094] (Step S211) In step S211, the control unit 139 adjusts the brightness of the fluorescent image in accordance with the determined parameters.

[0095] For example, when controlling the fluorescence gain, the control unit 139 outputs the generated control signal to the gain adjustment unit 133. The gain adjustment unit 133 adjusts the gain of the fluorescence imaging signal based on the control signal. When controlling the exposure time, the control unit 139 outputs the generated control signal to the imaging unit 11. When controlling the intensity of the excitation light, the control unit 139 outputs the generated control signal to the excitation light source 22 of the light source 23. The excitation light source 22 controls the output of the light amount in accordance with the control signal.

[0096] (Step S212) In step S212, the image synthesis unit 17 generates a synthetic image of the brightness-adjusted background image and the fluorescent image. The synthetic image is displayed on the display unit 31 of the display device 30 so that the user can confirm it. The display control unit 32 of the display device 30 acquires the synthetic image generated by the image synthesis unit 17 via the communication unit, and performs display control to display the acquired synthetic image on the display unit 31.

[0097] After step S212, the process returns to step S203, where the next imaging signals (RGB imaging signal, fluorescent imaging signal) are acquired, and the processes of each step are repeated.

[0098] For example, the control unit 139 may provide a process for determining whether or not pixel information (e.g., luminance) of the background image has changed using background evaluation information, and execute the processes from step S204 to step S212 only if the luminance of the background image has changed.

[0099] That is, if the luminance of the background image does not change, the processes from step S204 to step S212 relating to the brightness adjustment are not performed, thereby reducing power consumption. The process of determining whether the luminance of the background image has changed may be performed, for example, after step S203.

[0100] Furthermore, the background fluorescence ratio and the intensity ratio of the irradiated light may be set after the photometric region is determined. For example, the correspondence between the photometric region and the background fluorescence ratio and the intensity ratio of the irradiated light may be stored in the storage unit 145 of the image processing unit 13. Depending on the photometric region set in step S202, the control unit 139 may refer to the correspondence between the photometric region and the background fluorescence ratio and the intensity ratio of the irradiated light stored in the storage unit, and determine the background fluorescence ratio and the intensity ratio of the irradiated light corresponding to the set photometric region.

[0101] Also, for example, when the control unit 139 determines parameters for controlling the brightness of the background image in step S207, the intensity of the white light may be determined based on the intensity ratio of the irradiation light set in step S201, the adjustment range of the intensity of the white light, and the adjustment range of the intensity of the excitation light.

[0102] For example, when the intensity ratio of the illumination light is set to m (e.g., 2), the adjustment range of the intensity of the white light is 0% or more and 100% or less, and the adjustment range of the intensity of the excitation light is 50% or more and 100% or less, the intensity of the white light is controlled to be 25% or more and 50% or less, and if the brightness required to maintain the background fluorescence ratio setting is insufficient, a parameter for controlling the brightness of the background image may be used in addition to the intensity of the light source.

[0103] Alternatively, the intensity ratio of the irradiation light may be maintained only when the intensity of the white light and the intensity of the excitation light are adjustable within the adjustment range of the white light source 20 (first light source) and the adjustment range of the excitation light source 22 (second light source).

[0104] For example, assume that the intensity ratio of the irradiated light is set to 2, the adjustment range of the intensity of the white light is 0% to 100%, and the adjustment range of the intensity of the excitation light is 50% to 100%. When the control unit 139 controls the intensity of the white light in the range of 0% to 25%, the intensity of the excitation light may be controlled at 50% to maintain the intensity ratio of the irradiated light.

[0105] The intensity ratio of the irradiated light can be set within a range not exceeding the upper limit (100%) of the adjustment range, for example, when the intensity ratio of the irradiated light is doubled and the adjustment range of the intensity of the white light is 25% to 50%, the adjustment range of the intensity of the excitation light can be twice the adjustment range of the intensity of the white light (50% to 100%). Also, when the adjustment range of the intensity of the white light is 50% to 100%, the adjustment range of the intensity of the excitation light is 100% (50% of the lower limit of the adjustment range of the intensity of the white light x 2).

[0106] The imaging device 1 also has a first operation mode in which the ratio between the intensity of the first irradiation light and the intensity of the second irradiation light is kept within a predetermined range, and a second operation mode in which the range of the ratio between the intensity of the first irradiation light and the intensity of the second irradiation light is wider than that of the first operation mode. The control unit 139 of the imaging device 1 can switch between the first operation mode in which the intensity ratio of the irradiation light is kept within a predetermined range, and the second operation mode in which the restriction on the intensity ratio of the irradiation light is relaxed to a range wider than the predetermined range. For example, the control unit 139 can switch between the first operation mode and the second operation mode in accordance with changes in the imaging scene, such as changes in the type of light source or the relative distance between the imaging unit 11 and the subject P.

[0107] The light source 23 may include a white light source 20 and a third light source different from the excitation light source 22. For example, the third light source may be an excitation light source that outputs excitation light of a color different from that of the excitation light source 22 (second light source). The control unit 139 is capable of controlling the intensity of the third irradiation light irradiated by the excitation light source (third light source) and the sensitivity of the fluorescence image (third image) (the product of the exposure time and the gain of the fluorescence imaging signal).

[0108] The control unit 139 is capable of controlling at least one of the intensity of the first irradiation light irradiated by the first light source, the sensitivity of the first image, the intensity of the third irradiation light irradiated by the third light source, and the sensitivity of the third image based on the background total gain (first image information) and the fluorescence total gain (third image information) related to the fluorescence image (third image) so that the ratio between the intensity of the first irradiation light and the intensity of the third irradiation light is within a predetermined range.

[0109] That is, in the first light source, the second light source, and the third light source, the control unit 139 controls at least one of the intensity of the first irradiation light irradiated by the white light source 20 (first light source), the sensitivity of the first image, the intensity of the second irradiation light irradiated by the excitation light source 22 (second light source), and the sensitivity of the second image based on the background total gain (first image information) and the fluorescence total gain (second image information) so that the ratio between the intensity of the first irradiation light and the intensity of the second irradiation light is within a predetermined range.

[0110] Furthermore, the control unit 139 is capable of controlling at least one of the intensity of the first irradiation light, the sensitivity of the first image, the intensity of the third irradiation light irradiated by the third light source, and the sensitivity of the third image based on the background total gain (first image information) and the fluorescence total gain (third image information) related to adjusting the brightness of the third image obtained from the light of the third light source, so that the ratio between the intensity of the first irradiation light and the intensity of the third irradiation light irradiated by the third light source is within a predetermined range.

[0111] The control unit 139 receives the fluorescence total gain (third image information) related to the fluorescence image (third image), and is capable of adjusting the intensity of the third irradiation light irradiated by the excitation light source (third light source) and the sensitivity of the third image of the fluorescence image (third image) (the product of the exposure time and the gain of the fluorescence imaging signal). The control unit 139 determines which parameter to control among the gain of the fluorescence imaging signal, the exposure time, and the intensity of the excitation light source. The control unit 139 is capable of generating a control signal for controlling the determined parameter.

[0112] The ratio between the intensity of the first irradiation light and the intensity of the second irradiation light, and the ratio between the intensity of the first irradiation light and the intensity of the third irradiation light emitted by the third light source can be set to any desired ratio. Both ratios may be the same, or different ratios may be set.

[0113] The control unit 139 may control the intensities of the white light source 20 and the excitation light source 22 in different operation modes for a first combination of the white light source 20 and the excitation light source 22 and a second combination of the white light source 20 and a third light source different from the excitation light source 22. Furthermore, the control unit 139 may perform processing in a first operation mode in which the intensity ratio of the irradiation light between the white light source 20 and the excitation light source 22 is kept within a predetermined range for the first combination, and may perform processing in a second operation mode in which the intensity ratio of the irradiation light is controlled to be constant within a range wider than the predetermined range for the second combination.

[0114] Furthermore, the control unit 139 may perform processing in a first operation mode in which the intensity ratio of the illumination light from the first light source and the second light source is controlled to be constant within a predetermined range when the brightness of the background image and the fluorescent image is equal to or less than a predetermined threshold. Furthermore, the control unit 139 may switch from processing in the first operation mode to processing in a second operation mode in which the intensity ratio of the illumination light is controlled to be constant within a range wider than the predetermined range when the brightness of at least one of the background image and the fluorescent image changes beyond a predetermined threshold.

[0115] For example, in step S208, the control unit 139 may determine the intensity of the excitation light according to the operation mode. In addition, the intensity ratio of the irradiation light may be set to a fixed single value such as m times (e.g., "2 times") rather than a range.

[0116] (Example of time sharing control) Next, an example of time-division control will be described with reference to Fig. 3. Fig. 3 shows, from top to bottom, a timing chart relating to (a) the output timing of white light from the white light source 20, (b) the output timing of excitation light from the excitation light source 22, (c) the exposure timing of each image sensor, (d) the output timing of the imaging signal of each image sensor, and (e) the output timing of a composite image synthesized by the image synthesis unit 17.

[0117] Here, it is assumed that each image sensor (116 to 119) is of the rolling shutter type, and each pixel performs exposure and image signal output in sequence from top to bottom for each row.

[0118] As described above, the reflected R (red) light and the red fluorescence are dispersed in the same direction by the prism 111, and the irradiation timing of the white light source 20 and the excitation light source 22 and the exposure timing of each image sensor are switched and controlled in a time-division manner, so that the reflected R (red) light and the red fluorescence are captured by the image sensors (116, 119), respectively.

[0119] Specifically, the white light source 20 irradiates white light once every two frames in synchronization with the output timing of an image signal from the image sensor (a), while the excitation light source 22 irradiates excitation light every frame (b).

[0120] 3, an imaging signal during the period of exposure timing c1 is output from the image sensor 119 between times T1 and T2. During this period, the white light source 20 is always off, and the imaging signal includes a state in which the excitation light source 22 is on, and the red imaging signal during this period is treated as a fluorescent imaging signal.

[0121] The imaging signals during the exposure timing c2 are output from the image sensors (116, 117, 118) between times T2 and T3. During this period, the imaging signals include the state in which the white light source 20 is on, and the imaging signals (R2, B2, C2) output from the image sensors (116, 117, 118) during this period are acquired as RGB imaging signals.

[0122] The imaging signals (fluorescence 3, 5, 7, 9) output from the image sensor 119 during the periods of exposure timings c3, c5, c7, and c9 can be acquired as fluorescent imaging signals, similar to exposure timing c1. The imaging signals during the periods of exposure timings c4, c6, and c8 can be acquired as RGB imaging signals, similar to exposure timing c2.

[0123] Regarding the output timing of the composite image generated by the image synthesis unit 17, the image synthesis unit 17 outputs the composite image once every two frames in synchronization with the output timing of the imaging signal of the image sensor.

[0124] Specifically, the composite image output from time T4 is a background image (background 2) generated from the RGB imaging signal during exposure timing c2, with half the fluorescent components of the fluorescent images (fluorescence 1, fluorescence 3) taken at exposure timing c1 and exposure timing c3 superimposed as a marker.

[0125] Since the period of exposure timing c2 includes a state in which the excitation light source 22 is on, the B imaging signal may contain reflected light of the excitation light. Also, the R imaging signal may contain fluorescence. The fluorescence contained in the R imaging signal is overwritten with a fluorescent component marker when the composite image is generated, making it less noticeable. As for the reflected light of the excitation light contained in the B imaging signal, by controlling the ratio between the intensity of the white light and the intensity of the excitation light to be within a predetermined range as in the disclosed technology, a composite image is generated in which fluctuations in the white balance of the background image are suppressed.

[0126] According to the first embodiment, the ratio of the intensity of the white light to the intensity of the excitation light is controlled to be within a range of the intensity ratio of the set irradiation light. This makes it possible to automatically adjust the brightness while suppressing fluctuations in the white balance of the background image even when the reflected light of the excitation light is mixed into the background image.

[0127] Second embodiment In the first embodiment, the setting information for the fluorescence total gain is calculated from the background total gain and the background fluorescence ratio. In the second embodiment, the configuration of an imaging device 1 that calculates the setting information for the fluorescence total gain based on the brightness of a fluorescence image will be described with reference to the block diagram in FIG.

[0128] The imaging device 1 of the second embodiment includes a fluorescence evaluation information acquisition section 137 (second evaluation information acquisition section) in addition to the configuration of the first embodiment. The fluorescence evaluation information acquisition section 137 (second evaluation information acquisition section) acquires evaluation information (hereinafter also referred to as fluorescence evaluation information) of the brightness of the fluorescence image (second image) using an imaging signal (fluorescence imaging signal) whose gain has been adjusted by the gain adjustment section 133 (second image gain adjustment section). The fluorescence evaluation information is, for example, an average value of the luminance values ​​of each pixel of the fluorescence image. Alternatively, an integral of the pixel values ​​of the fluorescence image may be used. Note that the fluorescence evaluation information is not limited to this example, and the fluorescence evaluation information acquisition section 137 can acquire various pieces of information for each pixel of the fluorescence image as the fluorescence evaluation information. The fluorescence evaluation information acquisition section 137 outputs evaluation information (fluorescence evaluation information) of the brightness of the fluorescence image (second image) to the fluorescence total gain acquisition section 138.

[0129] The fluorescence total gain acquisition section 138 (second acquisition section) acquires a fluorescence total gain (second image information) related to the brightness adjustment of the fluorescence image (second image) based on evaluation information (fluorescence evaluation information) of the brightness of the fluorescence image (second image). By comparing the evaluation information with preset target information, the fluorescence total gain (second image information) is acquired (set) as related to the brightness adjustment of the fluorescence image (second image). For example, if the fluorescence evaluation information is large compared to the brightness target information, the fluorescence total gain acquisition section 138 sets a fluorescence total gain such that the brightness of the fluorescence image becomes darker. On the other hand, if the fluorescence evaluation information is smaller than the target information, the fluorescence total gain acquisition section 138 sets a fluorescence total gain such that the brightness of the fluorescence image becomes brighter. The fluorescence total gain acquisition section 138 outputs the set fluorescence total gain to the control section 139.

[0130] As in the first embodiment, the control unit 139 may determine parameters for controlling the brightness of the background image and the fluorescent image, and generate control signals for the determined parameters.

[0131] According to the second embodiment, even in an imaging device 1 in which the setting information for the fluorescence total gain is set based on the brightness of a fluorescence image, the ratio between the intensity of white light and the intensity of excitation light is controlled to be within the range of the set intensity ratio of irradiation light. This makes it possible to automatically adjust the brightness while suppressing fluctuations in the white balance of the background image, even when reflected light of excitation light is mixed into the background image.

[0132] (Third embodiment) In the first embodiment, the background fluorescence ratio and the intensity ratio of the irradiated light are set, and parameters for controlling the brightness of the fluorescence image are determined based on these. In the third embodiment, a configuration for controlling the brightness of the fluorescence image using the excitation light intensity and the gain of the fluorescence imaging signal as parameters, with the exposure time being adjusted within an adjustment range, will be described with reference to the block diagram in Fig. 5.

[0133] 5, a "background fluorescence ratio 2" (hereinafter also referred to as the second ratio) is used as a parameter related to the intensity of the excitation light, and an "additional fluorescence gain" is used as a parameter related to the gain of the fluorescence imaging signal. The configuration of an imaging device 1 that uses the background fluorescence ratio 2 and the additional fluorescence gain to adjust the brightness so that the background fluorescence ratio 2 is constant while keeping the intensity ratio of the irradiated light within a predetermined range will be described.

[0134] The imaging device 1 according to the third embodiment includes a background gain control section 140, a provisional fluorescence total gain acquisition section 141, a provisional fluorescence gain control section 142, and an additional fluorescence gain addition section 143, instead of the fluorescence total gain acquisition section 138 and the control section 139 in the configuration of the first embodiment. Here, the control section 144 of the third embodiment includes the background gain control section 140, the provisional fluorescence gain control section 142, and the additional fluorescence gain addition section 143. The provisional fluorescence total gain acquisition section 141 (second acquisition section) acquires a setting (hereinafter also referred to as a first setting) for adjusting the background fluorescence ratio 2 (second ratio). The provisional fluorescence total gain acquisition section 141 (second acquisition section) acquires the provisional fluorescence total gain (second image information) using the background total gain (first image information) and the background fluorescence ratio 2 (second ratio) adjusted by the first setting.

[0135] Furthermore, the control unit 144 (additional fluorescent light gain adder 143) acquires a setting (hereinafter also referred to as a second setting) for adjusting the additional gain relative to the sensitivity of the second image.

[0136] The control unit 144 (background gain control unit 140) controls the intensity of the first irradiation light and the sensitivity of the first image based on the background total gain (first image information), and the control unit 144 (temporary fluorescence gain control unit 142) controls the intensity of the second irradiation light and the sensitivity of the second image based on the temporary fluorescence total gain (second image information). The control unit 144 (additional fluorescence gain adder 143) further adds the additional gain adjusted by the second setting to the sensitivity of the second image. The sensitivity of the second image is indicated by the product of the exposure time and the gain of the fluorescence imaging signal, and by adding the additional gain to the sensitivity of the second image, the additional gain is added to the gain of the fluorescence imaging signal. In other words, the additional gain is added to the gain applied to the electrical signal (fluorescence imaging signal) used to generate the fluorescent image (second image). The addition of the additional gain increases the gain (amount of amplification) of the electrical signal (fluorescence imaging signal).

[0137] Background fluorescence ratio 2 does not include the additional fluorescence gain related to the gain of the fluorescence imaging signal, and therefore differs from the background fluorescence ratio (ratio (Sf / Sw) of background total gain Sw and fluorescence total gain Sf) described in the first embodiment. Therefore, in order to distinguish it from the background fluorescence ratio described in the first embodiment, in the third embodiment, background fluorescence ratio 2 (R02) is defined by the following equation (4) using temporary fluorescence total gain Sf2 and background total gain Sw.

[0138] Background fluorescence ratio 2 (R02) = provisional fluorescence total gain Sf2 / background total gain Sw (4) Here, the provisional fluorescence total gain Sf2 is the fluorescence total gain Sf described in the first embodiment minus the additional fluorescence gain related to the fluorescence imaging signal, and the fluorescence total gain Sf can be expressed by the following equation (5) using the provisional fluorescence total gain Sf2 and the additional fluorescence gain.

[0139] Fluorescence total gain Sf = provisional fluorescence total gain Sf2 × additional fluorescence gain (5) The control unit 144 acquires a ratio (background fluorescence 2:second ratio) between information (temporary fluorescence total gain) obtained by subtracting the gain (additional fluorescence gain) of the imaging signal of the fluorescence image (second image) from the fluorescence total gain (second image information) indicating the brightness of the fluorescence image (second image), and the background total gain (first image information) indicating the brightness of the background image (first image).

[0140] In the third embodiment, by using the background fluorescence ratio 2 and the additional fluorescence gain, it is possible to separately control the intensity control (intensity adjustment) of the excitation light source 22 (second light source) and the gain adjustment of the fluorescence imaging signal by the gain adjustment section 133 (second image gain adjustment section).

[0141] (Background total gain acquisition unit 136) In the third embodiment, the background total gain acquisition unit 136 (first acquisition unit) acquires a background total gain by adjusting the intensity of the white light source 20 (first light source) or the gain of the imaging signal in a state in which the exposure time of the imaging unit 11 is adjusted. The background total gain acquisition unit 136 acquires a background total gain (first image information) by the product of the intensity of the white light source 20 (second light source) and the gains of the multiple imaging signals (master gains) according to the following formula (6).

[0142] Background Total Gain = excitation light intensity × master gain (6) (Background gain control unit 140) The background gain control unit 140 acquires the background total gain from the background total gain acquisition unit 136, determines which of the parameters related to the background total gain (master gain, white light intensity) to use to control the brightness of the background image, generates a control signal for controlling the brightness of the background image using the determined parameter, and outputs the generated control signal.

[0143] When controlling the gain (master gain) of the background image, the background gain control unit 140 outputs a control signal for controlling the brightness of the background image to the background gain adjustment unit 134. The background gain adjustment unit 134 outputs the R imaging signal, the G imaging signal, and the B imaging signal with the gain (master gain) of the background image adjusted based on the control signal related to the gain adjustment.

[0144] Furthermore, when controlling the intensity of white light, the background gain control unit 140 outputs a control signal for controlling the brightness of the background image to the white light source 20 of the light source 23. The white light source 20 controls the output of the amount of light in accordance with the control signal.

[0145] At this time, when brightening the background image, background gain control unit 140 prioritizes increasing the intensity of the white light over the master gain. When the brightness of the background image required to maintain a background fluorescence ratio setting of 2 is insufficient by simply increasing the intensity of the white light within the adjustment range for adjusting the intensity of white light source 20, background gain control unit 140 generates a control signal for controlling the brightness of the background image and outputs the generated control signal to background gain adjustment unit 134.

[0146] Furthermore, when darkening the background image with the master gain increased, priority is given to lowering the master gain. In other words, when the background image is not dark enough to maintain the background fluorescence ratio of 2 within the adjustment range for adjusting the master gain by simply lowering the master gain, background gain control unit 140 generates a control signal for controlling the intensity of white light source 20, and outputs the generated control signal to white light source 20 of light source 23. This makes it possible to obtain an image with reduced effects of noise in controlling the brightness of the background image.

[0147] Specifically, the background gain control unit 140 controls the brightness of the background image as follows.

[0148] When the brightness of the background image (first image) required to maintain the setting of background fluorescence ratio 2 (ratio) is insufficient while the intensity of the first irradiation light is increased within the adjustment range of the white light source 20 (first light source), the background gain control unit 140 outputs a control signal generated to adjust the brightness of the background image (first image) to be even brighter to the background gain adjustment unit 134 (first image gain adjustment unit).

[0149] When the brightness of the background image is to be darkened in a state in which the brightness of the background image has been adjusted to be bright by the background gain adjustment unit 134, the background gain control unit 140 outputs a control signal to the background gain adjustment unit 134 for adjusting the gain of the multiple imaging signals (R imaging signal, G imaging signal, B imaging signal) to be smaller.

[0150] When the brightness of the background image is adjusted to be dark by the background gain adjustment unit 134 and the brightness of the background image is to be made darker, the background gain control unit 140 outputs a control signal to the white light source 20 (first light source) to adjust the intensity of the first irradiation light to be lowered within the adjustment range of the white light source 20.

[0151] (Temporary fluorescence total gain acquisition unit 141) The temporary fluorescence total gain acquisition unit 141 acquires a background fluorescence ratio 2 that is externally set by a user. The temporary fluorescence total gain acquisition unit 141 acquires the background total gain from the background total gain acquisition unit 136, and acquires the temporary fluorescence total gain from the background total gain and the background fluorescence ratio 2. The temporary fluorescence total gain acquisition unit 141 acquires the temporary fluorescence total gain based on the relationship in formula (4).

[0152] In the third embodiment, the temporary fluorescence total gain acquisition section 141 (second acquisition section) acquires temporary fluorescence total gain by adjusting the intensity of the excitation light source 22 (second light source) or the gain of the imaging signal in a state in which the exposure time of the imaging section 11 has been adjusted. The temporary fluorescence total gain acquisition section 141 acquires temporary fluorescence total gain (second image information) from the product of the intensity and gain (gain of the fluorescence imaging signal) of the excitation light source 22 (second light source) in accordance with the following equation (7).

[0153] Provisional Fluorescence Total Gain = excitation light intensity × fluorescence imaging signal gain (7) (Temporary fluorescent gain control unit 142) The temporary fluorescent gain control unit 142 acquires the temporary fluorescent total gain from the temporary fluorescent total gain acquisition unit 141, and determines which of the parameters related to the temporary fluorescent total gain (gain of the fluorescent imaging signal, intensity of the excitation light) to use to control the brightness of the fluorescent image. The temporary fluorescent gain control unit 142 then generates a control signal for controlling the brightness of the fluorescent scene image using the determined parameters, and outputs the generated control signal.

[0154] When controlling the gain of the fluorescent imaging signal, the temporary fluorescent gain control unit 142 outputs the generated control signal to the additional fluorescent gain addition unit 143 .

[0155] The additional fluorescence gain adder 143 acquires an additional fluorescence gain that is set externally by a user. The additional fluorescence gain adder 143 acquires a control signal related to the gain from the temporary fluorescence gain control unit 142, and generates a control signal by adding the additional fluorescence gain. The gain adjustment unit 133 adjusts the gain of the fluorescent imaging signal based on the control signal from the additional fluorescence gain adder 143.

[0156] Furthermore, when controlling the intensity of the excitation light, the temporary fluorescence gain controller 142 outputs the generated control signal to the excitation light source 22 of the light source 23. The excitation light source 22 controls the output of the amount of light in accordance with the control signal.

[0157] At this time, when brightening the fluorescent light image, the temporary fluorescent light gain control unit 142 prioritizes increasing the intensity of the excitation light over the gain of the fluorescent image capturing signal.

[0158] When the brightness of the fluorescence image (second image) required to maintain the setting of the background fluorescence ratio 2 (ratio) is insufficient with the intensity of the second irradiation light increased within the adjustment range of the excitation light source 22 (second light source), the control unit 144 (temporary fluorescence gain control unit 142) outputs a control signal generated to adjust the brightness of the fluorescence image (second image) to be even brighter, to a gain adjustment unit 133 (second image gain adjustment unit) that adjusts the imaging signal of the fluorescence image (second image).

[0159] When the brightness of the fluorescent image is to be darkened in a state in which the brightness of the fluorescent image has been adjusted to be bright by the gain adjustment section 133 (second image gain adjustment section), the control section 144 (temporary fluorescent gain control section 142) outputs a control signal to the gain adjustment section 133 (second image gain adjustment section) for adjusting the gain of the imaging signal of the fluorescent image to be smaller. Furthermore, when the brightness of the fluorescent image is to be darker in a state in which the brightness of the fluorescent image has been adjusted to be dark by the gain adjustment section 133, the control section 144 (temporary fluorescent gain control section 142) outputs a control signal to the excitation light source 22 for adjusting the intensity of the second irradiation light to be lowered within the adjustment range of the excitation light source 22.

[0160] If the brightness of the fluorescent image required to maintain a setting of 2 for the background fluorescence ratio is insufficient by simply increasing the intensity of the excitation light source 22 within the adjustment range for the intensity of the excitation light source 22, the temporary fluorescent gain control unit 142 generates a control signal for controlling the gain of the fluorescent imaging signal and outputs the generated control signal to the additional fluorescent gain addition unit 143.

[0161] Furthermore, when darkening the fluorescent image while the gain of the fluorescent imaging signal is increased, priority is given to lowering the gain of the fluorescent imaging signal. In other words, when the fluorescent image is not dark enough to maintain the background fluorescence ratio of 2 by simply lowering the gain of the fluorescent imaging signal within the adjustment range for adjusting the gain of the fluorescent imaging signal, the temporary fluorescent gain control unit 142 generates a control signal for controlling the intensity of the excitation light source 22, and outputs the generated control signal to the excitation light source 22 of the light source 23. This makes it possible to obtain a fluorescent image with reduced effects of noise in controlling the brightness of the fluorescent image.

[0162] Specifically, the control unit 144 (temporary fluorescent light gain control unit 142) controls the brightness of the fluorescent light image as follows.

[0163] The control unit 144 (temporary fluorescence gain control unit 142) adjusts the intensity of the excitation light source 22 within the adjustment range of the excitation light source 22 (second light source) and outputs a control signal to the excitation light source 22 to control the intensity of the excitation light source 22 when the background fluorescence 2 (second ratio) can be adjusted to a constant value while keeping the intensity ratio of the irradiation light set based on the background fluorescence 2 (second ratio) constant.

[0164] The control unit 144 (temporary fluorescence gain control unit 142) adjusts the intensity of the excitation light source 22 to keep the intensity ratio of the irradiation light set based on the background fluorescence 2 (second ratio) constant, and when the brightness of the fluorescence image required to maintain the setting of background fluorescence 2 (second ratio) is insufficient, outputs a control signal for generating an additional gain to increase the gain of the imaging signal of the fluorescence image (second image), and the gain adjustment unit 133 (second image gain adjustment unit) outputs an imaging signal to which the additional gain has been added based on the control signal.

[0165] The control unit 144 (temporary fluorescence gain control unit 142) outputs a control signal for adjusting the additional gain to a smaller value when the brightness of the fluorescent image is to be darkened in order to keep the background fluorescence 2 (second ratio) constant, while the gain of the imaging signal is adjusted by adding the additional gain.

[0166] In the third embodiment, both the background gain control unit 140 and the temporary fluorescence gain control unit 142 prioritize increasing the intensity of the irradiated light over the gain (master gain, gain of the fluorescent imaging signal). Therefore, if the intensities of the white light and the excitation light are controlled to change at the same rate within the adjustment ranges of the white light intensity and the excitation light intensity, the intensity ratio of the irradiated light (white light, excitation light) will be a constant value.

[0167] The range of the intensity ratio of the irradiation light can be adjusted by setting the background fluorescence ratio 2. The control unit 144 adjusts the intensity of at least one of the white light source 20 and the excitation light source 22 within the intensity adjustment range of the white light source 20 (first light source) and the intensity adjustment range of the excitation light source 22 (second light source) based on the background fluorescence 2 (second ratio), thereby adjusting the setting of the intensity ratio of the irradiation light of the white light source 20 and the excitation light source 22 (range of the ratio of the intensity of the irradiation light).

[0168] For example, the adjustment range of the white light intensity and the excitation light intensity are both from 0% to 100%. When the background fluorescence ratio 2 is set to 1.25, if the white light intensity and the excitation light intensity are both within the adjustment range, that is, if the white light intensity is controlled within the range of 0% to 80% and the excitation light intensity is controlled within the range of 0% to 100% (1.25 times the white light intensity), the intensity ratio of the irradiated light will be the maximum of 1.25 times.

[0169] In addition, when the intensity of the white light is controlled within the range of 80% to 100%, the intensity of the excitation light becomes 100% by multiplying the intensity of the white light by 1.25%. Since the upper limit of the intensity of the excitation light is reached and the intensity of the excitation light cannot be adjusted to 100% or more, if the intensity of the white light is increased to 80% or more, the intensity ratio of the irradiation light decreases.

[0170] Furthermore, when the intensity of the white light and the intensity of the excitation light are both controlled at 100%, the intensity ratio of the irradiated light is 1 times the minimum value.

[0171] In this way, when the background fluorescence ratio 2 is set to 1.25 times, the intensity ratio of the irradiated light is controlled to be within the range of 1 to 1.25 times.

[0172] Furthermore, for example, by setting the additional fluorescence gain to G1 (e.g., 1.6), the range of the intensity ratio of the irradiated light can be limited to 1 to 1.25, while the background fluorescence ratio 2 can be set to 1.25 x 1.6 = 2.

[0173] For example, assume that the adjustment range of the white light intensity is from 0% to 40%, the adjustment range of the excitation light intensity is from 0% to 100%, and the background fluorescence ratio 2 is set to 2. In this case, when the intensity of the white light is controlled within the range of 0% to 40% and the intensity of the excitation light is controlled within the range of 0% to 80% (twice the intensity of the white light), the intensity ratio of the irradiated light will be the minimum of 2.

[0174] Furthermore, if the intensity of the excitation light is controlled within a range of 80% or more and 100% or less, and the intensity ratio of the irradiated light is doubled, since the upper limit of the intensity of white light is 40%, as the intensity of the excitation light increases further from 80%, the intensity ratio of the irradiated light increases by more than 2.

[0175] When the intensity of the white light is controlled at 40% and the intensity of the excitation light is controlled at 100%, the intensity ratio of the irradiated light is 2.5 times. In other words, when the background fluorescence ratio 2 is set to 2 times within the adjustment range set for each light source, the intensity ratio of the irradiated light is controlled to be within the range of 2 times to 2.5 times.

[0176] In this example as well, for example, by setting the additional fluorescence gain to G2 (e.g., 2x), the range of the intensity ratio of the irradiated light can be limited to 2x or more and 2.5x or less, while the background fluorescence ratio can be made 2x2=4x.

[0177] Note that the settings of the background fluorescence ratio 2 and the additional fluorescence gain may be default values ​​that have been set in advance. In addition to this example, a storage unit 145 that holds a plurality of settings of the background fluorescence ratio 2 and the additional fluorescence gain may be provided inside the image processing unit 13, and the background fluorescence ratio 2 and the additional fluorescence gain may be acquired from the storage unit 145. For example, the additional fluorescence gain addition unit 143 may acquire a predetermined additional fluorescence gain from the storage unit 145 based on a control signal from the temporary fluorescence gain control unit 142.

[0178] In the above-described first to third embodiments, a three-plate prism 111 is used, and four imaging signals of red fluorescence and reflected R (red) light, G (green) light, and B (blue) light are obtained by controlling the white light source 20 and the excitation light source 22 in a time-division manner. However, if a four-plate prism capable of separating light into four light channels of R (red), G (green), B (blue), and fluorescence is used, each channel can be directly read out, and therefore the light source does not need to be controlled in a time-division manner.

[0179] Moreover, instead of the prism 111, color filters may be used to obtain R (red), G (green), and B (blue) light.

[0180] In the above-described first to third embodiments, the image processing unit 13 includes the gain adjustment units 131 and 132. However, the present invention is not limited to this example, and the image capturing unit 11 may include the gain adjustment units 131 and 132. Furthermore, the background gain adjustment unit 134 may be disposed in both the image capturing unit 11 and the image processing unit 13 so that the master gain can be adjusted in each of them. In this way, the master gain may be adjusted in the image processing unit 13 to compensate for the shortage caused by the adjustment of the master gain in the image capturing unit 11.

[0181] Similarly, the gain adjustment section 133 may be provided in both the imaging section 11 and the image processing section 13 so that the gain of the fluorescent imaging signal can be adjusted in each. The gain of the fluorescent imaging signal may be adjusted in the image processing section 13 to make up for any shortfall caused by the gain adjustment of the fluorescent imaging signal in the imaging section 11.

[0182] In the above-described first to third embodiments, it is assumed that red fluorescence is generated by blue excitation light, but similar processing can be performed even when multiple lights in other wavelength bands are used.

[0183] When processing multiple fluorescent lights (e.g., red and green) in multiple wavelength bands, for red fluorescence, the shutter 112 and image sensor 116 are used in common with R (red) light, and functions for the R (red) light and the red fluorescence are assigned in a time-division manner. For green fluorescence, the shutter 113 and image sensor 117 are used in common with G (green) light, and functions for the G (green) light and the green fluorescence are assigned in a time-division manner.

[0184] The disclosure of this specification includes the following processing device, processing system, processing method, and program. (Item 1) A first acquisition unit that acquires first image information related to brightness adjustment of a first image obtained from light including a light of a first light source and a light of a second light source different from the first light source; a second acquisition unit that acquires second image information related to brightness adjustment of a second image obtained from the light of the second light source; a control unit that controls the intensity of a first illumination light emitted by the first light source and the sensitivity of the first image, and the intensity of a second illumination light emitted by the second light source and the sensitivity of the second image, The control unit is a processing device that controls at least one of the intensity of the first irradiation light irradiated by the first light source, the sensitivity of the first image, the intensity of the second irradiation light irradiated by the second light source, and the sensitivity of the second image based on the first image information and the second image information so that the ratio between the intensity of the first irradiation light and the intensity of the second irradiation light is within a predetermined range. (Item 2) The control unit is When controlling the first light source, an intensity of the first irradiation light is controlled using at least one of an amplitude of the first irradiation light emitted from the first light source, a lighting time of the first irradiation light, a pulse width of the first irradiation light, and a pulse density of the first irradiation light; 2. The processing device according to item 1, wherein when controlling the second light source, an intensity of the second irradiation light is controlled using at least one of an amplitude of the second irradiation light irradiated from the second light source, a lighting time of the second irradiation light, a pulse width of the second irradiation light, and a pulse density of the second irradiation light. (Item 3) The processing device described in Item 1, wherein the control unit adjusts the sensitivity of the first image and the sensitivity of the second image using at least one of an exposure time of an imaging unit that generates an electrical signal by receiving light, a gain applied to the electrical signal generated by the imaging unit, and a gain applied to an electrical signal used by an image generation unit to generate the first image and the second image. (Item 4) The second acquisition unit acquires the second image information as information related to brightness adjustment of the second image by using a second ratio and the first image information; the first image information being a product of the intensity of the first light source and the sensitivity of the first image; the second image information being a product of the intensity of the second light source and the sensitivity of the second image; 2. The processing device according to item 1, wherein the second ratio is a ratio between the first image information and the second image information. (Item 5) The processing device described in Item 1, wherein the control unit is capable of adjusting the intensity of at least one of the first light source and the second light source within an adjustment range of the intensity of the first irradiation light and the intensity of the second irradiation light, thereby adjusting the range of the ratio between the intensity of the first irradiation light and the intensity of the second irradiation light. (Item 6) The second acquisition unit acquires a first setting that adjusts the second ratio, The control unit obtains a second setting that adjusts an additional gain for the sensitivity of the second image; the second acquisition unit acquires the second image information by using the first image information and the second ratio adjusted by the first setting; the control unit controls an intensity of the first irradiation light and a sensitivity of the first image based on the first image information; The control unit controls an intensity of the second irradiation light and a sensitivity of the second image based on the second image information, 5. The processing device according to item 4, wherein the control unit further adds an additional gain adjusted by the second setting to the sensitivity of the second image. (Item 7) The processing device described in Item 1, wherein the second acquisition unit acquires second image information regarding brightness adjustment of the second image based on a comparison between evaluation information of the brightness of the second image acquired using an imaging signal of an imaging unit that captures the second image and target information. (Item 8) The processing device described in Item 1, wherein the control unit controls at least one of the first light source and the second light source so that, within the adjustment range of the first light source and the adjustment range of the second light source, the ratio between the intensity of the first irradiation light irradiated from the first light source and the intensity of the second irradiation light irradiated from the second light source becomes a set intensity ratio of the irradiation lights. (Item 9) The processing device described in Item 8, wherein the control unit controls the intensity of the second irradiation light to be maximum when the intensity of the first irradiation light is maximum when the intensity of the first irradiation light is maximum, so that the ratio between the intensity of the first irradiation light and the intensity of the second irradiation light becomes the intensity ratio within the adjustment range of the first light source and the adjustment range of the second light source. (Item 10) The processing device described in Item 9, wherein the control unit controls the intensity of the first irradiation light so as not to exceed the adjustment range of the intensity of the second irradiation light based on the set intensity ratio of the irradiation light, the adjustment range of the first light source, and the adjustment range of the second light source. (Item 11) The processing device according to item 1, wherein the first light source is a light source for observing reflected light from a subject, and the second light source is a light source for observing fluorescence emitted from a fluorescent substance present in the subject. (Item 12) The present invention includes a first operation mode in which the ratio between the intensity of the first irradiation light and the intensity of the second irradiation light is kept within the predetermined range, and a second operation mode in which the range of the ratio between the intensity of the first irradiation light and the intensity of the second irradiation light is wider than that in the first operation mode, 2. The processing device according to item 1, wherein the control unit is capable of switching between the first operation mode and the second operation mode. (Item 13) The optical fiber further includes a third light source different from the first light source and the second light source, The control unit is At least one of the intensity of the first irradiation light, the sensitivity of the first image, the intensity of the second irradiation light, and the sensitivity of the second image is controlled based on the first image information and the second image information so that the ratio falls within the predetermined range; and The processing device described in item 1 controls at least one of the intensity of the first irradiation light, the sensitivity of the first image, the intensity of the third irradiation light, and the sensitivity of the third image based on the first image information and third image information related to brightness adjustment of a third image obtained from the light of the third light source so that a ratio between the intensity of the first irradiation light and the intensity of the third irradiation light emitted by the third light source is within a predetermined range. (Item 14) When controlling the intensity of the first irradiation light, the control unit controls the intensity of the second irradiation light by giving priority to the sensitivity of the second image, When controlling the sensitivity of the first image, the sensitivity of the second image is controlled with priority over the intensity of the second irradiation light; 2. The processing device according to item 1, wherein, when controlling the intensity of the second irradiation light, the intensity of the first irradiation light is controlled with priority over the sensitivity of the first image, and, when controlling the sensitivity of the second image, the sensitivity of the first image is controlled with priority over the intensity of the first irradiation light. (Item 15) A white balance adjustment unit that adjusts white balance of a plurality of image pickup signals obtained by an image pickup unit that captures the first image; a first image gain adjustment unit that adjusts gains of the plurality of imaging signals in the first image by using the plurality of imaging signals whose white balance has been adjusted and a control signal obtained from the control unit; a first evaluation information acquisition unit that acquires evaluation information of brightness of the first image by using the plurality of imaging signals in which the gain of the first image has been adjusted; a second image gain adjustment unit that adjusts a gain of an image signal obtained by the imaging unit based on the light of the second light source, 8. The processing device according to item 7, wherein the first acquisition unit acquires the first image information as information regarding brightness adjustment of the first image by comparing the evaluation information with preset target information. (Item 16) A first image generating unit that generates the first image based on the plurality of imaging signals whose gains have been adjusted by the first image gain adjusting unit; a second image generating unit that generates the second image based on the imaging signal whose gain has been adjusted by the second image gain adjusting unit; an image synthesis unit that outputs an image signal of a synthetic image obtained by synthesizing the first image and the second image; Item 16. The processing device according to item 15, further comprising: (Item 17) A processing device according to any one of items 1 to 16, a display device having a display control unit that performs display control to display an image in which the second image is superimposed on the first image on a display unit based on an image signal output from the processing device; and a light source having the first light source and the second light source, the light intensity of which can be adjusted based on a control signal output from the processing device; A processing system comprising: (Item 18) A first acquisition step of acquiring first image information related to brightness adjustment of a first image obtained from light including a light of a first light source and a light of a second light source different from the first light source; a second acquisition step of acquiring second image information related to brightness adjustment of a second image obtained from the light of the second light source; a control step of controlling the intensity of a first irradiation light emitted by the first light source and the sensitivity of the first image, and the intensity of a second irradiation light emitted by the second light source and the sensitivity of the second image, In the control process, at least one of the intensity of the first irradiation light irradiated by the first light source, the sensitivity of the first image, the intensity of the second irradiation light irradiated by the second light source, and the sensitivity of the second image is controlled based on the first image information and the second image information so that the ratio between the intensity of the first irradiation light and the intensity of the second irradiation light is within a predetermined range. (Item 19) A program for causing a computer to execute the processing method described in Item 18.

[0185] [Other embodiments] The disclosed technology can also be realized by supplying a program that realizes one or more functions of the above-mentioned embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0186] The disclosed technology is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to disclose the scope of the invention. [Explanation of symbols]

[0187] 1: imaging device, 11: imaging section, 13: image processing section, 15: background image generating section, 16: fluorescent image generating section, 17: image synthesis section, 18: image generating section, 20: white light source, 22: excitation light source, 111: prism 112, 113, 114, 115: Shutter 116, 117, 118, 119: Image sensor 130: white balance adjustment section, 131, 132, 133: gain adjustment section, 134: background gain adjustment section, 135: background evaluation information acquisition section, 136: background total gain setting section, 137: fluorescence evaluation information acquisition section, 138: fluorescence total gain acquisition section, 139: control section, 140: background gain control section, 141: provisional fluorescence total gain acquisition section, 142: provisional fluorescence gain control section, 143: additional fluorescence gain addition section

Claims

1. A first acquisition unit acquires first image information relating to brightness adjustment of a first image obtained from light including light from a first light source and light from a second light source different from the first light source, A second acquisition unit acquires second image information relating to brightness adjustment of the second image obtained from the light of the second light source, The system includes a control unit that controls the intensity of the first illumination light emitted by the first light source and the sensitivity of the first image, and the intensity of the second illumination light emitted by the second light source and the sensitivity of the second image, The control unit is a processing device that controls at least one of the following based on the first image information and the second image information: the intensity of the first illumination light emitted by the first light source, the sensitivity of the first image, the intensity of the second illumination light emitted by the second light source, and the sensitivity of the second image, so that the ratio of the intensity of the first illumination light to the intensity of the second illumination light is within a predetermined range.

2. The control unit, When controlling the first light source, the intensity of the first irradiation light is controlled using at least one of the amplitude of the first irradiation light emitted from the first light source, the illumination time of the first irradiation light, the pulse width of the first irradiation light, and the pulse density of the first irradiation light. The apparatus according to claim 1, wherein, when controlling the second light source, the intensity of the second irradiation light is controlled using at least one of the amplitude of the second irradiation light emitted from the second light source, the illumination time of the second irradiation light, the pulse width of the second irradiation light, and the pulse density of the second irradiation light.

3. The processing apparatus according to claim 1, wherein the control unit adjusts the sensitivity of the first image and the sensitivity of the second image using at least one of the following: the exposure time of the imaging unit that generates an electrical signal by receiving light; the gain applied to the electrical signal generated by the imaging unit; and the gain applied to the electrical signal used by the image generation unit to generate the first image and the second image.

4. The second acquisition unit uses the second ratio and the first image information to acquire the second image information as information related to brightness adjustment of the second image. The first image information is the product of the intensity of the first light source and the sensitivity of the first image. The second image information is the product of the intensity of the second light source and the sensitivity of the second image. The processing apparatus according to claim 1, wherein the second ratio is the ratio of the first image information to the second image information.

5. The apparatus according to claim 1, wherein the control unit is capable of adjusting the range of the ratio between the intensity of the first and second illumination light by adjusting the intensity of at least one of the first and second light sources within the adjustment range of the intensity of the first and second illumination light.

6. The second acquisition unit acquires the first setting for adjusting the second ratio, The control unit acquires a second setting for adjusting the additional gain for the sensitivity of the second image, The second acquisition unit acquires the second image information using the first image information and the second ratio adjusted by the first setting. The control unit controls the intensity of the first irradiation light and the sensitivity of the first image based on the first image information. The control unit controls the intensity of the second irradiation light and the sensitivity of the second image based on the second image information. The processing apparatus according to claim 4, wherein the control unit further adds an additional gain adjusted by the second setting to the sensitivity of the second image.

7. The processing apparatus according to claim 1, wherein the second acquisition unit acquires second image information relating to brightness adjustment of the second image based on a comparison between brightness evaluation information of the second image acquired using the imaging signal of the imaging unit that captures the second image and target information.

8. The apparatus according to claim 1, wherein the control unit controls at least one of the first light source and the second light source within the adjustment range of the first light source and the adjustment range of the second light source, such that the ratio of the intensity of the first irradiation light emitted from the first light source to the intensity of the second irradiation light emitted from the second light source becomes a set irradiation light intensity ratio.

9. The apparatus according to claim 8, wherein the control unit controls the ratio of the intensity of the first irradiated light to the intensity of the second irradiated light to be the intensity ratio within the adjustment range of the first light source and the adjustment range of the second light source, and controls the intensity of the second irradiated light to be the maximum when the intensity of the first irradiated light is at its maximum.

10. The apparatus according to claim 9, wherein the control unit controls the intensity of the first irradiation light so as not to exceed the adjustment range of the intensity of the second irradiation light, based on the set intensity ratio of the irradiation light and the adjustment range of the first light source and the adjustment range of the second light source.

11. The apparatus according to claim 1, wherein the first light source is a light source for observing reflected light from a subject, and the second light source is a light source for observing fluorescence emitted from a fluorescent substance present in the subject.

12. The system includes a first operating mode that maintains the ratio of the intensity of the first irradiation light to the intensity of the second irradiation light within a predetermined range, and a second operating mode in which the range of the ratio of the intensity of the first irradiation light to the intensity of the second irradiation light is wider than that of the first operating mode. The processing apparatus according to claim 1, wherein the control unit is capable of switching between the first operating mode and the second operating mode.

13. The system further comprises a third light source different from the first light source and the second light source, The control unit, Based on the first image information and the second image information, at least one of the following is controlled so that the ratio falls within the predetermined range: the intensity of the first illumination light, the sensitivity of the first image, the intensity of the second illumination light, and the sensitivity of the second image. The processing apparatus according to claim 1, which controls at least one of the intensity of the first illumination light, the sensitivity of the first image, the intensity of the third illumination light, and the sensitivity of the third image, based on the first image information and the third image information relating to brightness adjustment of the third image obtained from the light of the third light source, so that the ratio of the intensity of the first illumination light to the intensity of the third illumination light emitted by the third light source is within a predetermined range.

14. When the control unit controls the intensity of the first irradiation light, it controls the intensity of the second irradiation light with priority over the sensitivity of the second image. When controlling the sensitivity of the first image, the sensitivity of the second image is controlled with priority over the intensity of the second illumination light. When controlling the intensity of the second irradiation light, the intensity of the first irradiation light is controlled with priority over the sensitivity of the first image. The processing apparatus according to claim 1, wherein, when controlling the sensitivity of the second image, the sensitivity of the first image is controlled with priority over the intensity of the first irradiation light.

15. A white balance adjustment unit adjusts the white balance of multiple imaging signals obtained by the imaging unit that captures the first image, A first image gain adjustment unit adjusts the gain of the plurality of imaging signals in the first image using the plurality of imaging signals with adjusted white balance and control signals acquired from the control unit, A first evaluation information acquisition unit acquires brightness evaluation information of the first image using the plurality of imaging signals whose gains have been adjusted in the first image, The system further includes a second image gain adjustment unit that adjusts the gain of the imaging signal obtained by the imaging unit based on the light from the second light source, The processing apparatus according to claim 1, wherein the first acquisition unit acquires the first image information as information relating to brightness adjustment of the first image by comparing the evaluation information with pre-set target information.

16. A first image generation unit generates a first image based on the plurality of imaging signals whose gains have been adjusted by the first image gain adjustment unit, A second image generation unit generates the second image based on the imaging signal whose gain has been adjusted by the second image gain adjustment unit, An image synthesis unit that outputs an image signal of a composite image obtained by combining the first image and the second image, The apparatus according to claim 15, further comprising:

17. The processing apparatus according to any one of claims 1 to 16, A display device having a display control unit that performs display control to display an image on a display unit obtained by superimposing the second image on the first image based on an image signal output from the processing device, A light source having a first light source and a second light source, capable of adjusting the intensity of light based on a control signal output from the processing device, A processing system equipped with the following features.

18. A first acquisition step involves acquiring first image information relating to brightness adjustment of a first image obtained from light including light from a first light source and light from a second light source different from the first light source. A second acquisition step involves acquiring second image information relating to brightness adjustment of the second image obtained from the light of the second light source, The system includes a control step for controlling the intensity of the first illumination light emitted by the first light source and the sensitivity of the first image, and the intensity of the second illumination light emitted by the second light source and the sensitivity of the second image, The control step involves a processing method that controls at least one of the following based on the first image information and the second image information: the intensity of the first illumination light emitted by the first light source, the sensitivity of the first image, the intensity of the second illumination light emitted by the second light source, and the sensitivity of the second image, so that the ratio of the intensity of the first illumination light to the intensity of the second illumination light is within a predetermined range.

19. A program that causes a computer to execute the processing method described in claim 18.

Images