Endoscope system, control device, and control method of control device

Abstract

The endoscope system (10) includes an illumination unit (140), an imaging unit (230), and a control unit (110). The illumination unit (140) irradiates illumination light having a variable spectrum. The imaging unit (230) captures an image of an observation object irradiated with the illumination light. The control unit (110) controls the illumination unit (140) to irradiate illumination light having different spectra depending on whether liquid exists in the observation object or the transmittance of the liquid.

Description

Technical Field

[0001] The present invention relates to an endoscope system, a control device, a control method of the control device, and the like. Background Art

[0002] In diagnosis or surgery using an endoscope, the subject is sometimes observed underwater or through liquid accumulated in the observation area. For example, in a bladder endoscope, the endoscope is inserted into a body cavity filled with liquid to perform underwater observation. Alternatively, even in applications or situations that do not require underwater observation, it is conceivable to observe the subject through water or body fluids transported from the endoscope. Patent document 1 discloses the following endoscope device: an illumination unit includes a plurality of light sources with different emission wavelengths, and a color tone control unit that changes the ratio of the respective emitted light amounts to change the color tone of the illumination light, thereby providing an appropriate color tone to the observed object.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Laid-Open No. 2017-136405 Summary of the Invention

[0006] Problems to be solved by the invention

[0007] In underwater observation, as described above, the transmittance of the liquid may decrease due to the presence of tissue or body fluids, reducing the visibility of the object being observed. In such situations, it is desirable to improve the visibility of the object being observed. However, no method has been disclosed for adjusting the observation image to facilitate viewing by optimizing the illumination used for observation. For example, Patent Document 1 mentioned above does not disclose a mechanism for changing the illumination pattern in response to the turbidity of the water being supplied to the object being observed.

[0008] Means for solving problems

[0009] One embodiment of the present invention relates to an endoscope system, which includes: an illumination unit that irradiates illumination light with a variable spectrum; an imaging unit that captures an observation object irradiated with the illumination light; and a control unit that controls the illumination unit to irradiate the illumination light with different spectra based on whether there is liquid in the observation object or based on the transmittance of the liquid.

[0010] In addition, another embodiment of the present invention relates to a lighting control method, which performs the following control: irradiating illumination light with a variable spectrum, photographing the observation object irradiated with the illumination light, and irradiating the illumination light with different spectra depending on whether there is liquid in the observation object or depending on the transmittance of the liquid. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 This is an example of observation using an endoscope system.

[0012] Figure 2 This is a first structural example of an endoscope system.

[0013] Figure 3 This is an example of switching the observation mode depending on whether liquid is present.

[0014] Figure 4 This is an example of switching the observation mode according to the transmittance of the liquid.

[0015] Figure 5 This is an example of control to increase the long-wavelength component of normal light.

[0016] Figure 6 This is an example of switching the observation mode according to the amount of blood in the fluid.

[0017] Figure 7 It is an example of special light that includes amber light.

[0018] Figure 8 This is an example of the structure of a learning device.

[0019] Figure 9 is the first example of the training data.

[0020] Figure 10 This is a configuration example of an endoscope system in the case of using a learned model learned by the first example of training data.

[0021] Figure 11 is the second example of training data.

[0022] Figure 12 This is a configuration example of an endoscope system in the case of using a learned model learned using the second example of training data.

[0023] Figure 13 is the third example of the training data.

[0024] Figure 14 This is a configuration example of an endoscope system in the case of using a learned model learned using the third example of training data.

[0025] Figure 15 is the fourth example of the training data.

[0026] Figure 16 This is a configuration example of an endoscope system in the case of using a learned model learned using the fourth example of training data.

[0027] Figure 17 This is a second configuration example of the endoscope system.

[0028] Figure 18 This is an example of biological information.

[0029] Figure 19 This is an example of light balance settings.

[0030] Figure 20 This is an example of setting the brightness correction coefficient.

[0031] Figure 21 This is the first example of illumination light using five light sources including an amber light source.

[0032] Figure 22 This is the second example of illumination light when five light sources are used, including an amber light source.

[0033] Figure 23 This is the third example of illumination light when five light sources are used, including an amber light source.

[0034] Figure 24 This is the fourth example of illumination light when five light sources are used, including an amber light source.

[0035] Figure 25 This is the fifth example of illumination light when five light sources are used, including an amber light source.

[0036] Figure 26 This is a detailed structural example of the processing unit and the lighting mode switching control unit in the third structural example.

[0037] Figure 27 This is a processing flow chart in the third structural example. DETAILED DESCRIPTION

[0038] The present embodiment is described below. The present embodiment described below does not unduly limit the contents of the claims. Not all structures described in this embodiment are necessarily essential technical features of the present invention.

[0039] 1. First Configuration Example of Endoscope System

[0040] Figure 1 An example of observation using the endoscope system of this embodiment is shown. While the following description uses a case where a body cavity is filled with liquid as an example, the endoscope system of this embodiment can also be applied to a case where liquid exists between the distal end of the scope and the object being observed, that is, to a case where illumination light is irradiated onto the object through the liquid and the object is observed through the liquid.

[0041] like Figure 1As shown, a body cavity 2 such as a bladder or a joint is filled with a liquid 5. The liquid 5 is, for example, water, body fluid, or a mixture thereof. In addition, tissue pieces peeled from the body cavity 2 may also be contained in the liquid 5. The scope 200 of the endoscope system is inserted into the body cavity 2, and the scope 200 irradiates the illumination light 7 from its front end through the liquid 5 toward the observation object 1. The return light from the observation object 1 reaches the scope 200 through the liquid 5, and the scope 200 takes an image of the observation object 1. The observation object 1 is an object observed by the user in the body cavity 2, such as a lesion. However, the observation object 1 is not limited to a lesion, and the area in the body cavity 2 that appears in the field of view of the scope 200 is set as the observation object 1.

[0042] In such underwater observation, the visibility of the object 1 is reduced due to the influence of the spectral and scattering properties of the liquid 5. Furthermore, the visibility of the object 1 varies depending on the type of liquid 5, the concentration of the mixed body fluid, the type of tissue piece, the tissue content, or a combination thereof. For example, if the liquid 5 is almost pure water, the observed image is affected by the spectral and scattering properties of water. Alternatively, if the liquid 5 contains blood, the light absorption properties of hemoglobin make it difficult for the illumination light to reach the object 1, and the visual field becomes red, thereby reducing the visibility of the object 1. The degree of this reduced visibility varies depending on the concentration of blood mixed in the liquid 5. Alternatively, if exfoliated mucosal cells or bone fragments are floating in the liquid 5, these floating mucosal cells scatter the illumination light, making it difficult for the illumination light to reach the object 1, reducing the contrast of the observed image, and thus reducing the visibility of the object 1. The degree of this reduced visibility varies depending on the size and concentration of the mucosal cells and other substances mixed in the liquid 5. In the present embodiment, the above-described reduction in visibility due to the liquid 5 is improved by devising an improvement in the illumination light 7 for observing the observation object 1 .

[0043] Figure 2 This is a first configuration example of the endoscope system 10 of this embodiment. The endoscope system 10 includes a control device 100, a scope 200, and a display unit 400. Examples of the endoscope system 10 include bladder endoscopes, urinary system endoscopes such as prostatectomies, and arthroscopes used for joint surgery or observation. However, the endoscope system 10 can be either a rigid or flexible endoscope, and can be used for various applications and locations, such as surgery or organ observation.

[0044] The scope 200 is a portion inserted into the body cavity 2 to illuminate and image the interior of the body cavity 2, and includes an emission unit 210, an operation unit 220, and an imaging unit 230. The scope 200 may also include a treatment instrument such as an electric scalpel or forceps.

[0045] The emitting unit 210 emits illumination light from the distal end of the scope 200 toward the observation object 1. The emitting unit 210 includes, for example, a light guide that guides light from the illumination unit 140 provided in the control device 100, and a lens that directs the light guided by the light guide toward the observation object 1. Alternatively, the illumination unit 140 may be provided separately from the control device 100.

[0046] The operation unit 220 is a device for the user to operate the endoscope system 10. The operation unit 220 includes, for example, buttons, switches, or dials. In this embodiment, the operation unit 220 includes a water supply button for controlling water supply from the water supply port of the scope 200.

[0047] The imaging unit 230 is a device that captures images of the inside of the body cavity 2 and outputs imaging signals obtained by the imaging to the control device 100. The imaging unit 230 includes, for example, an objective lens and an image sensor that captures an image formed by the objective lens.

[0048] The control device 100 performs image processing based on the camera signal and controls the operation of the endoscope system 10. The control device 100 includes a control unit 110, a processing unit 120, a storage unit 130, and a lighting unit 140. The control device 100 is a device in which a circuit substrate provided with the control unit 110, the processing unit 120, and the storage unit 130, and the lighting unit 140 are housed in a housing. Alternatively, the control device 100 may be implemented by executing software that describes the operation of the control device 100 on a general-purpose information processing device such as a PC. Alternatively, a part of the control device 100 may be implemented by an information processing device or a cloud system that is separate from the control device 100. For example, the AI ​​processing described later may also be performed by an information processing device that is separate from the control device 100, or a cloud system that is connected to the control device 100 via a network.

[0049] The lighting unit 140 is a light source device that generates illumination light and causes the illumination light to enter the light guide of the emission unit 210. The lighting unit 140 includes a plurality of light sources that can independently control the amount of light and emit light in different wavelength bands. Each light source is, for example, an LED, a laser diode, a light source that emits light from a laser diode to illuminate a fluorescent body and generate fluorescence, or a combination thereof. Alternatively, the lighting unit 140 can also be implemented by a white light source such as a xenon lamp and a filter device that can switch a plurality of optical filters that pass the desired wavelength band. The transmittance spectral characteristics of the plurality of optical filters are different from each other. In addition, in Figure 2 In FIG. 1 , the emission unit 210 as a lens and a light guide and the illumination unit 140 as a light source device are shown separately. However, the lens, light guide, and light source device may be included and referred to as an illumination unit.

[0050] The processing unit 120 is a circuit or device that controls various components of the endoscope system 10 and processes the image signals from the imaging unit 230. The processing unit 120 outputs the processed image to the display unit 400 for display. The display unit 400 is, for example, a display device such as an LCD monitor. Furthermore, the processing unit 120 outputs image information necessary for controlling the spectrum of the illumination light to the control unit 110. This image information may be the captured image itself or image parameters extracted from the captured image. Examples of image parameters include contrast, color, and brightness.

[0051] The control unit 110 is a circuit or device that controls the spectrum of the illumination light by controlling the illumination unit 140 based on image information from the processing unit 120. If the illumination unit 140 includes multiple light sources, the control unit 110 controls the spectrum of the illumination light by generating control parameters for controlling the on / off switching and light intensity of each light source based on the image information. Alternatively, if the illumination unit 140 includes a white light source and a filter device, the control unit 110 controls the spectrum of the illumination light by generating control parameters for switching the optical filters of the filter device.

[0052] The control unit 110 may also generate intermediate information representing the state of the liquid 5 based on the image information, and generate control parameters based on this intermediate information. Examples of the intermediate information include the transmittance of the liquid 5, the type of body fluid contained in the liquid 5, the concentration of the body fluid contained in the liquid 5, the type of tissue fragment contained in the liquid 5, the concentration of the tissue fragment contained in the liquid 5, or a combination thereof. Alternatively, the processing unit 120 may generate the intermediate information based on the captured image, and the control unit 110 may generate the control parameters based on the intermediate information from the processing unit 120.

[0053] The processing unit 120 and the control unit 110 are each implemented by one or more circuits or devices. Alternatively, the processing unit 120 and the control unit 110 may be formed by an integrated circuit or device. The hardware constituting the processing unit 120 and the control unit 110 may be, for example, a processor, an FPGA, an ASIC, or a circuit substrate on which multiple circuit elements are mounted.

[0054] The storage unit 130 stores various data or programs used in the processing performed by the control unit 110 and the processing unit 120. For example, the storage unit 130 is a semiconductor memory such as RAM or ROM, a magnetic storage device such as a hard disk drive, or an optical storage device such as an optical disk drive. Specifically, the storage unit 130 stores data or programs for determining an appropriate spectrum of illumination light corresponding to the state of the liquid 5.

[0055] For example, the storage unit 130 may store a learned model obtained through machine learning. In this case, the control unit 110 uses the learned model to infer appropriate illumination light control parameters for the liquid 5 based on image information or intermediate information. Furthermore, when using intermediate information, the control unit 110 may also use the learned model to infer control parameters based on image information.

[0056] Alternatively, the storage unit 130 may store a database for generating the aforementioned control parameters. The database contains information such as the spectral characteristics and scattering characteristics of the liquid 5. The control unit 110 uses, for example, the contrast, brightness, and color information of the captured image and the database to determine the control parameters or intermediate information. Alternatively, the database may be a lookup table that associates the contrast, brightness, and color information of the captured image with the control parameters or intermediate information. By referring to the lookup table, the control unit 110 obtains the control parameters or intermediate information corresponding to the contrast, brightness, and color information of the captured image.

[0057] Alternatively, the storage unit 130 may store a database for determining illumination light control parameters based on intermediate information. For example, the database may be a lookup table that associates intermediate information, such as the transmittance of the liquid 5, with appropriate illumination light control parameters. The control unit 110 refers to the lookup table to obtain the light intensity of each light source or the control parameters of the optical filter to be selected, corresponding to the intermediate information.

[0058] According to the above embodiment, the endoscope system 10 includes: an illumination unit 140 for emitting illumination light 7 having a variable spectrum; an imaging unit 230 for capturing an image of the observation object 1 illuminated by the illumination light 7; and a control unit 110. The control unit 110 controls the illumination unit 140 to emit illumination light 7 having different spectra, depending on whether or not liquid 5 is present in the observation object 1 or on the transmittance of the liquid 5.

[0059] In this way, even when the visibility of the observation object 1 is reduced due to the presence of tissue or body fluid in the liquid, the observation object 1 can be observed with the illumination light 7 of an appropriate spectrum, depending on whether the liquid 5 is present in the observation object 1 or the transmittance of the liquid 5. The appropriate spectrum refers to a spectrum that improves the visibility of the observation object 1 through the liquid 5 compared to the case of observation without changing the spectrum of the illumination light 7.

[0060] Specifically, the captured image is affected by the transmittance of the liquid 5, resulting in an image with, for example, a change in contrast. Therefore, when a captured image is input to the control unit 110, control parameters are determined based on the captured image, thereby achieving spectral control corresponding to the transmittance. Alternatively, when the transmittance of the liquid 5 is input to the control unit 110 as control information, the control unit 110 determines control parameters based on the transmittance of the liquid 5, thereby achieving spectral control corresponding to the transmittance. Alternatively, the transmittance of the liquid 5 is determined by the type or concentration of the body fluid or tissue piece contained in the liquid 5. Therefore, when the type or concentration of the body fluid or tissue piece contained in the liquid 5 is input to the control unit 110 as intermediate information, control parameters are determined based on this intermediate information, thereby achieving spectral control corresponding to the transmittance.

[0061] Here, “whether the liquid 5 exists in the observation object 1 ” specifically refers to whether the liquid 5 exists between the distal end of the scope 200 and the observation object 1 .

[0062] "Transmittance of liquid 5" refers to the transmittance of illuminating light through liquid 5 between the tip of scope 200 and the object 1 being observed. Transmittance refers to the degree to which light passes through liquid 5. Transmittance is, for example, the ratio of the amount of light reflected from the object 1 to the amount of light emitted from the tip of scope 200. Alternatively, if image parameters such as contrast change due to reduced transmittance, image parameters such as the contrast value of the captured image can also be used as transmittance. Furthermore, if the attenuation of light in liquid 5 is considered to determine the amount of transmitted light, the attenuation rate can be used as the inverse of transmittance. This scenario is also included in the case of "based on the transmittance of liquid 5."

[0063] Furthermore, in this embodiment, the control unit 110 controls switching between multiple observation modes, including two or more of a first observation mode, a second observation mode, and a third observation mode. The first observation mode is selected when the liquid 5 is not present, the second observation mode is selected when the transparency of the liquid 5 is relatively high, and the third observation mode is selected when the transparency of the liquid 5 is relatively low. The control unit 110 performs at least one of the following controls: control of switching the illumination light 7 between a first illumination light associated with the first observation mode and a second illumination light associated with the second observation mode, in response to switching between the first and second observation modes; control of switching the illumination light 7 between the second illumination light associated with the second observation mode and the third illumination light associated with the third observation mode, in response to switching between the second and third observation modes; and control of switching the illumination light between the first illumination light associated with the first observation mode and the third illumination light associated with the third observation mode, in response to switching between the first and third observation modes.

[0064] In this way, by switching the observation mode according to the state of the liquid 5 , the illumination light is switched to an appropriate spectrum according to the state of the liquid 5 .

[0065] Here, the first through third observation modes are modes in which the spectra of the illumination light differ from one another. Specifically, the spectra of the first through third illumination light differ from one another. Specifically, the second illumination light is light of a spectrum that has a higher transmittance for the liquid 5 than the first illumination light. The third illumination light is light of a spectrum that has a higher transmittance for the liquid 5 than the second illumination light. The spectrum of light with the highest transmittance varies depending on the type of body fluid or tissue piece contained in the liquid 5. An example will be described below.

[0066] Furthermore, illumination light control is not limited to the above. The number of observation modes is not limited to three; a larger number of observation modes and corresponding illumination light levels can be set. For example, by switching observation modes in multiple stages based on transmittance, the spectrum of the illumination light can be controlled to gradually change according to transmittance.

[0067] use Figures 3 to 7 , an example of switching between the first to third observation modes will be described.

[0068] Figure 3 This is an example of switching the observation mode based on the presence or absence of liquid 5. When liquid 5 is absent, the control unit 110 controls the illumination unit 140 to illuminate with normal light. When the control unit 110 determines that liquid 5 is present, it controls the illumination unit 140 to switch the observation mode to illuminate with turbidity-reducing light. In this example, normal light corresponds to the first illumination light in the first observation mode, and turbidity-reducing light corresponds to the second or third illumination light in the second observation mode.

[0069] The second illumination light is illumination light having a relatively higher ratio of long-wavelength components in its spectrum than the ratio of long-wavelength components in the spectrum of the first illumination light. The third illumination light is illumination light having a relatively higher ratio of long-wavelength components in its spectrum than the ratio of long-wavelength components in the spectrum of the second illumination light. When transitioning from a state in which liquid 5 is absent to a state in which liquid 5 is present, the first illumination light is switched to the second illumination light when the turbidity of liquid 5 is relatively low, and the first illumination light is switched to the third illumination light when the turbidity of liquid 5 is relatively high.

[0070] In this way, when illuminating and imaging the observation object 1 through the liquid 5, illumination light with increased long-wavelength components can be irradiated. In a state where light scattering is significant, such as when the liquid 5 is turbid due to tissue fragments, the long-wavelength component is less scattered in the liquid 5 than the short-wavelength component. Therefore, by using illumination light with increased long-wavelength components, the visibility of the observation object 1 in the presence of the liquid 5 can be improved.

[0071] Here, the long-wavelength component refers to a component in a wavelength band on the long-wavelength side of the wavelength band that constitutes the illumination light. Specifically, the long-wavelength component refers to a component in a wavelength band on the long-wavelength side of the wavelength band that constitutes the illumination light, or, if the wavelength band that constitutes the illumination light is divided into multiple color regions, a component in a color region on the long-wavelength side of the wavelength band that constitutes the illumination light. For example, when the illumination light is visible light and the wavelength band of visible light is divided into RGB color regions, the long-wavelength component refers to a component in the R color region or a component in a wavelength band that belongs to the R color region.

[0072] Figure 4 This is an example of switching the observation mode based on the transmittance of the liquid 5. When the turbidity of the liquid 5 is low, that is, when the transmittance of the liquid 5 is relatively high, the control unit 110 controls the illumination unit 140 to emit turbidity-reducing light A. When the control unit 110 determines that the turbidity of the liquid 5 is increasing and the transmittance is decreasing, it controls the illumination unit 140 to switch the observation mode and emit turbidity-reducing light B. In this example, turbidity-reducing light A corresponds to the second illumination light in the second observation mode, and turbidity-reducing light B corresponds to the third illumination light in the third observation mode.

[0073] In a state where light scattering is high, such as when the liquid 5 is turbid due to tissue fragments, etc., the greater the turbidity, that is, the lower the transmittance, the greater the scattering. According to this embodiment, when the liquid 5 is highly turbid and the transmittance is low, the visibility of the observation object 1 can be improved by using illumination light with increased long-wavelength components.

[0074] In this embodiment, the second and third illumination lights are illumination lights obtained by increasing the long wavelength component of normal light. The increase in the long wavelength component in the third illumination light is greater than the increase in the long wavelength component in the second illumination light.

[0075] Figure 5 This example shows a control scheme that increases the long-wavelength component of normal light. Here, the illumination unit 140 includes a light source LDV that emits violet light, a light source LDB that emits blue light, a light source LDG that emits green light, and a light source LDR that emits red light. The light intensity ratio of the light sources LDV, LDB, LDG, and LDR is set to V:B:G:R, and this light intensity ratio is standardized. The control unit 110 switches the light intensity ratio V:B:G:R based on a control parameter indicating the light intensity ratio V:B:G:R, thereby switching the spectrum of the illumination light. When the light intensity ratio of the first illumination light is set to V:B:G:R = v1:b1:g1:r1, the light intensity ratio of the second illumination light is set to V:B:G:R = v2:b2:g2:r2, and the light intensity ratio of the third illumination light is set to V:B:G:R = v3:b3:g3:r3, r1 < r2 < r3.

[0076] In this way, in underwater observation, illumination light having a long wavelength component adjusted based on the spectrum of ordinary light is used. Therefore, in underwater observation, observation can be performed with substantially the same hue as in normal observation using ordinary light.

[0077] Here, normal observation refers to an observation mode for observing an assumed object in an endoscope system, for example, corresponding to an observation mode for observing using white illumination light when no liquid is present in the object. In addition, normal light refers to illumination light used in normal observation mode, for example, corresponding to white illumination light.

[0078] In addition, in this embodiment, when the control unit 110 controls the illumination unit 140 so that the ratio of the relative long wavelength component in the spectrum of the illumination light increases, the processing unit 120 performs image processing on the captured image to reduce the contribution of the long wavelength component. Figure 3 and Figure 4 In the example of , the processing unit 120 performs this image processing in the second observation mode and the third observation mode.

[0079] Increasing the long-wavelength component of illumination light increases the red tint of the image or decreases the image's color temperature. The effect of the long-wavelength component of illumination light on the image is referred to as the "long-wavelength contribution." According to this embodiment, by performing image processing to reduce the contribution of the long-wavelength component, the red tint of the image can be reduced or the decrease in the image's color temperature can be suppressed. This allows underwater observation to be performed with a color tone closer to that observed with ordinary light.

[0080] Here, image processing that reduces the contribution of long-wavelength components refers to color conversion processing of captured images. Examples of color conversion processing include white balance processing that raises the color temperature of an image, or processing that reduces the components of color regions corresponding to the long-wavelength components of illumination light. For example, the latter process involves multiplying the R component of an image by 1 / x when the light intensity of the light source LDR is multiplied by x.

[0081] Furthermore, in the present embodiment, the processing unit 120 performs structure enhancement processing on the captured image so that the contrast of the target region becomes equal to or greater than a predetermined threshold value.

[0082] Thus, even when the contrast of the interest region is reduced by body fluid or tissue fragments contained in the liquid 5, the contrast of the interest region can be enhanced to a predetermined threshold or higher by the structure enhancement process. This improves the visibility of the interest region during underwater observation.

[0083] Here, the region of interest can be the entire field of view of the imaging unit 230 or a portion of that field of view. For example, the processing unit 120 can detect lesions through image recognition using machine learning or other methods, and use the detected lesion area as the region of interest. Structural emphasis processing includes, for example, processing that increases high-frequency components in an image or contrast enhancement processing using grayscale conversion.

[0084] Figure 6 This is an example of switching the observation mode based on the amount of blood in the liquid 5. When the amount of blood in the liquid 5 is small, that is, when the blood concentration is relatively low, the control unit 110 controls the illumination unit 140 to illuminate with turbidity-reducing light. When the control unit 110 determines that the amount of blood in the liquid 5 has increased and the blood concentration has risen, it controls the illumination unit 140 to switch the observation mode and illuminate with bleeding point observation light. In this example, the turbidity-reducing light corresponds to the second illumination light in the second observation mode, and the bleeding point observation light corresponds to the third illumination light in the third observation mode.

[0085] During surgical procedures using an endoscope, bleeding may sometimes occur due to incision or excision of the affected area. During underwater observation, the bleeding blood mixes with the liquid 5, and is illuminated and observed through this liquid 5. Consequently, the transmittance of the illumination light decreases due to the light absorption of hemoglobin, or the image becomes redder due to the light absorption characteristics of hemoglobin, thereby reducing the visibility of the observed object 1. Furthermore, in cases where the amount of bleeding in the observed object 1 is barely visible, it is preferable to ensure the field of view by stopping the bleeding. According to this embodiment, when the observed object 1 is visually visible to a certain extent with a small amount of blood, the second illumination light, which serves as turbidity-reducing light, can be irradiated to improve visibility. Furthermore, when the observed object 1 is visually visible with a large amount of blood, the third illumination light, which serves as a bleeding point observation light, can be irradiated to improve the visibility of the bleeding point, thereby assisting the user in the hemostasis operation.

[0086] The second illumination light is illumination light with an increased long-wavelength component of ordinary light, and the third illumination light is special light containing narrow-band light corresponding to the long-wavelength component. This also includes the case where the relative ratio of the long-wavelength component in the spectrum of the third illumination light is greater than the relative ratio of the long-wavelength component in the spectrum of the second illumination light.

[0087] In this way, when the transmittance of the liquid 5 is low and the visibility of the observation object 1 is very low, special light can be irradiated instead of light obtained by adjusting the normal light. This allows the irradiation of special light that has high transmittance for body fluids or tissue pieces mixed in the liquid 5, or special light that facilitates observation of a specific observation object 1, thereby improving the visibility of the observation object 1.

[0088] Here, narrowband light refers to light emitted by light sources such as fluorescent lamps, LED (Light Emitting Diode) lamps, laser light sources, or light sources with filters that pass a portion of the wavelength band. This refers to light with discrete spectral characteristics. This term also includes light with high intensity in some wavelength bands and significantly weaker intensity in other wavelength bands. Furthermore, narrowband light can also refer to light with a wavelength band narrower than that of white illumination light. Alternatively, when visible light is divided into RGB wavelength bands, narrowband light can also refer to light with a wavelength band narrower than each of the RGB wavelength bands.

[0089] Specifically, the observation object 1 is a living body, and the liquid 5 contains blood. The third illumination light contains light in an amber wavelength band. Figure 6 In the embodiment of the present invention, the bleeding spot observation light includes light in the amber wavelength range. Furthermore, depending on the application, the second illumination light may also include light in the amber wavelength range, or both the second illumination light and the third illumination light may include light in the amber wavelength range. Hereinafter, light in the amber wavelength range will also be referred to as amber light.

[0090] exist Figure 7 , an example of special light including amber light is shown. Figure 7 Observation using special light is also called RDI (Red Dichromatic Imaging). The illumination unit 140 includes, in addition to the aforementioned light sources LDV, LDB, LDG, and LDR, a light source LDA that emits amber light. The control unit 110 controls the illumination unit 140 to cause the light sources LDG, LDA, and LDR to emit light.

[0091] The wavelength band of the amber light emitted by the light source LDA exists between the wavelength band of the green light emitted by the light source LDG and the wavelength band of the red light emitted by the light source LDR. Specifically, the wavelength band of the amber light is a wavelength band with a peak wavelength of 600nm. It should be noted that the peak wavelength of the amber light is not limited to 600nm. Specifically, Figure 7 The light absorption characteristics of venous blood are shown as an example of the light absorption characteristics of hemoglobin. Within the range of 550 nm to 750 nm, the absorption coefficient reaches a maximum at approximately 576 nm and a minimum at approximately 730 nm. Amber light only needs to exist in the wavelength range from approximately 576 nm, where the hemoglobin absorption coefficient reaches its maximum, to approximately 730 nm, where it reaches its minimum. Within the wavelength range of approximately 576 nm to approximately 730 nm, the shorter the wavelength of light, the greater the hemoglobin absorption coefficient. In other words, amber light is more strongly absorbed by hemoglobin than red light.

[0092] By using special light including amber light, such as the RDI described above, the concentration of blood in the liquid 5 can be easily visually identified, thereby making it easier to visually identify bleeding spots with high blood concentrations. Furthermore, in the above description, the third illumination light is special light, but this is not limiting. The third illumination light may also be light with amber light added to normal light.

[0093] In addition, in this embodiment, the processing unit 120 may also perform the following processing: based on the captured image captured when the observation object 1 is irradiated with the third illumination light including light in the amber wavelength band, detect bleeding spots in the observation object 1 and display the detected bleeding spots.

[0094] In this way, when the amount of bleeding is large and the visibility of the observation object 1 is reduced, the visibility of the bleeding point is improved by irradiating the third illumination light including light in the amber wavelength band, and the hemostasis operation can be assisted by detecting and displaying the bleeding point.

[0095] As described later, the processing unit 120 may detect bleeding spots using a detection process using machine learning. Alternatively, the processing unit 120 may detect bleeding spots using image color information based on, for example, the fact that the saturation of red near a bleeding spot is higher than that of surrounding red.

[0096] The above description uses three levels of illumination light switching as an example, but the illumination light can also be switched in more levels. Specifically, the control unit 110 can also control the illumination unit 140 so that the more body fluid or tissue pieces contained in the liquid 5, the greater the ratio of the relatively long wavelength component in the spectrum of the illumination light.

[0097] In this way, it is possible to irradiate the liquid 5 with illumination light of an appropriate spectrum in a larger number of steps according to the concentration of the body fluid or tissue piece contained in the liquid 5 , thereby improving the visibility of the observation object 1 .

[0098] In addition, the endoscope system 10 includes a supply unit that supplies the liquid 5 to the observation object 1. The supply unit is, for example, Figure 17 The control unit 110 may also control the lighting unit 140 so that the ratio of the relatively long wavelength component in the spectrum of the illumination light increases as the amount of the liquid 5 supplied by the supply unit increases.

[0099] As the amount of liquid 5 supplied from the supply unit increases, the amount of liquid 5 present in the body cavity increases. Consequently, the amount of liquid 5 between the distal end of the scope 200 and the object 1 increases, that is, the distance between the distal end of the scope 200 and the object 1 increases. Therefore, as the amount of liquid 5 supplied from the supply unit increases, the transmittance decreases, making the object 1 more difficult to visually identify. However, according to this embodiment, by increasing the ratio of the long-wavelength component of the illumination light, the visibility of the object 1 can be improved.

[0100] Furthermore, the control unit 110 determines the supply amount of the liquid 5 based on a water supply control signal from the operating unit 220 or water supply amount information from the supply unit. For example, the water supply control signal becomes active when water supply is turned on, and the control unit 110 determines the supply amount based on the validity period of the water supply control signal. Alternatively, the water supply amount information indicates the total amount of water supplied or the amount of water supplied per unit time, and the control unit 110 determines the supply amount based on this water supply amount information.

[0101] While the above description describes control of illumination light according to transmittance, this control may also be performed as follows. Specifically, when the transmittance of the liquid 5 is equal to or greater than a predetermined threshold, the control unit 110 may control the illumination unit 140 so that the difference between the image captured by the imaging unit 230 and a reference image captured in the absence of the liquid 5 is less than a predetermined threshold. Furthermore, when the transmittance of the liquid 5 is less than a predetermined threshold, the control unit 110 may control the illumination unit 140 so that the difference between the image captured by the imaging unit 230 and a second reference image different from the reference image is less than a predetermined threshold.

[0102] In this way, when the liquid 5 contains a small amount of body fluid or tissue fragments and the transparency of the liquid 5 is relatively high, a captured image close to the baseline image captured in the absence of the liquid 5 can be obtained. In other words, up to a certain level of transparency, underwater observation can be performed with a hue roughly the same as that of ordinary observation. Furthermore, when the liquid 5 contains a large amount of body fluid or tissue fragments and the transparency of the liquid 5 is relatively low, a captured image close to a second baseline image different from the baseline image can be obtained. In other words, by allowing a hue different from that of ordinary observation, the visual recognition of the observed object 1 can be prioritized.

[0103] Here, an example of the difference between the captured image and the reference image is the difference between the image parameters of the captured image and the reference image. The image parameters include hue, color balance, brightness, a combination of these, or statistical values ​​thereof.

[0104] exist Figures 3 to 7 , an example of improving visibility by increasing the long-wavelength component of the illumination light is described, but illumination light control for improving visibility is not limited to this. For example, visibility can also be improved by increasing light in a wavelength band with low absorption in the spectral characteristics of the body fluid or tissue piece contained in the liquid 5.

[0105] For example, if the liquid 5 contains blood, light in a wavelength band with a low absorption coefficient for hemoglobin can be used to increase transmittance. Specifically, the ratio of the blue component to the violet component in the spectrum of the second illumination light can be greater than the ratio of the blue component to the violet component in the spectrum of the normal light. Specifically, if the light intensity ratio of the first illumination light (normal light) is set to V:B:G:R = v1:b1:g1:r1, and the light intensity ratio of the second illumination light is set to V:B:G:R = v2:b2:g2:r2, then b2 / v2 > b1 / v1. It should be noted that, depending on the intended use, the ratio of the blue component to the violet component in the spectrum of the third illumination light can be greater than the ratio of the blue component to the violet component in the spectrum of the normal light, and the ratio of the blue component to the violet component in the spectra of the second and third illumination lights can be greater than the ratio of the blue component to the violet component in the spectrum of the normal light.

[0106] like Figure 7 As shown, hemoglobin exhibits absorption peaks near 400 nm and 520 nm, with a relatively low absorption coefficient between these peaks. The wavelength of violet light is around 400 nm, while the wavelength of blue light lies between these two absorption peaks. Therefore, blue light is less likely to be absorbed by hemoglobin than violet light. Therefore, by irradiating the liquid 5 containing blood with a higher proportion of blue light compared to normal light, the transmittance can be increased, thereby enhancing the visibility of the object 1.

[0107] 2. Processing Examples Using Machine Learning

[0108] Next, a method for determining a control parameter for controlling the spectrum of illumination light or detecting bleeding spots by processing using machine learning will be described. First, a learning device 500 that generates a learned model will be described.

[0109] Figure 8 This is a configuration example of the learning device 500 . The learning device 500 includes a storage unit 520 and a processing unit 510 .

[0110] The storage unit 520 is a semiconductor memory, a magnetic storage device, an optical storage device, etc. The storage unit 520 stores training data 522 in which input data 523 and correct answer labels 524 are associated with each other.

[0111] Processing unit 510 is a processor, FPGA, or ASIC. It inputs input data 523 to learning model 511 and obtains the derivation results of learning model 511. Processing unit 510 includes an updating unit 512, which updates learning model 511 based on the error between the derivation results and the correct answer label 524. Learning is performed by repeating this process. Processing unit 510 causes storage unit 520 to store the learned model 22, which is the learned model 511.

[0112] The learned model 22 is transmitted to the endoscope system 10 and stored in the storage unit 130 of the endoscope system 10. The endoscope system 10 uses the learned model 22 stored in the storage unit 130 to perform determination of control parameters and the like.

[0113] When derivation in the endoscope system 10 is performed using general-purpose hardware such as a processor, the learned model 22 includes a program describing the derivation algorithm and parameters used by the derivation algorithm. The parameters are acquired through learning. For example, the derivation algorithm is a neural network. In this case, the parameters are weight coefficients between nodes in the neural network. The processing unit 120, which is general-purpose hardware, executes the program using the parameters, thereby implementing derivation processing using the learned model 22.

[0114] Alternatively, when the derivation in the endoscope system 10 is performed using dedicated hardware that implements the derivation algorithm, the learned model 22 includes parameters used by the derivation algorithm. The processing unit 120, which is dedicated hardware, executes the derivation algorithm using the parameters, thereby implementing the derivation process using the learned model 22.

[0115] Next, an example of training data and an example of derivation processing based on a learned model obtained by machine learning using the training data will be described.

[0116] Figure 9 This is the first example of training data. For each learning image TIMi, the light intensity ratio at the time of capture, R:A:G:B:V, is represented by ai:bi:ci:di:ei, and the recommended light intensity ratio, R:A:G:B:V, is represented by αi:βi:γi:δi:εi. i is an integer from 1 to n, where n is an integer from 2 to 4. The learning image and the light intensity ratio at the time of capture are input data 523, and the recommended light intensity ratio is the correct answer label 524.

[0117] Figure 10This is an example of the configuration of an endoscope system using a learned model 115 that has been learned using the first example of training data. Only relevant components are illustrated here. The control unit 110 inputs the captured image IMG from the processing unit 120 and the control parameter LINF from the light source control unit 111 into the learned model 115. The captured image IMG input to the learned model 115 can be a color image after development processing or an original image such as a RAW image before development processing. Furthermore, the captured image IMG is not limited to data constructed as an image. For example, signal information for each pixel can also be input to the learned model 115. The control parameter LINF represents the light intensity ratio R:A:G:B:V when the captured image IMG is captured. The control unit 110 obtains the control parameter CPAR through derivation processing using the learned model 115. The control unit 110 includes a light source control unit 111, which uses the control parameter CPAR to control the light intensity ratio of each light source of the illumination unit 140.

[0118] Figure 11 This is the second example of training data. For the learning image TIMi, the light intensity ratio R:A:G:B:V = ai:bi:ci:di:ei at the time of capture, the liquid transmittance xi, and the amount of blood contained in the liquid yi are associated. The learning image and the light intensity ratio at the time of capture are input data 523, and the liquid transmittance and the amount of blood contained in the liquid are correct labels 524.

[0119] Figure 12 This is an example of the configuration of an endoscope system using a learned model 115 learned using the second example of training data. Only relevant components are illustrated here. The control unit 110 inputs the captured image IMG from the processing unit 120 and the control parameter LINF from the light source control unit 111 into the learned model 115. The control unit 110 obtains information EKINF related to the liquid through derivation processing using the learned model 115. This information EKINF includes the transmittance of the liquid and the amount of blood contained in the liquid. Alternatively, other information related to the liquid may be used as training data, and this information may be obtained through derivation processing. The control unit 110 includes a parameter determination unit 116 and a light source control unit 111. The parameter determination unit 116 determines the control parameter CPAR based on the information EKINF. For example, the parameter determination unit 116 derives the control parameter based on the learned model obtained by machine learning the relationship between the information related to the liquid and the control parameter. Alternatively, the parameter determination unit 116 may determine the control parameter by referring to a lookup table that associates the information related to the liquid with the control parameter. The light source control unit 111 controls the light intensity ratio of each light source of the illumination unit 140 using the control parameter CPAR from the parameter determination unit 116 .

[0120] In the above Figures 9 to 12In the example, control unit 110 performs processing based on learned model 115, which is obtained by machine learning the relationship between the learning image and information related to the liquid, or the relationship between the learning image and information related to the recommended illumination light. Based on the captured image IMG captured by the imaging unit 230 and the learned model 115, control unit 110 determines control parameters CPAR for controlling the spectrum of the illumination light, and controls illumination unit 140 based on the determined control parameters CPAR. Specifically, learned model 115 is obtained by machine learning the relationship between the learning image, information about the illumination light used in capturing the learning image, and information related to the liquid. Alternatively, learned model 115 is obtained by machine learning the relationship between the learning image, information about the illumination light used in capturing the learning image, and information related to the recommended illumination light.

[0121] In this way, by using the derivation process of machine learning, it is possible to determine the spectrum of illumination light appropriate for the state of the liquid from the captured image. By using machine learning, it is expected that illumination light more optimized for the state of various liquids can be irradiated.

[0122] Figure 13 This is a third example of training data. The learning image TIMi is associated with the light intensity ratio R:A:G:B:V = ai:bi:ci:di:ei at the time of capture and bleeding point information SKi. The learning image and the light intensity ratio at the time of capture are input data 523, and the bleeding point information is the correct answer label 524. The bleeding point information SKi is information indicating the location or area of ​​the bleeding point in the image, such as a rectangle containing the bleeding point or a line along the boundary of the bleeding point.

[0123] Figure 14 This is a structural example of an endoscope system in which a learned model 121 learned by the third example of training data is used. Only related components are illustrated here. The processing unit 120 inputs the camera image IMG and the control parameter LINF from the control unit 110 to the learned model 121. The processing unit 120 detects the position or area of ​​the bleeding point by using the derivation process of the learned model 121. The processing unit 120 includes a display processing unit 123, and the display processing unit 123 displays the detected position or area of ​​the bleeding point together with the camera image IMG on the display unit 400. In addition, the control unit 110 displays the position or area of ​​the bleeding point together with the camera image IMG on the display unit 400. Figures 9 to 12 The control parameter CPAR is determined by the method described in .

[0124] Figure 15This is the fourth example of training data. The learning image TIMi is associated with the liquid transparency xi, the amount of blood contained in the liquid yi, and bleeding point information SKi. The learning image is input data 523, and the transparency, blood volume, and bleeding point information are correct labels 524.

[0125] Figure 16 This is an example of the configuration of an endoscope system using a learned model 121 learned using the fourth example of training data. Only relevant components are illustrated here. The processing unit 120 inputs the camera image IMG into the learned model 121. Using the derivation process of the learned model 121, the processing unit 120 detects information EKINF related to the liquid and the location or area of ​​the bleeding point. Information EKINF represents the transparency of the liquid and the amount of blood contained in the liquid. The processing unit 120 includes a display processing unit 123, which displays the detected location or area of ​​the bleeding point along with the camera image IMG on the display unit 400. The control unit 110 includes a parameter determination unit 116, which determines a control parameter CPAR based on the information EKINF. The light source control unit 111 uses the control parameter CPAR from the parameter determination unit 116 to control the light intensity ratio of each light source of the illumination unit 140.

[0126] In the above Figures 13 to 16 In the example shown in FIG. 1 , the processing unit 120 performs processing based on a learned model 121 obtained by machine learning the relationship between the learning image and bleeding spots in the observation object. The processing unit 120 detects bleeding spots based on the captured image IMG captured by the imaging unit 230 and the learned model 121. Specifically, the learned model 121 is obtained by machine learning the relationship between the learning image, information about the liquid, and bleeding spots. Alternatively, the learned model 121 is obtained by machine learning the relationship between the learning image, information about the illumination light used to capture the learning image, and bleeding spots.

[0127] In this way, by using the inference processing of machine learning, bleeding spots can be detected from the captured images. By using machine learning, it is expected that further optimized bleeding spot detection can be achieved for various lighting light or liquid conditions.

[0128] Alternatively, the learned model for determining control parameters and the learned model for detecting bleeding points can be integrated. In this case, the training data associates at least the learning image, the control parameters, and the bleeding point information. The learning image serves as the input data, while the control parameters and bleeding point information serve as the correct labels. The control unit 110 inputs the captured image to the learned model and obtains the control parameters and bleeding point information through derivation processing using the learned model. Alternatively, the processing unit 120 can perform this derivation processing.

[0129] A part or all of the control unit 110 and the processing unit 120 of the present embodiment described above may be realized by a program. In this case, the endoscope system 10 may be configured as follows.

[0130] Specifically, the endoscope system 10 includes a memory that stores information and a processor that operates based on the information stored in the memory. Examples of the information include programs and various data. The programs describe some or all of the functions of the control unit 110 and the processing unit 120. The processor executes these programs to implement some or all of the functions of the control unit 110 and the processing unit 120.

[0131] Specifically, the endoscope system 10 includes an illumination unit 140, an imaging unit 230, and a processor. The processor controls the illumination unit 140 to emit illumination light with different spectra, depending on the presence of liquid in the observation object or the transmittance of the liquid. While only a portion of this embodiment is described here, the processor can perform some or all of the functions of the control unit 110 and processing unit 120 described in this embodiment.

[0132] The processor may include hardware, and the hardware may include at least one of a circuit for processing digital signals and a circuit for processing analog signals. For example, the processor may be composed of one or more circuit devices and one or more circuit elements mounted on a circuit substrate. One or more circuit devices are, for example, ICs, etc. One or more circuit elements are, for example, resistors, capacitors, etc. The processor may be, for example, a CPU (Central Processing Unit). However, the processor is not limited to the CPU, and various processors such as a GPU (Graphics Processing Unit) or a DSP (Digital Signal Processor) may be used. In addition, the processor may also be an integrated circuit device such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). In addition, the processor may also include an amplifier circuit, a filter circuit, etc. for processing analog signals. The memory may be a semiconductor memory such as SRAM, DRAM, or a register, or a magnetic storage device such as a hard disk device, or an optical storage device such as an optical disk device. For example, a memory stores computer-readable instructions, and the processor executes these instructions to implement the functions of each part of the endoscope as processes. The instructions here can be commands that constitute a program command set or commands that instruct the processor's hardware circuits to operate.

[0133] The above-described program can be stored in, for example, a computer-readable information storage medium. The information storage medium can be implemented, for example, as an optical disc, a memory card, a HDD, or a semiconductor memory. An example of a semiconductor memory is a ROM. The control unit 110 and the processing unit 120 perform various processes of this embodiment based on the program and data stored in the information storage medium.

[0134] Furthermore, the embodiment described above can also be implemented as an illumination control method. This illumination control method can also be referred to as an operating method for the endoscope system 10. Specifically, the illumination control method controls the illumination of spectrally variable illumination light, images the object being illuminated by the illumination light, and illuminates the object with varying illumination light spectra depending on the presence of liquid within the object or the transmittance of the liquid. While this is a partial description of the embodiment, the operations of the endoscope system 10 described in this embodiment can also be implemented as an illumination control method.

[0135] 3. Application Examples

[0136] Several examples will be described as specific application targets of the endoscope system 10 of the present embodiment.

[0137] The first application example is a urinary endoscope such as a cystourethroscope and a prostatectomy. A prostatectomy is an endoscope used for prostate removal. In the treatment using a urinary endoscope, the following first and second scenarios are envisioned.

[0138] The first scenario involves observing a red, turbid liquid, blood. In this scenario, the following methods are used: illumination light in a wavelength band where hemoglobin has low absorbance, or illumination light with increased intensity at wavelengths where hemoglobin has high absorbance. Consider the following first and second spectral examples.

[0139] In the first spectrum example, based on white illumination light using red, green, blue, and violet light, the amount of blue light is set to > the amount of violet light, and the amount of first green light is set to > the amount of second green light, thereby maintaining whiteness and improving visual recognition. The second green light and the first green light both belong to the green wavelength band, and the wavelength of the second green light is longer than that of the first green light. The peak wavelength of the second green light is around 580nm. The absorption characteristics of hemoglobin have absorption peaks in the wavelength bands of violet light and the second green light. Therefore, the transmittance of the illumination light is improved by relatively reducing the amount of light in these wavelength bands. In addition, since the wavelength of blue light is > that of violet light and the wavelength of the first green light is < that of the second green light, the effects of light intensity adjustment on color tone are in opposite directions, making it easier to maintain whiteness.

[0140] In the second spectrum example, visibility is improved by using illumination light with a higher proportion of blue light compared to white illumination light. When the visible light range of illumination light is divided into blue, green, and red, hemoglobin has a high absorbance for blue, which reduces the blue component in the captured image. In the second spectrum example, increasing the proportion of blue light compensates for the reduction in the blue component in the image.

[0141] The second scenario involves observing a relatively large tissue piece, where the liquid appears whitish. In this scenario, illumination light with a long-wavelength component that is less susceptible to scattering is used. Using only red light would produce a low-resolution image of the surface structure, so blue and green light are also used as illumination light. In the spectral example for the second scenario, the amount of red light is set to > (the amount of violet light + the amount of blue light), using white illumination light using red, green, blue, and violet light as a reference.

[0142] The second application example is an arthroscope used for joint surgery. During arthroscope treatment, consider an observation scenario where fluid appears whitish due to tiny bone fragments. In this scenario, the influence of scattering is greater than in the second scenario using a uroscope, so illumination light with a higher concentration of long-wavelength components is used. Specifically, in this scenario, the amount of red light is increased by (the amount of violet light + the amount of blue light) compared to the second scenario using a uroscope.

[0143] 4. Second Configuration Example

[0144] Figure 17 This is a second configuration example of the endoscope system 10. The endoscope system 10 includes a control device 100, a scope 200, a display unit 400, and an observation mode switching button 420. Components already described in the first configuration example are denoted by the same reference numerals, and descriptions of these components are omitted as appropriate.

[0145] The scope 200 includes an emission unit 210, an imaging unit 230, a water supply port 250, and a connector 240. The water supply port 250 is an opening provided at the front end of the scope 200 for supplying water to the observation object 1. The control device 100 is also provided with a connector 240, and the scope 200 and the control device 100 are connected via the connector 240.

[0146] The control device 100 includes a control unit 110, a processing unit 120, a storage unit 130, and an illumination unit 140. The control unit 110 includes a light source control unit 111, a system control unit 112, and an illumination mode switching control unit 113. The storage unit 130 stores a light source information database 131 and a water information database 132. The illumination unit 140 includes light sources LDR, LDA, LDG, LDB, and LDV. If amber light is not used, the light source LDA can be omitted.

[0147] The processing unit 120 performs image processing on the imaging signal from the imaging unit 230 to generate a captured image. The system control unit 112 causes the display unit 400 to display the captured image from the processing unit 120.

[0148] The system control unit 112 switches the observation mode based on the operation input to the observation mode switching button 420. The lighting mode switching control unit 113 selects a predetermined light source control condition from the setting table contained in the light source information database 131 according to the switched observation mode. In addition, the lighting mode switching control unit 113 obtains image information based on the camera image from the processing unit 120 and selects the light source control condition based on the image information. The image information is, for example, the color, brightness, or a combination thereof of the image. The system control unit 112 outputs the selected light source control condition to the light source control unit 111. Based on the light source control condition, the light source control unit 111 outputs a control parameter for controlling the light amount of each light source to the lighting unit 140. Each light source of the lighting unit 140 emits light according to the light amount indicated by the control parameter. The illumination light emitted by the lighting unit 140 is irradiated onto the observation object 1 from the emission portion 210 of the scope 200.

[0149] The water supply device 150 supplies water from the water supply port 250 of the scope 200 to the observation object 1 according to an instruction from the system control unit 112. The water supply device 150 corresponds to the supply unit described in the first configuration example.

[0150] The information contained in the light source information database 131 is described. The light source information database 131 has information on the peak wavelength and the relationship between the driving current and the light output for each light source mounted on the endoscope system 10. In addition, the light source information database 131 has information on the output balance of white light achieved when each light source is combined and information on the total amount of illumination light at that time. The peak wavelength and wavelength band of each light source, as well as examples of white light, are as follows. Figure 5 Although not shown, the lighting unit 140 may include a photodiode or the like for monitoring the light amount of each light source. The light source control unit 111 may also control each light source based on the output of the photodiode and the light source control conditions.

[0151] The light source information database 131 also includes information on the light distribution characteristics of the illumination light emitted from the illumination window and the central brightness derived from the illumination light quantity and the light distribution characteristics. The illumination window is provided at the front end of the scope 200 and is a window through which the illumination light is emitted. Figure 17 The portion of the illumination lens of the emission unit 210 shown exposed at the front end of the mirror body 200 corresponds to the illumination window. Light distribution characteristics can also be maintained for the light emitted by each light source. The brightness at the center is also called the center intensity. The light distribution characteristic represents the relationship between the angle between the center line of the illumination window and the illumination direction, with the angle set at 0 degrees, and the illumination intensity in that direction. For example, the intensity is maximum at 0 degrees and decreases with increasing angles.

[0152] Furthermore, the light source information database 131 includes information related to the configuration of the lighting windows based on the ID information of the scope 200 connected to the control device 100. The information related to the configuration of the lighting windows includes, for example, the number, arrangement, and diameter of the lighting windows provided at the front end of the scope 200, or a combination thereof.

[0153] The information contained in water information database 132 will be described. Water information database 132 contains information on the transmittance of water delivered from water delivery port 250 in the visible light range. Transmittance information includes, for example, the relationship between wavelength and absorption coefficient, the relationship between wavelength and scattering coefficient, or both. Furthermore, water information database 132 contains information on the refractive index of water, as an optical property related to the material of water. This refractive index information is based on Fresnel reflection of water. For example, water information database 132 contains information on the refractive index of water as 1.333.

[0154] The water information database 132 also contains biological information of the subject to be observed. The biological information is, for example, information on the transmittance of hemoglobin contained in blood. Figure 18 An example of biological information is shown. Information on the transmittance of hemoglobin is, for example, the relationship between wavelength and absorption coefficient, the relationship between wavelength and scattering coefficient, or both.

[0155] Figure 19 、 Figure 20 It is a diagram for explaining the operation of the second structural example. Figure 19 This shows an example of light balance settings. Figure 20 This shows an example of setting the brightness correction coefficient.

[0156] The lighting mode switching control unit 113 extracts the light balance and brightness correction value of the corresponding underwater lighting mode stored in the light source information database 131 according to the underwater observation mode set by the observation mode switching button 420. There are multiple settings for the underwater observation mode depending on the water supply amount and the conditions of the water mixed with blood. Here, the underwater observation mode is set to three modes, underwater 1 to underwater 3, but it is not limited to this. The lighting mode switching control unit 113 changes the driving conditions of each light source based on the extracted correction conditions, thereby changing the color, brightness, or both of the illumination light during underwater observation. In addition, as a trigger for switching the lighting conditions of the underwater lighting mode, the lighting mode switching control unit 113 can also switch to underwater 1 to 3 based on the pressing time of the water supply button.

[0157] exist Figure 19 and Figure 20 In the "Normal" mode, it refers to the normal observation mode other than the underwater observation mode, and is a mode that emits white illumination light. Figure 19 In the example, V1:V2:V3:V4 are the light intensity ratios of the light sources LDV, LDB, LDG, and LDR. The color temperature of the illumination light is set according to each observation mode, and the light intensity ratio is set to achieve the color temperature. Figure 20 In the example, the coefficient is a factor that is multiplied by the light intensity and is applied to all light sources. In other words, the total amount of illumination light is controlled by the coefficient.

[0158] Figures 21 to 25 This is an example of illumination light when five light sources including an amber light source are used. Figure 21 Indicates the spectral characteristics of the light emitted by each light source. LDV, LDB, LDG, LDA, and LDR have peak wavelengths around 410nm, 460nm, 510nm, 600nm, and 640nm, respectively. LDV, LDB, LDA, and LDR have half-value widths of around tens of nm, and LDG has a half-value width of around 100nm. Figure 21 In the figure, the integrated value of the spectrum of each light source is normalized to 1.

[0159] exist Figures 22 to 25 , the spectrum of the entire illumination light when each light source emits light according to the set light amount is shown.

[0160] Figure 22 This is the first example of illumination light. This first example corresponds to the white illumination light used in normal observation mode. Specifically, this illumination light combines the light sources LDV, LDB, LDA, and LDR at a predetermined ratio, creating a spectrum close to white. In this first example, the amber light source LDA is off.

[0161] Figure 23This is the second example of illumination light. This second example is used in underwater observation mode and is selected when the liquid is turbid. The ratio of the long-wavelength component in this second example is increased compared to the first example. This long-wavelength component is the red light emitted by the light source LDR. Increasing the long-wavelength component in this illumination light counteracts turbidity while maintaining a spectrum close to white light. In this second example, the amber light source LDA is turned off.

[0162] Figure 24 This is the third example of illumination light. This example is used in underwater observation mode and is selected when the liquid is turbid due to blood. In this example, the ratio of the light intensity of the blue light source LDB to the light intensity of the violet light source LDV is increased compared to the first example. Here, the violet light source LDV is turned off, and the light intensity ratio of the blue light source LDB is increased compared to the first example. In this illumination light, the turbidity caused by blood is addressed by increasing the light intensity ratio of the blue light source LDB to the violet light source LDV, while maintaining a spectrum close to white light. In this third example, the amber light source LDA is turned off.

[0163] Figure 25 This is the fourth example of illumination light. This example is used in underwater observation mode and is selected to facilitate the observation of bleeding points. In this fourth example, the amber light source LDA is added to the third example. Furthermore, the light intensity ratio of the red light source LDR is the same as in the first example of ordinary light. By utilizing amber light with a peak wavelength near 600nm, this illumination light emphasizes uneven blood concentration and facilitates the observation of bleeding points.

[0164] The second configuration example described above includes a table that takes into account the amount of absorbed or reflected illumination light based on the physical properties of the water delivered from the water delivery device 150, namely, absorption, scattering, and refractive index, and the physical properties of hemoglobin and blood, namely, absorption, scattering, and refractive index. This table corrects the balance and brightness of the illumination light. By having such a table, it is possible to set illumination light suitable for each underwater observation mode. This allows observation to be performed under appropriate illumination conditions for both normal and underwater observation. Furthermore, by controlling the illumination light using multiple light sources, more detailed color and brightness adjustments can be achieved.

[0165] 5. Third structural example

[0166] A third configuration example using AI processing in the processing unit 120 and the lighting mode switching control unit 113 will be described. Figure 17 The same as the second configuration example shown, the third configuration example may also omit the observation mode switching button 420. Components already described in the first or second configuration example are denoted by the same reference numerals, and descriptions of these components are omitted as appropriate.

[0167] In the third structural example, the processing unit 120 detects the degree of water contained in the camera image, the degree of water turbidity, and the amount of blood contained in the water through AI processing. Here, AI processing refers to the inference processing using a learned model obtained through machine learning. Specifically, the processing unit 120 identifies the observation state through AI processing based on the camera image and the water information database 132. The observation state is underwater observation with the front end of the scope 200 and the observation object 1 in water, or underwater observation with the front end of the scope 200 not in water but the observation object 1 in water. In addition, the processing unit 120 identifies the water state in the camera image through AI processing based on the camera image and the water information database 132. The water state is, for example, the boundary between an area with water and an area without water, an area with water, the color concentration of the image in the area with water, or a combination thereof.

[0168] The lighting mode switching control unit 113 uses AI processing to correct the light intensity ratio of each light source based on the observation state and water state identified by the processing unit 120. Specifically, machine learning is performed on the observation state and water state based on normal observation images, and the learned model obtained through this machine learning is stored in the storage unit 130 of the endoscope system 10. The lighting mode switching control unit 113 uses the derivation processing of the learned model to correct and optimize the lighting conditions. During the image recognition and AI processing, the correction light intensity can also be updated and controlled based on changes in the observation state or water state within the image over time.

[0169] Figure 26 This is a detailed example of the configuration of the processing unit 120 and the lighting mode switching control unit 113 in the third configuration example. The processing unit 120 includes a water detection unit 161 and a water state identification unit 162. The lighting mode switching control unit 113 includes a light correction mode extraction unit 163 and a light correction value setting unit 164. These units perform AI processing.

[0170] Figure 27This is a processing flow chart in the third structural example. The description starts with the process in normal observation shown in S1. In step S11, the camera unit 230 captures the observation object 1, and the processing unit 120 generates a camera image. In step S12, the system control unit 112 and the light source control unit 111 perform normal light intensity control based on the brightness of the camera image. Normal light intensity control is a dimming control that aims to maintain the brightness of the camera image at a constant level. In step S13, a water supply on / off signal indicating whether the water supply is on or off is input from the system control unit 112 to the water detection unit 161. The water detection unit 161 determines whether the water supply from the water supply device 150 to the observation object 1 is on based on the water supply on / off signal. When the water detection unit 161 determines that the water supply is off, steps S11 to S13 are repeated.

[0171] If the water detection unit 161 determines that water supply is on, the above-water or underwater observation process shown in S2 is executed. In step S21, the imaging unit 230 captures the observation object 1, and the processing unit 120 generates a captured image. In step S22, the water detection unit 161 determines whether water supply from the water supply device 150 to the observation object 1 is on based on the water supply on / off signal. If the water detection unit 161 determines that water supply is off, the normal observation process shown in S1 is executed. If the water detection unit 161 determines that water supply is on, in step S23, the water state recognition unit 162 extracts the water state through AI processing. The water state can be the boundary between areas with water and areas without water, areas with water, the color density of the image within the areas with water, or a combination thereof. In step S24, based on the water state extracted through AI processing, the light intensity increase / decrease correction value for each light source is extracted based on the information stored in the light source information database 131 and the water information database 132. The light intensity increase / decrease correction value is, for example, a correction value for obtaining an image with the same color and brightness as normal light, or a correction value corresponding to the amount of blood, such as when using amber light for illumination. In step S24, the light intensity correction value setting unit 164 outputs light source control information to the system control unit 112 based on the light intensity increase / decrease correction value. The system control unit 112 and the light source control unit 111 then perform light intensity correction for each light source based on the light source control information.

[0172] According to the third configuration example described above, AI processing is performed based on information obtained through image recognition regarding water areas, such as whether they are transparent, mixed with blood, or turbid. This AI processing extracts correction values ​​for the illumination light that take into account the optical properties of water, namely absorption, scattering, and reflection. By using these correction values ​​to modify the illumination light, it is possible to provide an easily observable image that minimizes the effects of water. Furthermore, even if the state of the water changes over time, the AI ​​processing can use the water information and learning data generated through image recognition to set the appropriate illumination color and brightness, thereby creating a lighting pattern suitable for underwater observation conditions.

[0173] The present embodiment and its variations have been described above, but the present invention is not directly limited to the embodiments and their variations. During the implementation stage, the technical features can be modified and concretized without departing from the main purpose. In addition, the multiple technical features disclosed in the above-mentioned embodiments and variations can be appropriately combined. For example, several technical features can be deleted from all the technical features described in the embodiments and variations. Furthermore, the technical features described in different embodiments and variations can also be appropriately combined. In this way, various modifications and applications can be made without departing from the main purpose of the present invention. In addition, in the specification or the drawings, a term that is recorded at least once together with a different term in a broader sense or with the same meaning can be replaced with the different term at any position in the specification or the drawings.

[0174] Description of labels

[0175] 1 Observation object, 2 Body cavity, 5 Liquid, 7 Illumination light, 10 Endoscope system, 22 Learned model, 100 Control device, 110 Control unit, 111 Light source control unit, 112 System control unit, 113 Illumination mode switching control unit, 115 Learned model, 116 Parameter determination unit, 120 Processing unit, 121 Learned model, 123 Display processing unit, 130 Storage unit, 131 Light source information database, 132 Water information database, 140 Illumination unit, 150 Water delivery device, 161 Water detection unit, 162 Water state recognition unit, 163 Light correction mode extraction unit, 164 Light correction value setting unit, 200 Mirror body, 210 Emission unit, 220 Operation unit, 230 Camera unit, 240 Connector, 250 Water supply port, 400 Display unit, 420 Observation mode switching button, 500 Learning device, 510 Processing unit, 511 Learning model, 512 Update unit, 520 Storage unit, 522 Training data, 523 Input data, 524 Correct answer label, CPAR control parameter, IMG Camera image, LDA, LDB, LDG, LDR, LDV light source.

Claims

1. An endoscope system, characterized in that: The endoscope system comprises: a light source that generates illumination light; an imaging element that captures an image of an observation object irradiated with the illumination light; and processor, The processor is configured to perform the following processing: Determine whether there is liquid in the observation object, When the liquid does not exist in the observation object, switching to a first observation mode in which the observation object is illuminated with a first illumination light, When the liquid is present in the observation object, switching to a second observation mode in which the observation object is illuminated with a second illumination light, The ratio of the relatively long wavelength component of the second illumination light is greater than the ratio of the relatively long wavelength component of the first illumination light.

2. The endoscope system according to claim 1, wherein: The processor controls switching between a plurality of observation modes, the plurality of observation modes including two or more of the first observation mode, the second observation mode, and the third observation mode, the first observation mode being selected when the liquid is not present, the second observation mode being selected when the transmittance of the liquid is relatively high, and the third observation mode being selected when the transmittance of the liquid is relatively low. The processor performs at least one of the following controls: switching the illumination light between the first illumination light associated with the first observation mode and the second illumination light associated with the second observation mode according to switching between the first observation mode and the second observation mode; switching the illumination light between the second illumination light associated with the second observation mode and the third illumination light associated with the third observation mode according to switching between the second observation mode and the third observation mode; and switching the illumination light between the first illumination light associated with the first observation mode and the third illumination light associated with the third observation mode according to switching between the first observation mode and the third observation mode.

3. The endoscope system according to claim 2, wherein: The third illumination light is illumination light having a larger ratio of a relatively long-wavelength component in a spectrum than a relatively long-wavelength component in a spectrum of the second illumination light.

4. The endoscope system according to claim 3, wherein: The second illumination light and the third illumination light are illumination lights obtained by increasing the long wavelength component of ordinary light. The degree of increase of the long-wavelength component in the third illumination light is greater than the degree of increase of the long-wavelength component in the second illumination light.

5. The endoscope system according to claim 3, wherein: The second illumination light is obtained by increasing the long wavelength component of ordinary light. The third illumination light is special light including narrow-band light corresponding to the long-wavelength component.

6. The endoscope system according to claim 2, wherein: The observation object is a living organism, said fluid comprising blood, At least one of the second illumination light and the third illumination light includes light in an amber wavelength band.

7. The endoscope system according to claim 2, wherein: The observation object is a living organism, said fluid comprising blood, A ratio of a blue component to a violet component in a spectrum of at least one of the second illumination light and the third illumination light is greater than a ratio of the blue component to the violet component in a spectrum of normal light.

8. The endoscope system according to claim 2, wherein: The processor performs processing based on the image captured by the image sensor. The third illumination light includes light in an amber wavelength band, The processor performs a process of detecting bleeding spots in the observation object based on the captured image captured when the observation object is irradiated with the third illumination light, and performs a process of displaying the detected bleeding spots.

9. The endoscope system according to claim 1, wherein: The endoscope system includes a supply unit that supplies the liquid to the observation object. The processor controls the light source so that the ratio of the relatively long wavelength component in the spectrum of the illumination light increases as the supply amount of the liquid supplied by the supply unit increases.

10. The endoscope system according to claim 1, wherein: The observation object is a living organism, The processor controls the light source so that the more body fluid or tissue pieces contained in the liquid, the greater the ratio of the relatively long-wavelength component in the spectrum of the illumination light.

11. The endoscope system according to claim 1, wherein: When the transmittance of the liquid is equal to or greater than a predetermined threshold, the processor controls the light source so that a difference between an image captured by the imaging element and a reference image captured in the absence of the liquid is equal to or less than a predetermined threshold.

12. The endoscope system according to claim 11, wherein: When the transmittance of the liquid is less than the predetermined threshold, the processor controls the light source so that a difference between an image captured by the imaging element and a second reference image different from the reference image is less than or equal to a predetermined value.

13. The endoscope system according to claim 1, wherein: The processor performs processing based on the image captured by the image sensor. When the processor controls the light source so that the ratio of the relative long-wavelength component in the spectrum of the illumination light increases, the processor performs image processing on the captured image to reduce the contribution of the long-wavelength component.

14. The endoscope system according to claim 1, wherein: The processor performs processing based on the image captured by the image sensor. The processor performs structure enhancement processing on the captured image so that the contrast of the target area becomes equal to or greater than a predetermined threshold.

15. The endoscope system according to claim 1, wherein: The processor performs processing based on a learned model, wherein the learned model is obtained by machine learning of a relationship between a learning image and information related to the liquid, or a relationship between the learning image and information related to the recommended illumination light. The processor determines a control parameter for controlling the spectrum of the illumination light based on the image captured by the imaging element and the learned model, and controls the light source based on the determined control parameter.

16. The endoscope system according to claim 15, wherein: The learned model is a model obtained by machine learning the relationship between the learning image, the information of the illumination light used in shooting the learning image, and the information related to the liquid, or a model obtained by machine learning the relationship between the learning image, the information of the illumination light used in shooting the learning image, and the information related to the recommended illumination light.

17. The endoscope system according to claim 1, wherein: The processor performs processing based on a learned model, wherein the learned model is obtained by machine learning of a relationship between a learning image and bleeding points in the observation object. The processor detects the bleeding point based on the image captured by the imaging element and the learned model.

18. The endoscope system according to claim 17, wherein: The learned model is a model obtained by machine learning the relationship between the learning image, the information related to the liquid, and the bleeding points, or a model obtained by machine learning the relationship between the learning image, the information of the illumination light used in shooting the learning image, and the bleeding points.

19. A control device, characterized in that: The control device includes a processor configured to perform the following processing: Determine whether there is liquid in the observed object, When the liquid does not exist in the observation object, switching to a first observation mode in which the observation object is illuminated with a first illumination light, When the liquid is present in the observation object, switching to a second observation mode in which the observation object is illuminated with a second illumination light, The ratio of the relatively long wavelength component of the second illumination light is greater than the ratio of the relatively long wavelength component of the first illumination light.

20. A control method for a control device, characterized in that: The control method of the control device comprises the following steps: Determine whether there is liquid in the observed object, When the liquid does not exist in the observation object, switching to a first observation mode in which the observation object is illuminated with a first illumination light, When the liquid is present in the observation object, switching to a second observation mode in which the observation object is illuminated with a second illumination light, The ratio of the relatively long wavelength component of the second illumination light is greater than the ratio of the relatively long wavelength component of the first illumination light.

Images