Endoscope system and method for controlling endoscope system

Abstract

An endoscope system (10) is provided with an illumination unit (140), an imaging unit (230), and a control unit (110). The illumination unit (140) emits illumination light having a variable spectrum. The imaging unit (230) captures an image of the object to be observed irradiated with the illumination light. The control unit (110) controls the illumination unit (140) so as to irradiate illumination light having different spectrums in accordance with whether or not a liquid is present in the object to be observed, or in accordance with the transmittance of the liquid.

Description

[0001] This application is filed based on Article 48 of the Detailed Rules of the Patent Law, and is a divisional application of the invention patent application “Endoscope system and illumination control method” (International Application No. PCT / JP2020 / 018636) with the filing date of May 8, 2020 and the application number of 202080100259.7. TECHNICAL FIELD

[0002] The present application relates to an endoscope system and a control method of an endoscope system. BACKGROUND

[0003] In diagnosis or surgery using an endoscope, or the like, sometimes water observation is performed on an object, or the object is observed via a liquid accumulated in an observation region. For example, in a bladder endoscope or the like, a scope is inserted into a body cavity filled with a liquid to perform water observation. Alternatively, even in a use or a situation not premised on water observation, it is conceivable that the object is observed via water or a body fluid or the like delivered from an endoscope. In Patent Literature 1, an endoscope device is disclosed in which an illumination unit has a plurality of light sources having different emission wavelengths, a color tone control section that changes a ratio of each emission light amount to change a color tone of illumination light, and is capable of providing an appropriate color tone to an observation object.

[0004] PRIOR ART DOCUMENTS

[0005] PATENT LITERATURE

[0006] Patent Literature 1: Japanese Patent Application Publication No. 2017-136405 SUMMARY

[0007] PROBLEMS TO BE SOLVED BY THE INVENTION

[0008] In the water observation or the like described above, sometimes the transmittance of the liquid decreases due to the mixing of tissue or a body fluid or the like in the liquid, and the visual recognition of the observation object decreases. In such a case, it is desirable to be able to improve the visual recognition of the observation object, but in the past, there has been no disclosure of a method of adjusting an observation image to be easily observed by putting effort on illumination for observation. For example, in the above-described Patent Literature 1, there is no disclosure of a structure of changing an illumination mode in correspondence with a turbidity state of water provided to an observation object.

[0009] MEANS FOR SOLVING THE PROBLEMS

[0010] One embodiment of the present application relates to an endoscope system including: an illumination section that irradiates illumination light whose spectrum is variable; an imaging section that images an observation object on which the illumination light is irradiated; and a control section that controls the illumination section to irradiate the illumination light whose spectrum is different, in accordance with whether or not a liquid is present in the observation object or in accordance with a transmittance of the liquid.

[0011] Further, another aspect of the present application relates to an illumination control method, which controls to radiate illumination light whose spectrum is variable, to perform imaging of an observation object to which the illumination light is radiated, and to radiate the illumination light whose spectrum is different depending on whether or not a liquid exists in the observation object or depending on a transmittance of the liquid. BRIEF DESCRIPTION OF DRAWINGS

[0012] Figure 1 is an example of observation using an endoscope system.

[0013] Figure 2 is a first configuration example of an endoscope system.

[0014] Figure 3 is an example of switching an observation mode depending on whether or not a liquid exists.

[0015] Figure 4 is an example of switching an observation mode depending on a transmittance of a liquid.

[0016] Figure 5 is an example of control to increase a long wavelength component of normal light.

[0017] Figure 6 is an example of switching an observation mode depending on a blood amount of a liquid.

[0018] Figure 7 is an example of special light including amber light.

[0019] Figure 8 is a configuration example of a learning device.

[0020] Figure 9 is a first example of training data.

[0021] Figure 10 is a configuration example of an endoscope system in a case where a learned model learned through the first example of training data is used.

[0022] Figure 11 is a second example of training data.

[0023] Figure 12 is a configuration example of an endoscope system in a case where a learned model learned through the second example of training data is used.

[0024] Figure 13 is a third example of training data.

[0025] Figure 14 is a configuration example of an endoscope system in a case where a learned model learned through the third example of training data is used.

[0026] Figure 15is the fourth example of the training data.

[0027] Figure 16 is a configuration example of the endoscope system in a case where the learned model learned through the fourth example of the training data is used.

[0028] Figure 17 is a second configuration example of the endoscope system.

[0029] Figure 18 is an example of the biological information.

[0030] Figure 19 is a configuration example of the light amount balance.

[0031] Figure 20 is a configuration example of the brightness correction coefficient.

[0032] Figure 21 is a first example of the illumination light in a case where five light sources to which an amber light source is added are used.

[0033] Figure 22 is a second example of the illumination light in a case where five light sources to which an amber light source is added are used.

[0034] Figure 23 is a third example of the illumination light in a case where five light sources to which an amber light source is added are used.

[0035] Figure 24 is a fourth example of the illumination light in a case where five light sources to which an amber light source is added are used.

[0036] Figure 25 is a fifth example of the illumination light in a case where five light sources to which an amber light source is added are used.

[0037] Figure 26 is a detailed configuration example of the processing section and the illumination mode switching control section in the third configuration example.

[0038] Figure 27 is a processing flowchart in the third configuration example. DETAILED DESCRIPTION

[0039] Hereinafter, the present embodiment will be described. Note that the present embodiment described below does not unduly limit the content recited in the claims. Also, all the configurations described in the present embodiment are not necessarily essential technical features of the present application.

[0040] 1. First configuration example of endoscope system

[0041] Figure 1An example of observation using the endoscope system of this embodiment is shown. Furthermore, the following description uses the case where the body cavity is filled with fluid as an example; however, the endoscope system of this embodiment can be applied to situations where there is fluid between the tip of the endoscope and the object being observed, i.e., situations where illumination light is shone onto the object through the fluid and the object is observed through that fluid.

[0042] like Figure 1 As shown, a body cavity 2, such as a bladder or joint, is filled with fluid 5. Fluid 5 may be water, bodily fluids, or a mixture thereof. Additionally, tissue fragments detached from the body cavity 2 may also be contained in the fluid 5. The endoscope 200 is inserted into the body cavity 2, and illumination light 7 is projected from its tip through the fluid 5 onto the object of observation 1. Return light from the object of observation 1 reaches the endoscope 200 through the fluid 5, and the endoscope 200 captures an image of the object of observation 1. The object of observation 1 is the object observed by the user within the body cavity 2, such as a lesion. However, the object of observation 1 is not limited to lesions; any area within the body cavity 2 visible within the field of view of the endoscope 200 is designated as the object of observation 1.

[0043] In such underwater observations, the visual discernibility of the observed object 1 is reduced due to the spectral and scattering properties of liquid 5. Furthermore, the visual discernibility of the observed object 1 varies depending on the type of liquid 5, the concentration of the mixed body fluids, the type of tissue fragment, the amount of tissue fragments, or a combination thereof. For example, if liquid 5 is almost pure water, the observed image is affected by the spectral and scattering properties of water. Or, if liquid 5 contains blood, the illumination light has difficulty reaching the observed object 1 due to the light absorption properties of hemoglobin, and the field of view turns red, thus reducing the visual discernibility of the observed object 1. The degree of this reduction in visual discernibility varies depending on the concentration of blood mixed in liquid 5. Alternatively, if detached mucosal cells or bone fragments are floating in liquid 5, the illumination light is scattered by these floating mucosal cells, making it difficult for the illumination light to reach the observed object 1, and reducing the contrast of the observed image, thus reducing the visual discernibility of the observed object 1. The degree of this reduction in visual discernibility varies depending on the size and concentration of the mucosal cells, etc., mixed in liquid 5. In this embodiment, by focusing on the illumination light 7 used to observe the object 1, the reduction in visual recognition caused by the liquid 5 described above is improved.

[0044] Figure 2is a first configuration example of the endoscope system 10 of the present embodiment. The endoscope system 10 includes a control device 100, a scope 200, and a display section 400. As the endoscope system 10, for example, a cystoscope or the like, or an endoscope system of a urinary system such as a prostate resectoscope, or a scope for a joint for surgery or observation, or the like can be assumed. However, the endoscope system 10 can be either of a rigid scope or a flexible scope, and for example, can be an endoscope system for various uses and sites such as surgery or organ observation.

[0045] The scope 200 is a portion that is inserted into the body cavity 2 to perform illumination and imaging of the inside of the body cavity 2, and includes an emission section 210, an operation section 220, and an imaging section 230. In addition, the scope 200 can also include a treatment instrument such as an electrosurgical knife or forceps.

[0046] The emission section 210 is a device that emits illumination light from the front end of the scope 200 toward the observation target 1. The emission section 210 includes, for example, a light guide that guides light from an illumination section 140 provided to the control device 100, and a lens that irradiates the light guided by the light guide toward the observation target 1. In addition, the illumination section 140 can also be provided separately from the control device 100.

[0047] The operation section 220 is a device for a user to operate the endoscope system 10. The operation section 220 includes, for example, a button, a switch, or a dial, or the like. In the present embodiment, the operation section 220 includes a water feeding button for controlling water feeding from a water feeding port of the scope 200.

[0048] The imaging section 230 is a device that images the inside of the body cavity 2, and outputs an imaging signal obtained by imaging to the control device 100. The imaging section 230 includes, for example, an objective lens and an image sensor that images an image formed by the objective lens.

[0049] The control device 100 performs image processing based on the imaging signal and action control of the endoscope system 10. The control device 100 includes a control section 110, a processing section 120, a storage section 130, and an illumination section 140. The control device 100 is a device that houses a circuit board on which the control section 110, the processing section 120, and the storage section 130 are provided, and the illumination section 140 in a housing. Alternatively, the control device 100 can also be realized by a general-purpose information processing device such as a PC executing software that describes the actions of the control device 100. Alternatively, a part of the control device 100 can also be realized by an information processing device or a cloud system separate from the control device 100. For example, the AI processing and the like described later can also be executed by an information processing device provided separately from the control device 100, or a cloud system connected to the control device 100 through a network.

[0050] The illumination section 140 is a light source device that generates illumination light, and causes the illumination light to be incident on the light guide of the emission section 210. The illumination section 140 includes a plurality of light sources that can independently control the light amount and whose wavelength bands of emitted light are different from each other. Each light source is, for example, an LED, a laser diode, a light source that generates fluorescent light by irradiating a phosphor with emitted light from a laser diode, or a combination thereof. Alternatively, the illumination section 140 can also be implemented by a white light source such as a xenon lamp and a filter device that can switch a plurality of optical filters that pass a desired wavelength band. The transmittance spectral characteristics of the plurality of optical filters are different from each other. In addition, in the present embodiment, the emission section 210 that is a lens and a light guide and the illumination section 140 that is a light source device are separately shown, but the lens, the light guide, and the light source device can be included in the illumination section. Figure 2

[0051] The processing section 120 is a circuit or a device that performs control of each section of the endoscope system 10 and image processing of an image signal from the imaging section 230. The processing section 120 outputs the processed image to the display section 400 and displays it on the display section 400. The display section 400 is, for example, a display device such as a liquid crystal monitor. In addition, the processing section 120 outputs image information required to control the spectrum of the illumination light to the control section 110. The image information can be the imaging image itself, or can be an image parameter extracted from the imaging image. The image parameter is, for example, a contrast value, a color, a brightness, or the like.

[0052] The control section 110 is a circuit or a device that controls the spectrum of the illumination light by controlling the illumination section 140 based on the image information from the processing section 120. In the case where the illumination section 140 includes a plurality of light sources, the control section 110 controls the spectrum of the illumination light by generating a control parameter that controls the on-off and light amount of each light source in accordance with the image information. Alternatively, in the case where the illumination section 140 includes a white light source and a filter device, the control section 110 controls the spectrum of the illumination light by generating a control parameter that switches the optical filter of the filter device.

[0053] The control section 110 can also generate intermediate information that indicates the state of the liquid 5 in accordance with the image information, and generate a control parameter in accordance with the intermediate information. The intermediate information is, for example, the transmittance of the liquid 5, the kind of body fluid included in the liquid 5, the concentration of the body fluid included in the liquid 5, the kind of tissue piece included in the liquid 5, the concentration of the tissue piece included in the liquid 5, or a combination thereof. In addition, the intermediate information can be generated by the processing section 120 from the imaging image, and the control section 110 can generate a control parameter in accordance with the intermediate information from the processing section 120.

[0054] ​The processing section 120 and the control section 110 are each implemented by one or a plurality of circuits or devices. Alternatively, the processing section 120 and the control section 110 can also be constituted by an integrated circuit or device. The hardware constituting the processing section 120 and the control section 110 can be, for example, a processor, an FPGA, an ASIC, or a circuit board on which a plurality of circuit elements are mounted.

[0055] The storage section 130 stores various data or programs used in the processing performed by the control section 110 and the processing section 120. The storage section 130 is, for example, a semiconductor memory such as a RAM or a ROM, a magnetic storage device such as a hard disk drive, or an optical storage device such as an optical disk drive. Specifically, the storage section 130 stores data or programs for determining an appropriate spectrum of the illumination light in accordance with the state of the liquid 5.

[0056] For example, the storage section 130 can also store a learned model obtained through machine learning. In this case, the control section 110 infers the control parameter of the illumination light appropriate for the liquid 5 from the image information or the intermediate information through processing using the learned model. In addition, in the case of using the intermediate information, the control section 110 can also infer the control parameter from the image information through processing using the learned model.

[0057] Alternatively, the storage section 130 can also store a database for generating the above-described control parameter. In the database, for example, the spectral characteristics and scattering characteristics of the liquid 5 and the like are described. The control section 110, for example, uses the information of the contrast value, brightness, color, and the like of the captured image and the database to calculate the control parameter or the intermediate information. Alternatively, the database can be a lookup table in which the information of the contrast value, brightness, color, and the like of the captured image is associated with the control parameter or the intermediate information. The control section 110 acquires the control parameter or the intermediate information corresponding to the information of the contrast value, brightness, color, and the like of the captured image by referring to the lookup table.

[0058] Alternatively, the storage section 130 can also store a database for determining the control parameter of the illumination light from the intermediate information. The database is, for example, a lookup table in which the intermediate information such as the transmittance of the liquid 5 is associated with the control parameter of the appropriate illumination light. The control section 110 acquires the control parameter of the light amount of each light source or the optical filter to be selected corresponding to the intermediate information by referring to the lookup table.

[0059] According to the above-described embodiments, the endoscope system 10 includes an illumination section 140 that irradiates illumination light 7 whose spectrum is variable, a capturing section 230 that captures an observation target 1 that has been irradiated with the illumination light 7, and a control section 110. The control section 110 controls the illumination section 140 to irradiate the illumination light 7 whose spectrum is different, in accordance with the presence or absence of the liquid 5 in the observation target 1 or in accordance with the transmittance of the liquid 5.

[0060] Thus, even in a case where the visual recognizability of the observation object 1 is reduced due to the mixing of tissue or body fluid or the like in the liquid, the observation object 1 can be observed with the illumination light 7 of an appropriate spectrum in accordance with the presence or absence of the liquid 5 in the observation object 1 or in accordance with the transmittance of the liquid 5. The appropriate spectrum refers to a spectrum in which the visual recognizability of the observation object 1 via the liquid 5 is improved compared to a case where the observation is performed without changing the spectrum of the illumination light 7.

[0061] Specifically, the captured image is affected by the transmittance of the liquid 5, and becomes an image in which, for example, the contrast or the like is changed. Therefore, in a case where the captured image is input to the control section 110, the control parameter is determined in accordance with the captured image, and thus the spectrum control corresponding to the transmittance is realized. Alternatively, in a case where the transmittance of the liquid 5 is input to the control section 110 as the control information, the control section 110 determines the control parameter in accordance with the transmittance of the liquid 5, and thus the spectrum control corresponding to the transmittance is realized. Alternatively, the transmittance of the liquid 5 is determined by the kind or the concentration of the body fluid or the tissue piece included in the liquid 5. Therefore, in a case where the kind or the concentration of the body fluid or the tissue piece included in the liquid 5 is input to the control section 110 as the intermediate information, the control parameter is determined in accordance with the intermediate information, and thus the spectrum control corresponding to the transmittance is realized.

[0062] Here, the "presence or absence of the liquid 5 in the observation object 1" specifically refers to the presence or absence of the liquid 5 between the distal end of the scope 200 and the observation object 1.

[0063] The "transmittance of the liquid 5" refers to the transmittance of the illumination light in the liquid 5 present between the distal end of the scope 200 and the observation object 1. The transmittance refers to the degree to which the light is transmitted through the liquid 5. The transmittance is, for example, the ratio of the amount of light returned from the observation object 1 to the amount of light emitted from the distal end of the scope 200. Alternatively, if it is considered that the image parameter such as the contrast value is changed due to the reduction in the transmittance, the contrast value or the like of the captured image can also be used as the transmittance. In addition, if it is considered that the transmitted light is determined by the attenuation of the light in the liquid 5, the attenuation rate can also be used as the inverse value of the transmittance. This case is also included in the case of "in accordance with the transmittance of the liquid 5".

[0064] Further, in the present embodiment, the control section 110 performs control to switch a plurality of observation modes including two or more of a first observation mode, a second observation mode, and a third observation mode, the first observation mode being a mode selected when there is no liquid 5, the second observation mode being a mode selected when the transmittance of the liquid 5 is relatively high, and the third observation mode being a mode selected when the transmittance of the liquid 5 is relatively low. The control section 110 performs at least one of the following controls: control to switch the illumination light 7 between first illumination light related to the first observation mode and second illumination light related to the second observation mode in accordance with switching between the first observation mode and the second observation mode; control to switch the illumination light 7 between the second illumination light related to the second observation mode and third illumination light related to the third observation mode in accordance with switching between the second observation mode and the third observation mode; and control to switch the illumination light between the first illumination light related to the first observation mode and the third illumination light related to the third observation mode in accordance with switching between the first observation mode and the third observation mode.

[0065] Thus, by switching the observation mode corresponding to the state of the liquid 5, the illumination light of an appropriate spectrum corresponding to the state of the liquid 5 is switched.

[0066] Here, the first to third observation modes are modes in which the spectrum of the illumination light differs from one another. That is, the first to third illumination lights are different from one another in spectrum. Specifically, the second illumination light is light of a spectrum having a higher transmittance with respect to the liquid 5 than the first illumination light. The third illumination light is light of a spectrum having a higher transmittance with respect to the liquid 5 than the second illumination light. The transmittance becomes higher in which spectrum of light differs depending on the kind of the body fluid or the tissue piece included in the liquid 5. One example thereof will be described hereinafter.

[0067] Further, the illumination light control is not limited to the above. The observation modes are not limited to three, and more observation modes and illumination lights corresponding to the respective observation modes can be provided. For example, by switching the observation modes in a plurality of stages in accordance with the transmittance, the spectrum of the illumination light can be controlled to gradually change in accordance with the transmittance.

[0068] Use Figures 3-7 will be described with respect to an example of switching the above first to third observation modes.

[0069] Figure 3 is an example of switching the observation mode in accordance with the presence or absence of the liquid 5. The control section 110 controls the illumination section 140 to irradiate normal light when there is no liquid 5. The control section 110 controls the illumination section 140 to irradiate turbidity-reducing light by switching the observation mode when it is determined that there is the liquid 5. In this example, the normal light corresponds to the first illumination light in the first observation mode, and the turbidity-reducing light corresponds to the second illumination light or the third illumination light in the second observation mode.

[0070] The second illumination light is illumination light in which the ratio of the relatively long-wavelength component in the spectrum is larger than the ratio of the relatively long-wavelength component in the spectrum of the first illumination light. The third illumination light is illumination light in which the ratio of the relatively long-wavelength component in the spectrum is larger than the ratio of the relatively long-wavelength component in the spectrum of the second illumination light. In the case of a transition from a state in which the liquid 5 is not present to a state in which the liquid 5 is present, when the turbidity of the liquid 5 is relatively small, the first illumination light is switched to the second illumination light, and when the turbidity of the liquid 5 is relatively large, the first illumination light is switched to the third illumination light.

[0071] Thus, in the case of illuminating the observation object 1 via the liquid 5 and imaging, it is possible to irradiate illumination light in which the long-wavelength component is increased. In a state in which the scattering of light is large like a state in which the liquid 5 is turbid due to a tissue piece or the like, the scattering of light in the liquid 5 is small for the long-wavelength component compared to the short-wavelength component. Therefore, by using illumination light in which the long-wavelength component is increased, it is possible to improve the visual recognition of the observation object 1 in the presence of the liquid 5.

[0072] Here, the long-wavelength component is a component of a waveband on the long-wavelength side among wavebands that constitute the illumination light. Specifically, the long-wavelength component is a component that belongs to a waveband on the long-wavelength side from the center among wavebands that constitute the illumination light, or a component that belongs to a color region on the long-wavelength side from the center in the case in which the wavebands that constitute the illumination light are divided into a plurality of color regions. For example, when the illumination light is visible light and the wavebands of the visible light are divided into RGB color regions, the long-wavelength component is a component of the R color region or a component that belongs to the waveband of the R color region.

[0073] Figure 4 is an example in which the observation mode is switched in accordance with the transmittance of the liquid 5. The control section 110 controls the illumination section 140 to irradiate the turbidity-reducing light A when the turbidity of the liquid 5 is small, that is, when the transmittance of the liquid 5 is relatively high. The control section 110 controls the illumination section 140 to irradiate the turbidity-reducing light B by switching the observation mode when it is determined that the turbidity of the liquid 5 has increased and the transmittance has decreased. In this example, the turbidity-reducing light A corresponds to the second illumination light in the second observation mode, and the turbidity-reducing light B corresponds to the third illumination light in the third observation mode.

[0074] In a state in which the scattering of light is large like a state in which the liquid 5 is turbid due to a tissue piece or the like, the more turbid, that is, the lower the transmittance, the greater the scattering. According to the present embodiment, in the case in which the turbidity of the liquid 5 is large and the transmittance is low, by using illumination light in which the long-wavelength component is increased, it is possible to improve the visual recognition of the observation object 1.

[0075] In addition, in the present embodiment, the second illumination light and the third illumination light are illumination light in which the long wavelength component of ordinary light is increased. The degree of increase in the long wavelength component in the third illumination light is greater than the degree of increase in the long wavelength component in the second illumination light.

[0076] Figure 5 An example of control to increase the long wavelength component of ordinary light is shown. Here, the illumination section 140 includes a light source LDV that emits violet light, a light source LDB that emits blue light, a light source LDG that emits green light, and a light source LDR that emits red light. The light amount ratio of the light sources LDV, LDB, LDG, and LDR is set to V:B:G:R, and the light amount ratio is normalized. The control section 110 switches the light amount ratio V:B:G:R in accordance with a control parameter that indicates the light amount ratio V:B:G:R, thereby switching the spectrum of the illumination light. When the light amount ratio of the first illumination light is set to V:B:G:R=v1:b1:g1:r1, the light amount ratio of the second illumination light is set to V:B:G:R=v2:b2:g2:r2, and the light amount ratio of the third illumination light is set to V:B:G:R=v3:b3:g3:r3, r1

[0077] Thus, in underwater observation, the illumination light in which the long wavelength component is adjusted on the basis of the spectrum of ordinary light is used. Thereby, in underwater observation, it is also possible to perform observation in approximately the same color tone as in ordinary observation using ordinary light.

[0078] Here, ordinary observation refers to an observation mode in which a subject is observed in an endoscope system, and corresponds to, for example, an observation mode in which observation is performed using white illumination light when there is no liquid in the subject. In addition, ordinary light refers to illumination light used in the ordinary observation mode, and corresponds to, for example, white illumination light.

[0079] In addition, in the present embodiment, in a case where the control section 110 controls the illumination section 140 so that the ratio of the relative long wavelength component in the spectrum of the illumination light becomes large, the processing section 120 performs image processing that reduces the contribution of the long wavelength component on the captured image. In Figure 3 and Figure 4 In the example of Figs. 17 and 18, the processing section 120 performs this image processing in the second observation mode and the third observation mode.

[0080] If the long wavelength component of the illumination light is increased, the red color of the image is increased, or the color temperature of the image is reduced. The influence of the long wavelength component of the illumination light on the image is the "contribution of the long wavelength component". According to the present embodiment, by performing the image processing that reduces the contribution of the long wavelength component, it is possible to reduce the red color of the image, or it is possible to suppress the reduction in the color temperature of the image. Thereby, it is possible to perform underwater observation in a color tone closer to the color tone when observed with ordinary light.

[0081] Here, the image processing that reduces the contribution of the long-wavelength component is a color conversion process on the captured image. The color conversion process is, for example, a white balance process that increases the color temperature of the image, or a process that reduces the component of the color region of the image that corresponds to the long-wavelength component of the illumination light. The latter process is, for example, a process that sets the R component of the image to 1 / x times when the light amount of the light source LDR is set to x times.

[0082] In addition, in the present embodiment, the processing section 120 performs a structure emphasis process on the captured image so that the contrast of the region of interest is equal to or higher than a prescribed threshold value.

[0083] Thus, even in the case where the contrast of the region of interest is reduced due to the body fluid or the tissue piece included in the liquid 5, the structure emphasis process can emphasize the contrast of the region of interest to be equal to or higher than the prescribed threshold value. Thus, the visual recognition of the region of interest in underwater observation can be improved.

[0084] Here, the region of interest can be the entire field of view of the imaging section 230, or a partial region of the field of view. For example, the processing section 120 can perform lesion detection by image recognition using machine learning or the like, and the detected lesion region can be the region of interest. The structure emphasis process is, for example, a process that increases the high-frequency component of the image, or a contrast emphasis process based on a grayscale conversion, or the like.

[0085] Figure 6 is an example of switching the observation mode according to the blood amount of the liquid 5. The control section 110 controls the illumination section 140 to irradiate the turbidity-reducing light when the blood included in the liquid 5 is small, that is, when the concentration of the blood is relatively low. The control section 110 controls the illumination section 140 to irradiate the bleeding point observation light by switching the observation mode when it is determined that the blood included in the liquid 5 is increased, and the concentration of the blood is increased. In this example, the turbidity-reducing light corresponds to the second illumination light in the second observation mode, and the bleeding point observation light corresponds to the third illumination light in the third observation mode.

[0086] In a surgical operation using an endoscope or the like, bleeding can occur due to incision or resection of a lesion or the like. In water observation, blood of the bleeding is mixed with the liquid 5, and illumination and observation are performed through the liquid 5. Therefore, the degree of transmission of the illumination light decreases due to absorption of hemoglobin, or the image becomes red due to the absorption characteristics of hemoglobin, and thus the visual recognition of the observation object 1 decreases. In addition, in a case where the amount of bleeding is so large that the observation object 1 cannot be visually recognized, it is preferable to secure the field of view by hemostasis. According to the present embodiment, in a state where the blood is so small that the observation object 1 can be visually recognized to some extent, by irradiating the second illumination light which is turbidity-reducing light, the visual recognition can be improved. In addition, in a state where the blood is so large that the observation object 1 cannot be visually recognized, by irradiating the third illumination light which is bleeding point observation light, the visual recognition of the bleeding point can be improved, and the user's hemostasis work can be assisted.

[0087] The second illumination light is illumination light in which a long-wavelength component of ordinary light is increased, and the third illumination light is special light including narrow-band light corresponding to the long-wavelength component. This case is also included in a case where the ratio of the relative long-wavelength component in the spectrum of the third illumination light is larger than the ratio of the relative long-wavelength component in the spectrum of the second illumination light.

[0088] Thus, in a case where the degree of transmission of the liquid 5 is low and the visual recognition of the observation object 1 is very low, special light can be irradiated instead of light obtained by adjusting ordinary light. Thereby, special light having a high degree of transmission with respect to a body fluid or a tissue piece mixed in the liquid 5, or special light with which a specific observation object 1 can be easily observed can be irradiated, and thus the visual recognition of the observation object 1 can be improved.

[0089] Here, the narrow-band light refers to light irradiated from a light source such as a fluorescent lamp, an LED (Light Emitting Diode) lamp, or a laser light source, or a light source having a light filter that passes a part of the wavelength band, and is light in which the light distribution characteristics are discrete. Here, the light in which the light distribution characteristics are discrete also includes a case where a part of the wavelength band is strong and the other wavelength band is significantly weak. In addition, the narrow-band light can be light in a wavelength band narrower than that of white illumination light. Or, when the wavelength band of visible light is divided into RGB, the narrow-band light can be light in a wavelength band narrower than each of the wavelength bands of RGB.

[0090] Specifically, the observation object 1 is a living body, and the liquid 5 includes blood. The third illumination light includes light in a wavelength band of amber. In Figure 6 Here, the bleeding point observation light includes light in a wavelength band of amber. In addition, depending on the use, the second illumination light can include light in a wavelength band of amber, and the second illumination light and the third illumination light can include light in a wavelength band of amber. Hereinafter, the light in the wavelength band of amber will also be referred to as amber light.

[0091] InFigure 7 In the above embodiment, an example of the special light including amber light is shown. Figure 7 The observation based on the special light of the above embodiment is also called RDI (Red Dichromatic Imaging). The illumination section 140 includes, in addition to the light sources LDV, LDB, LDG, and LDR described above, a light source LDA that emits amber light. The control section 110 controls the illumination section 140 so that the light sources LDG, LDA, and LDR emit light.

[0092] The wavelength band of the amber light emitted by the light source LDA exists between the wavelength band of the green light emitted by the light source LDG and the wavelength band of the red light emitted by the light source LDR. Specifically, the wavelength band of the amber light is a wavelength band with a peak wavelength of 600 nm. Note that the peak wavelength of the amber light is not limited to 600 nm. Specifically, in the case where the peak wavelength of the amber light is 600 nm, the wavelength band of the amber light is a wavelength band with a peak wavelength of 600 nm. In the case where the peak wavelength of the amber light is 650 nm, the wavelength band of the amber light is a wavelength band with a peak wavelength of 650 nm. Figure 7 In the above embodiment, an example of the special light including amber light is shown.

[0093] By using the special light including amber light like the RDI described above, it is easy to visually recognize the concentration of blood in the liquid 5, and thus it is easy to visually recognize a bleeding point with a high concentration of blood. In addition, in the above embodiment, the third illumination light is set to the special light, but it is not limited thereto, and the third illumination light can be light to which amber light is added to normal light.

[0094] In addition, in the present embodiment, the processing section 120 can also perform processing of detecting a bleeding point in the observation target 1 from an imaging image captured when the third illumination light including a wavelength band of amber light is irradiated to the observation target 1, and displaying the detected bleeding point.

[0095] Thus, in the case where the amount of bleeding is large and the visual recognition of the observation target 1 is reduced, the visual recognition of the bleeding point is improved by irradiating the third illumination light including a wavelength band of amber light, and the hemostasis work can be assisted by performing detection and display of the bleeding point.

[0096] Further, as described later, the processing section 120 detects the bleeding point by using a detection process using machine learning. Alternatively, the processing section 120 can also detect the bleeding point using color information of the image based on the chroma of red in the vicinity of the bleeding point being higher than the chroma of red in the surroundings, and the like.

[0097] The above has been described taking the case of switching the illumination light in three levels as an example, but the illumination light can also be switched in more levels. Specifically, the control section 110 can also control the illumination section 140 so that the more the body fluid or tissue pieces contained in the liquid 5, the larger the ratio of the relative long-wavelength component in the spectrum of the illumination light.

[0098] In this way, the illumination light of an appropriate spectrum can be irradiated in more levels according to the concentration of the body fluid or tissue pieces contained in the liquid 5, and thus the visual recognition of the observation object 1 can be improved.

[0099] In addition, the endoscope system 10 includes a supply section that supplies the liquid 5 to the observation object 1. The supply section is, for example, the water supply device 150 shown in the drawing. The control section 110 can also control the illumination section 140 so that the more the supply amount of the liquid 5 supplied by the supply section, the larger the ratio of the relative long-wavelength component in the spectrum of the illumination light. Figure 17

[0100] The more the supply amount of the liquid 5 from the supply section, the more the amount of the liquid 5 present in the body cavity, and thus the more the amount of the liquid 5 between the distal end of the scope 200 and the observation object 1, i.e., the distance between the distal end of the scope 200 and the observation object 1 becomes larger. Therefore, the more the supply amount of the liquid 5 from the supply section, the lower the degree of transmission, and the more difficult it is to visually recognize the observation object 1, but according to the present embodiment, by increasing the ratio of the long-wavelength component of the illumination light, the visual recognition of the observation object 1 can be improved.

[0101] In addition, the control section 110 judges the supply amount of the liquid 5 according to the water supply control signal from the operation section 220 or the water supply amount information from the supply section. For example, the water supply control signal is a signal that becomes effective when water supply is turned on, and the control section 110 judges the supply amount based on the effective period of the water supply control signal. Alternatively, the water supply amount information is information indicating the total amount of water supply or the water supply amount per unit time, and the like, and the control section 110 judges the supply amount based on this water supply amount information.

[0102] ​The above describes control of the illumination light corresponding to the transmittance, but in this control, the following control can also be performed. That is, in a case where the transmittance of the liquid 5 is equal to or higher than a prescribed threshold value, the control section 110 can also control the illumination section 140 so that the difference between the image captured by the imaging section 230 and a reference image captured in the absence of the liquid 5 is equal to or lower than a prescribed value. In addition, in a case where the transmittance of the liquid 5 is lower than the prescribed threshold value, the control section 110 can also control the illumination section 140 so that the difference between the image captured by the imaging section 230 and a second reference image different from the reference image is equal to or lower than a prescribed value.

[0103] Thus, in a case where the transmittance of the liquid 5 is relatively high due to the small amount of body fluid or tissue pieces contained in the liquid 5, an image captured by the imaging section 230 close to the reference image captured in the absence of the liquid 5 can be obtained. That is, to some extent of transmittance, observation in water can also be performed with approximately the same color tone as ordinary observation. Also, in a case where the transmittance of the liquid 5 is relatively low due to the large amount of body fluid or tissue pieces contained in the liquid 5, an image captured by the imaging section 230 close to the second reference image different from the reference image can be obtained. That is, a color tone different from ordinary observation is allowed, and the visual recognition of the observation object 1 can be prioritized.

[0104] Here, as an example of the difference between the captured image and the reference image, the difference between the image parameters of the captured image and the reference image. The image parameters are the hue, color balance, brightness, combinations thereof, or statistical values thereof of the image.

[0105] In Figures 3-7 Here, an example of improving the visual recognition by increasing the long-wavelength component of the illumination light is described, but the control of the illumination light for improving the visual recognition is not limited thereto. For example, the visual recognition can also be improved by increasing the light of a wavelength band in which the absorption is small in the spectral characteristics of the body fluid or tissue pieces contained in the liquid 5.

[0106] For example, in a case where the liquid 5 contains blood, light of a wavelength band in which the absorption coefficient of hemoglobin is low can also be used to improve the transmittance. That is, the ratio of the blue component to the violet component in the spectrum of the second illumination light can be larger than the ratio of the blue component to the violet component in the spectrum of the ordinary light. That is, when the light amount ratio of the first illumination light as the ordinary light is set to V:B:G:R=v1:b1:g1:r1, and the light amount ratio of the second illumination light is set to V:B:G:R=v2:b2:g2:r2, b2 / v2>b1 / v1. Note that according to the use, the ratio of the blue component to the violet component in the spectrum of the third illumination light can be larger than the ratio of the blue component to the violet component in the spectrum of the ordinary light, and the ratio of the blue component to the violet component in the spectrum of the second illumination light and the third illumination light can be larger than the ratio of the blue component to the violet component in the spectrum of the ordinary light.

[0107] like Figure 7 As shown, in the light absorption characteristics of hemoglobin, there are absorption peaks near 400 nm and 520 nm, with relatively small absorption coefficients between them. The wavelength of violet light is near 400 nm, while the wavelength of blue light lies between these two absorption peaks. Therefore, blue light is less easily absorbed by hemoglobin compared to violet light. Thus, by irradiating the liquid 5 containing blood with a higher proportion of blue light compared to ordinary light, the transmittance can be increased, thereby improving the visual recognition of the observed object 1.

[0108] 2. Examples of processing using machine learning

[0109] Next, methods for determining control parameters of the illumination light spectrum or detecting bleed points using machine learning processing will be explained. First, the learning device 500 that generates the learned model will be described.

[0110] Figure 8 This is a structural example of a learning device 500. The learning device 500 includes a storage unit 520 and a processing unit 510.

[0111] The storage unit 520 is a semiconductor memory, magnetic storage device, or optical storage device, etc. The storage unit 520 stores training data 522 that maps the input data 523 to the correct answer label 524.

[0112] The processing unit 510 is a processor, FPGA, or ASIC, etc. The processing unit 510 inputs input data 523 to the learning model 511 and obtains the derivation result of the learning model 511. The processing unit 510 includes an update unit 512, which updates the learning model 511 based on the error between the derivation result and the correct solution label 524. Learning is performed by repeating this process. The processing unit 510 causes the storage unit 520 to store the learned model 22 as the learned learning model 511.

[0113] The learned model 22 is transmitted to the endoscope system 10 and stored in the storage unit 130 of the endoscope system 10. The endoscope system 10 uses the learned model 22 stored in the storage unit 130 to determine control parameters, etc.

[0114] When the derivation in the endoscope system 10 is executed using general-purpose hardware such as a processor, the learned model 22 includes a program describing the derivation algorithm and parameters used by the derivation algorithm. The parameters are obtained through learning. The derivation algorithm is, for example, a neural network, in which case the parameters are weight coefficients between nodes in the neural network, etc. The processing unit 120, as general-purpose hardware, executes the program using the parameters, thereby realizing the derivation processing using the learned model 22.

[0115] Alternatively, in a case where the derivation in the endoscope system 10 is performed by dedicated hardware in which the derivation algorithm is hardware-ized, the learned model 22 contains parameters used by the derivation algorithm. The processing section 120 as the dedicated hardware performs the derivation algorithm by using the parameters, thereby realizing the derivation processing using the learned model 22.

[0116] Next, an example of training data and an example of the derivation processing based on the learned model using the training data subjected to machine learning will be described.

[0117] Figure 9 is a first example of the training data. For the learning image TIMi, there are the light amount ratio R:A:G:B:V = ai:bi:ci:di:ei at the time of imaging and the recommended light amount ratio R:A:G:B:V = ai:bi:ci:di:ei. i is an integer of 1 or more and n or less, and n is an integer of 2 or more. The learning image and the light amount ratio at the time of imaging are the input data 523, and the recommended light amount ratio is the correct answer label 524.

[0118] Figure 10 is a structure example of the endoscope system in a case where the learned model 115 learned by the first example of the training data is used. Only the relevant constituent elements are illustrated here. The control section 110 inputs the imaging image IMG from the processing section 120 and the control parameter LINF from the light source control section 111 to the learned model 115. The imaging image IMG input to the learned model 115 can be a color image after development processing or a raw image or the like before development processing. In addition, the imaging image IMG is not limited to data configured as an image. For example, signal information of each pixel can be input to the learned model 115. The control parameter LINF indicates the light amount ratio R:A:G:B:V at the time of imaging the imaging image IMG. The control section 110 acquires the control parameter CPAR by using the derivation processing of the learned model 115. The control section 110 includes the light source control section 111, and the light source control section 111 controls the light amount ratio of each light source of the illumination section 140 using the control parameter CPAR.

[0119] Figure 11 is a second example of the training data. For the learning image TIMi, there are the light amount ratio R:A:G:B:V = ai:bi:ci:di:ei at the time of imaging, the transmittance xi of the liquid, and the blood amount yi contained in the liquid. The learning image and the light amount ratio at the time of imaging are the input data 523, and the transmittance of the liquid and the blood amount contained in the liquid are the correct answer label 524.

[0120] Figure 12is a configuration example of the endoscope system in a case where the learned model 115 learned using the second example of the training data is used. Only the relevant constituent elements are illustrated here. The control section 110 inputs the captured image IMG from the processing section 120 and the control parameter LINF from the light source control section 111 to the learned model 115. The control section 110 acquires information EKINF related to the liquid by using the inference processing of the learned model 115. The information EKINF is the transmittance of the liquid and the amount of blood contained in the liquid. Further, other information related to the liquid can also be adopted as the training data, and the adopted information is obtained by the inference processing. The control section 110 includes the parameter determination section 116 and the light source control section 111. The parameter determination section 116 determines the control parameter CPAR in accordance with the information EKINF. For example, the parameter determination section 116 infers the control parameter based on a learned model obtained by machine learning of the relationship between the information related to the liquid and the control parameter. Alternatively, the parameter determination section 116 can determine the control parameter by referring to a look-up table in which the information related to the liquid and the control parameter are corresponded. The light source control section 111 controls the light amount ratio of each light source of the illumination section 140 using the control parameter CPAR from the parameter determination section 116.

[0121] In the example described above Figures 9-12 , the control section 110 performs processing based on the learned model 115 obtained by machine learning of the relationship between the learning image and the information related to the liquid, or the relationship between the learning image and the information related to the recommended illumination light. The control section 110 determines the control parameter CPAR that controls the spectrum of the illumination light based on the captured image IMG captured by the imaging section 230 and the learned model 115, and controls the illumination section 140 based on the determined control parameter CPAR. Specifically, the learned model 115 is a model obtained by machine learning of the relationship between the learning image, the information of the illumination light used in the imaging of the learning image, and the information related to the liquid. Alternatively, the learned model 115 is a model obtained by machine learning of the relationship between the learning image, the information of the illumination light used in the imaging of the learning image, and the information related to the recommended illumination light.

[0122] In this way, by using the inference processing of machine learning, it is possible to determine the spectrum of the illumination light appropriate for the state of the liquid from the captured image. By using machine learning, it is expected that further optimized illumination light can be irradiated for the states of various liquids.

[0123] Figure 13is the third example of training data. For the learning image TIMi, there is a correspondence with the light amount ratio R:A:G:B:V=ai:bi:ci:di:ei at the time of imaging and the bleeding point information SKi. The learning image and the light amount ratio at the time of imaging are input data 523, and the bleeding point information is the correct answer label 524. The bleeding point information SKi is information indicating the position or area of the bleeding point in the image, and is, for example, a rectangle including the bleeding point, or a line along the boundary of the bleeding point, and the like.

[0124] Figure 14 is a structural example of an endoscope system in a case where the learned model 121 is used that has been learned by the third example of training data. Only the relevant constituent elements are illustrated here. The processing section 120 inputs the imaging image IMG and the control parameter LINF from the control section 110 to the learned model 121. The processing section 120 detects the position or area of the bleeding point by using the inference processing of the learned model 121. The processing section 120 includes the display processing section 123 that causes the detected position or area of the bleeding point to be displayed on the display section 400 together with the imaging image IMG. Further, the control section 110 determines the control parameter CPAR by the method explained in Figures 9-12 .

[0125] Figure 15 is the fourth example of training data. For the learning image TIMi, there is a correspondence with the transmittance xi of the liquid, the amount yi of blood included in the liquid, and the bleeding point information SKi. The learning image is input data 523, and the transmittance, the amount of blood, and the bleeding point information are the correct answer label 524.

[0126] Figure 16 is a structural example of an endoscope system in a case where the learned model 121 is used that has been learned by the fourth example of training data. Only the relevant constituent elements are illustrated here. The processing section 120 inputs the imaging image IMG to the learned model 121. The processing section 120 detects the information EKINF related to the liquid and the position or area of the bleeding point by using the inference processing of the learned model 121. The information EKINF is the transmittance of the liquid and the amount of blood included in the liquid. The processing section 120 includes the display processing section 123 that causes the detected position or area of the bleeding point to be displayed on the display section 400 together with the imaging image IMG. The control section 110 includes the parameter determination section 116 that determines the control parameter CPAR from the information EKINF. The light source control section 111 controls the light amount ratio of each light source of the illumination section 140 using the control parameter CPAR from the parameter determination section 116.

[0127] In the above Figures 13-16In the example of FIG. 12, the processing section 120 performs processing based on a learned model 121 that is obtained by machine learning of a relationship between a learning image and a bleeding point in the observation target. The processing section 120 detects a bleeding point based on the captured image IMG captured by the imaging section 230 and the learned model 121. Specifically, the learned model 121 is a model obtained by machine learning of a relationship between a learning image, information about a liquid, and a bleeding point. Alternatively, the learned model 121 is a model obtained by machine learning of a relationship between a learning image, information about an illumination light used in capturing of the learning image, and a bleeding point.

[0128] Thus, by using the inference processing of machine learning, it is possible to detect a bleeding point from a captured image. By using machine learning, it is expected that further optimized bleeding point detection can be achieved for various illumination lights or states of a liquid.

[0129] Further, the learned model that determines the control parameter and the learned model that detects the bleeding point can be integrated. In this case, in the training data, at least a learning image, a control parameter, and bleeding point information are associated with each other. The learning image is input data, and the control parameter and the bleeding point information are correct answer labels. The control section 110 inputs a captured image to the learned model, and obtains a control parameter and bleeding point information by using inference processing of the learned model. Further, the inference processing can be performed by the processing section 120.

[0130] A part or all of the control section 110 and the processing section 120 of the present embodiment described above can be realized by a program. In this case, the endoscope system 10 can be configured as follows.

[0131] That is, the endoscope system 10 includes a memory that stores information and a processor that operates based on the information stored in the memory. The information is, for example, a program and various data, and the like. The functions of a part or all of the control section 110 and the processing section 120 are described in the program. The processor realizes the functions of a part or all of the control section 110 and the processing section 120 by executing the program.

[0132] Specifically, the endoscope system 10 includes the illumination section 140, the imaging section 230, and the processor. The processor controls the illumination section 140 to irradiate illumination lights having different light spectra according to whether or not a liquid is present in the observation target or according to the transmittance of the liquid. In addition, a part of the present embodiment is mentioned here, but the processor can execute the functions of a part or all of the control section 110 and the processing section 120 described in the present embodiment.

[0133] The processor can include hardware, and the hardware can include at least one of a circuit that processes a digital signal and a circuit that processes an analog signal. For example, the processor can be constituted by one or a plurality of circuit devices mounted on a circuit board, one or a plurality of circuit elements. The one or a plurality of circuit devices are, for example, ICs and the like. The one or a plurality of circuit elements are, for example, resistors, capacitors, and the like. The processor can be, for example, a CPU (Central Processing Unit). However, the processor is not limited to the CPU, and various processors such as a GPU (Graphics Processing Unit) or a DSP (Digital Signal Processor) can be used. In addition, the processor can also be an integrated circuit device such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). In addition, the processor can also include an amplification circuit, a filter circuit, or the like that processes an analog signal. The memory can be a semiconductor memory such as an SRAM or a DRAM, can be a register, can be a magnetic storage device such as a hard disk device, or can be an optical storage device such as an optical disk device. For example, the memory stores commands that can be read by a computer, and by executing the commands by the processor, the functions of each part of the endoscope device are realized as processing. The commands here can be commands of a command set that constitutes a program, or can be commands that instruct the hardware circuit of the processor to act.

[0134] In addition, the above-described program can be stored in an information storage medium that is a medium that can be read by a computer, for example. The information storage medium can be realized by, for example, an optical disk, a memory card, an HDD, or a semiconductor memory. The semiconductor memory is, for example, a ROM. The control section 110 and the processing section 120 perform various processing of the present embodiment in accordance with the program and data stored in the information storage medium.

[0135] In addition, the present embodiment described above can also be executed as an illumination control method. The illumination control method can also be referred to as a working method of the endoscope system 10. Specifically, the illumination control method controls so that illumination light whose irradiation spectrum is variable, an observation object that is irradiated with the illumination light is captured, and illumination light whose irradiation spectrum is different is irradiated in accordance with whether or not a liquid is present in the observation object or in accordance with the transmittance of the liquid. In addition, a part of the present embodiment is mentioned here, but the operation of the endoscope system 10 described in the present embodiment can be executed as an illumination control method.

[0136] 3. Application Example

[0137] Several examples are shown regarding the specific application target of the endoscope system 10 of this embodiment.

[0138] A first application example is a urinary tract scope such as a cystoscope and a resectoscope. The resectoscope is an endoscope for prostate resection. In a procedure using the urinary tract scope, the following first scenario and second scenario are assumed.

[0139] The first scenario is an observation scenario in which the liquid is red turbid with blood. In this scenario, a method of using illumination light of a waveband in which the absorbance of hemoglobin is low, or illumination light in which the amount of light of a wavelength in which the absorbance of hemoglobin is high is increased is used. The following first spectral example and second spectral example are assumed.

[0140] In the first spectral example, based on white illumination light using red light, green light, blue light, and violet light, the amount of light of the blue light is set to be greater than the amount of light of the violet light, and the amount of light of the first green light is set to be greater than the amount of light of the second green light, so that whiteness is maintained and visual recognition is improved. Both the second green light and the first green light belong to the waveband of green, and the second green light has a longer wavelength than the first green light. The peak wavelength of the second green light is around 580 nm. In the absorbance characteristics of hemoglobin, there is an absorbance peak in the waveband of the violet light and the second green light, so the degree of transmission of the illumination light is improved by relatively reducing the amount of light of this waveband. In addition, since the wavelength of the blue light > the wavelength of the violet light, and the wavelength of the first green light < the wavelength of the second green light, the effect of the adjustment of the amount of light on the color tone becomes in the opposite direction, and it is easy to maintain whiteness.

[0141] In the second spectral example, visual recognition is improved by using illumination light in which the ratio of the blue light is increased based on white illumination light. When the visible light region of the illumination light is divided into blue, green, and red, the absorbance of hemoglobin with respect to blue is high, so the blue component of the captured image is reduced. In the second spectral example, the reduction in the blue component in the image is corrected by increasing the ratio of the blue light.

[0142] The second scenario is an observation scenario in which the liquid is whitened due to a relatively large tissue piece. In this scenario, illumination light in which the long wavelength component that is not easily affected by scattering is increased is used. If only red light is used, an image in which the resolution of the surface structure is low is obtained, so blue light and green light are also used in the illumination light. In the spectral example in the second scenario, based on white illumination light using red light, green light, blue light, and violet light, the amount of light of the red light is set to be greater than the amount of light of the violet light and the amount of light of the blue light.

[0143] The second application example is an arthroscope for a surgical operation of a joint. In a treatment using the arthroscope, a viewing scene in which liquid whitens due to small bone chips is envisaged. In this scene, the influence of scattering is greater than in the second scene when the urinary-organ mirror is used, and therefore illumination light that further increases the long-wavelength component is used. That is, in the present scene, the light quantity of red light is increased compared to the second scene when the urinary-organ mirror is used / (light quantity of violet light + light quantity of blue light).

[0144] 4. Second configuration example

[0145] Figure 17 is a second configuration example of the endoscope system 10. The endoscope system 10 includes a control device 100, a scope 200, a display section 400, and a viewing mode switching button 420. The same reference numerals are assigned to the structural elements that have been explained in the first configuration example, and the explanation regarding the structural elements is appropriately omitted.

[0146] The scope 200 includes an emission section 210, an imaging section 230, a water feeding port 250, and a connector 240. The water feeding port 250 is an opening provided at the front end of the scope 200 in order to perform water feeding to the observation object 1. The connector 240 is also provided to the control device 100, and the scope 200 and the control device 100 are connected by the connectors 240 of the scope 200 and the control device 100.

[0147] The control device 100 includes a control section 110, a processing section 120, a storage section 130, and an illumination section 140. The control section 110 includes a light source control section 111, a system control section 112, and an illumination mode switching control section 113. The storage section 130 stores a light source information database 131 and a water information database 132. The illumination section 140 includes light sources LDR, LDA, LDG, LDB, and LDV. In addition, the light source LDA can be omitted in a case where amber light is not used.

[0148] The processing section 120 performs image processing on the imaging signal from the imaging section 230, and generates an imaging image. The system control section 112 causes the display section 400 to display the imaging image from the processing section 120.

[0149] The system control section 112 switches the observation mode in accordance with an operation input to the observation mode switching button 420. The illumination mode switching control section 113 selects a prescribed light source control condition from a setting table included in the light source information database 131 in accordance with the switched observation mode. In addition, the illumination mode switching control section 113 acquires image information based on the captured image from the processing section 120, and selects a light source control condition in accordance with the image information. The image information is, for example, the color, brightness, or a combination thereof, or the like of the image. The system control section 112 outputs the selected light source control condition to the light source control section 111. The light source control section 111 outputs a control parameter that controls the light amount of each light source to the illumination section 140 based on the light source control condition. Each light source of the illumination section 140 emits light in accordance with the light amount indicated by the control parameter. The illumination light emitted from the illumination section 140 is irradiated to the observation object 1 from the emission section 210 of the scope 200.

[0150] The water feeding device 150 feeds water to the observation object 1 from the water feeding port 250 of the scope 200 in accordance with an instruction from the system control section 112. The water feeding device 150 corresponds to the supply section described in the first configuration example.

[0151] The information included in the light source information database 131 will be described. The light source information database 131 has information of the peak wavelength and the relationship information of the drive current-light output for each light source mounted on the endoscope system 10. In addition, the light source information database 131 has output balance information of white light when each light source is combined and total illumination light amount information at that time. The peak wavelength and the wavelength band of each light source, and the example of white light are, for example, as shown in Figure 5 In addition, although not shown, the illumination section 140 can also have a photodiode or the like that monitors the light amount of each light source. The light source control section 111 can also control each light source based on the output of the photodiode and the light source control condition.

[0152] In addition, the light source information database 131 has information of the distribution characteristic of the illumination light emitted from the illumination window, and the brightness of the center derived from the illumination light amount and the distribution characteristic. The illumination window is provided at the front end of the scope 200, and is a window that emits illumination light. Figure 17 The part of the emission lens of the emission section 210 that is exposed at the front end of the scope 200 corresponds to the illumination window. The distribution characteristic can also be maintained for the emission light of each light source. The brightness of the center is also referred to as the center intensity. The distribution characteristic is a characteristic that indicates the relationship between the angle of the center line of the illumination window and the illumination intensity in the direction of the irradiation when the center line is set to be 0 degrees in angle. For example, it is a characteristic in which the intensity is maximum at 0 degrees, and the intensity attenuates as the angle becomes larger.

[0153] Furthermore, the light source information database 131 contains information related to the structure of the illumination window based on the ID information of the mirror body 200 connected to the control device 100. This information includes, for example, the number, arrangement, diameter, or combination of illumination windows located at the front end of the mirror body 200.

[0154] This describes the information contained in the water information database 132. The water information database 132 contains information on the transmittance in the visible light region for water supplied from the water outlet 250. The transmittance information includes, for example, the wavelength-absorption coefficient relationship, the wavelength-scattering coefficient relationship, or both. Additionally, as an optical property related to the material properties of water, the water information database 132 also contains information on the refractive index of water. The refractive index information is based on Fresnel reflection of water. For example, the water information database 132 contains information on the refractive index of water as 1.333.

[0155] In addition, the water information database 132 contains biological information about the subject to be observed. Biological information includes, for example, information about the transmittance of hemoglobin contained in blood. Figure 18 An example of information representing a living organism. Information about the transmittance of hemoglobin could be, for example, the relationship between wavelength and absorption coefficient, wavelength and scattering coefficient, or both.

[0156] Figure 19 , Figure 20 This is a diagram illustrating the actions of the second structural example. Figure 19 Example of setting to represent light balance. Figure 20 This represents an example of setting the brightness correction factor.

[0157] The lighting mode switching control unit 113 extracts the light balance and brightness correction values ​​of the corresponding underwater lighting mode stored in the light source information database 131, based on the underwater observation mode set by the observation mode switching button 420. Depending on the water flow and the condition of the water containing blood, there are multiple settings for the underwater observation mode. Here, the underwater observation modes are set to three modes: underwater 1 to underwater 3, but this is not a limitation. The lighting mode switching control unit 113 changes the driving conditions of each light source based on the extracted correction conditions, thereby changing the color, brightness, or both of the illumination light during underwater observation. Furthermore, as a trigger for switching the underwater lighting mode, the lighting mode switching control unit 113 can also switch between underwater 1 to 3 based on the pressing time of the water flow button.

[0158] exist Figure 19 and Figure 20 In this context, "ordinary" refers to a standard observation mode that is not an underwater observation mode, but rather a mode illuminated by white light. Figure 19In this case, V1 : V2 : V3 : V4 and the like are light amount ratios of the light sources LDV, LDB, LDG, LDR. The color temperature of the illumination light is set in correspondence with each observation mode, and the light amount ratio that realizes the color temperature is set. In Figure 20 In this case, the coefficient is a coefficient that is multiplied by the light amount, and is commonly applied to all light-emitting light sources. That is, the total light-emitting amount of the illumination light is controlled by the coefficient.

[0159] Figures 21-25 is an example of the illumination light in the case of using five light sources to which amber light sources are added. Figure 21 The spectral characteristics of the light emitted by each light source are indicated by LDV, LDB, LDG, LDA, LDR. LDV, LDB, LDA, LDR have peak wavelengths near 410 nm, near 460 nm, near 510 nm, near 600 nm, and near 640 nm, respectively. LDV, LDB, LDA, LDR have a half-value width of about 10 nm, and LDG has a half-value width of about 100 nm. In Figure 21 In this case, the integral values of the spectra of the light sources are normalized to 1.

[0160] In Figures 22-25 In this case, the spectrum of the illumination light as a whole when each light source emits light at the set light amount is shown.

[0161] Figure 22 is a first example of the illumination light. The first example corresponds to white illumination light used in a normal observation mode. That is, this illumination light combines the light sources LDV, LDB, LDA, LDR at a prescribed ratio, and is set to a spectrum close to white light. In the first example, the amber light source LDA is turned off.

[0162] Figure 23 is a second example of the illumination light. The second example is illumination light used in a water observation mode, and is selected when the liquid is whitish. The ratio of the long-wavelength component in the second example is increased compared to the long-wavelength component in the first example. The long-wavelength component here is red light emitted by the light source LDR. In this illumination light, the turbidity is dealt with by increasing the long-wavelength component, and the spectrum close to white light is maintained. In the second example, the amber light source LDA is turned off.

[0163] Figure 24 is a third example of the illumination light. The third example is illumination light used in a water observation mode, and is selected when the liquid is turbid due to blood. In the third example, the ratio of the light-emitting amount of the blue light source LDB to the light-emitting amount of the violet light source LDV is increased compared to the first example. Here, the violet light source LDV is turned off, and the light amount ratio of the blue light source LDB is increased compared to the first example. In this illumination light, the turbidity caused by blood is dealt with by increasing the ratio of the light-emitting amount of the blue light source LDB to the light-emitting amount of the violet light source LDV, and the spectrum close to white light is maintained. In the third example, the amber light source LDA is turned off.

[0164] Figure 25 is the fourth example of the illumination light. The fourth example is an illumination light used in the water observation mode, which is selected when blood spots are easily observed. In the fourth example, the light emission of the amber light source LDA is added to the third example. In addition, the light amount ratio of the red light source LDR is the same as that of the first example of the ordinary light. In this illumination light, by utilizing the amber light having a peak wavelength around 600 nm, the concentration unevenness of blood is emphasized, and blood spots are easily observed.

[0165] According to the above second structural example, a table is provided, which is based on the physical information of water, i.e., absorption, scattering, and refractive index, and the physical information of hemoglobin and blood, i.e., absorption, scattering, and refractive index, which are sent from the water feeding device 150, and which considers the light amount of the illumination light that is absorbed or reflected. This table is a table that corrects the balance and brightness of the illumination light amount. By having such a table, the illumination light suitable for each water observation mode can be set. Thereby, in ordinary observation and water observation, observation can be performed under appropriate illumination light conditions. In addition, since the control of the illumination light is constituted by a plurality of light sources, more detailed color and brightness adjustment can be achieved.

[0166] 5. Third Structural Example

[0167] A third structural example in which AI processing is used in the processing section 120 and the illumination mode switching control section 113 will be described. The structure of the endoscope system 10 is the same as that of the second structural example shown in Figure 17 However, the observation mode switching button 420 can be omitted in the third structural example. The same reference numerals are attached to the structural elements that have been described in the first structural example or the second structural example, and the description of the structural elements will be appropriately omitted.

[0168] In the third structural example, the processing section 120 detects, by AI processing, how much water is contained in the captured image, the degree of turbidity of the water, and the amount of blood contained in the water. Here, the AI processing refers to an inference processing using a learned model obtained by machine learning. Specifically, the processing section 120 identifies the observation state by AI processing based on the captured image and the water information database 132. The observation state is water observation in which the distal end of the scope 200 and the observation object 1 are in the water, or water observation on water in which the distal end of the scope 200 is not in the water but the observation object 1 is in the water. In addition, the processing section 120 identifies the water state in the captured image by AI processing based on the captured image and the water information database 132. The water state is, for example, a boundary between a region having water and a region not having water, a region having water, the concentration of the color of an image in the region having water, or a combination thereof, or the like.

[0169] The illumination mode switching control section 113 corrects the light amount ratio of each light source by AI processing according to the observation state and the water state recognized by the processing section 120. Specifically, the observation state and the water state are machine-learned according to a normal observation image, and a learned model obtained by the machine-learning is stored in the storage section 130 of the endoscope system 10. The illumination mode switching control section 113 optimizes the illumination conditions by inference processing using the learned model to correct the illumination conditions. In the image recognition processing and the AI processing, it is also possible to perform update of the correction light amount, control, in the case where the observation state or the water state within the image changes over time.

[0170] Figure 26 is a detailed configuration example of the processing section 120 and the illumination mode switching control section 113 in the third configuration example. The processing section 120 includes a water detection section 161 and a water state recognition section 162. The illumination mode switching control section 113 includes a light amount correction mode extraction section 163 and a light amount correction value setting section 164. The water state recognition section 162, the light amount correction mode extraction section 163, and the light amount correction value setting section 164 perform AI processing.

[0171] Figure 27 is a processing flowchart in the third configuration example. The description will be made starting from the flow in the normal observation shown in S1. In step S11, the imaging section 230 images the observation object 1, and the processing section 120 generates an imaged image. In step S12, the system control section 112 and the light source control section 111 perform normal light amount control according to the brightness of the imaged image. The normal light amount control is dimming control intended to maintain the brightness of the imaged image constant. In step S13, a water supply on / off signal indicating whether the water supply is on or off is input from the system control section 112 to the water detection section 161. The water detection section 161 judges whether the water supply from the water supply device 150 to the observation object 1 is on or off according to the water supply on / off signal. In the case where the water detection section 161 judges that the water supply is off, steps S11 to S13 are repeated.

[0172] In a case where the water detection section 161 determines that the water supply is on, the flow of water-on or water-in observation shown in S2 is executed. In step S21, the imaging section 230 images the observation target 1, and the processing section 120 generates an image. In step S22, the water detection section 161 determines whether the water supply from the water supply device 150 to the observation target 1 is on or off, based on the water supply on / off signal. In a case where the water detection section 161 determines that the water supply is off, the flow of normal observation shown in S1 is executed. In a case where the water detection section 161 determines that the water supply is on, in step S23, the water state recognition section 162 extracts a water state by AI processing. The water state is a boundary between a region having water and a region not having water, a region having water, a concentration of a color of an image in the region having water, or a combination thereof, or the like. In step S24, based on the water state extracted by the AI processing, a light amount increase / decrease correction value of each light source is extracted based on information possessed by the light source information database 131 and the water information database 132. The light amount increase / decrease correction value is, for example, a correction value for obtaining an image having the same color and brightness as normal light, or a correction value corresponding to a blood volume using illumination light of amber color. In step S24, the light amount correction value setting section 164 outputs light source control information to the system control section 112 based on the light amount increase / decrease correction value, and the system control section 112 and the light source control section 111 execute light amount correction of each light source based on the light source control information.

[0173] According to the above third configuration example, the AI processing is executed based on information of the water region obtained by image recognition, that is, transparency, mixed with blood, or turbidity. By the AI processing, a correction value of the illumination light that takes into account optical characteristic information of water, that is, absorption, scattering, and reflection, is extracted. By changing the illumination light using the correction value, an easily observable image that suppresses the influence of water can be provided. In addition, even if the state of the water changes over time, the AI processing can use the water information based on image recognition and the learning data to set an appropriate color and brightness of the illumination light while setting an illumination mode that is suitable for water-in observation.

[0174] The above describes the present embodiment and its modification examples, but the present application is not directly limited to each embodiment and its modification example, and in the implementation stage, the technical features can be modified and embodied within the scope of the gist. In addition, the plurality of technical features disclosed in each embodiment and the modification example can be appropriately combined. For example, several technical features can be deleted from all the technical features described in each embodiment and the modification example. Furthermore, the technical features described in different embodiments and modification examples can be appropriately combined. In this way, various modifications and applications can be made within the scope of the gist of the present application. In addition, in the specification or the drawings, a term recorded at least once together with a different term of a broader meaning or a synonymous term can be replaced with the different term anywhere in the specification or the drawings.

[0175] EXPLANATION OF REFERENCE NUMERALS

[0176] 1 observation object, 2 body cavity, 5 liquid, 7 illumination light, 10 endoscope system, 22 learned model, 100 control device, 110 control section, 111 light source control section, 112 system control section, 113 illumination mode switching control section, 115 learned model, 116 parameter determination section, 120 processing section, 121 learned model, 123 display processing section, 130 storage section, 131 light source information database, 132 water information database, 140 illumination section, 150 water feeding device, 161 water detection section, 162 water state recognition section, 163 light amount correction mode extraction section, 164 light amount correction value setting section, 200 scope, 210 emission section, 220 operation section, 230 imaging section, 240 connector, 250 water feeding port, 400 display section, 420 observation mode switching button, 500 learning device, 510 processing section, 511 learned model, 512 update section, 520 storage section, 522 training data, 523 input data, 524 correct answer label, CPAR control parameter, IMG imaging image, LDA, LDB, LDG, LDR, LDV light source.

Claims

1. A control method of an endoscope system, characterized by, including: illuminating an observation object with illumination light whose spectrum is variable, capturing an image of the observation object illuminated with the illumination light, generating a captured image, generating a control parameter based on image information of the captured image, controlling the spectrum of the illumination light, and illuminating the observation object with the illumination light, or generating intermediate information based on the image information, and generating a control parameter based on the intermediate information, controlling the spectrum of the illumination light, and illuminating the observation object with the illumination light, or using the image information and a database, finding the control parameter or the intermediate information, and controlling the spectrum of the illumination light with the control parameter, and illuminating the observation object with the illumination light.

2. The control method of an endoscope system according to claim 1, wherein the image information includes the captured image itself or image parameters extracted from the captured image, the control parameter includes a control parameter that controls turning on / off and light quantity of each light source, or a control parameter that switches an optical filter of a filter device.

3. The control method of an endoscope system according to claim 1, wherein the intermediate information includes transmittance of a liquid, a kind of a body fluid contained in the liquid, a concentration, a kind of a tissue piece, a concentration of a tissue piece, and a combination thereof.

4. The control method of an endoscope system according to claim 1, wherein a control parameter of the illumination light is inferred from the image information and / or the intermediate information by processing using a learned model, and the observation object is illuminated with the illumination light according to the determined control parameter.

5. The control method of an endoscope system according to claim 4, wherein the learned model is a model obtained by machine learning of a relationship between a learning image and information related to the liquid, or a relationship between the learning image and information related to the recommended illumination light, or the learned model is a model obtained by machine learning of a relationship between the learning image, information of the illumination light used in photographing of the learning image, and information related to the liquid, or a model obtained by machine learning of a relationship between the learning image, information of the illumination light used in photographing of the learning image, and information related to the recommended illumination light.

6. The control method of an endoscope system according to claim 4, wherein the learned model is a model obtained by machine learning of a relationship between the learning image, information related to the liquid, and the bleeding point, or a model obtained by machine learning of a relationship between the learning image, information of the illumination light used in photographing of the learning image, and the bleeding point.

7. The control method of an endoscope system according to claim 1, wherein the image information includes contrast value, brightness, and color of the captured image, and the database includes spectral characteristics and scattering characteristics of a liquid.

8. The control method of an endoscope system according to claim 1, wherein The database stores a look-up table in which the image information is associated with the control parameter or the intermediate information, and the control parameter or the intermediate information corresponding to the image information is acquired by referring to the look-up table.

9. The control method of the endoscope system according to claim 1, wherein The database stores a look-up table in which the intermediate information is associated with the control parameter, and the control parameter corresponding to the intermediate information is acquired by referring to the look-up table.

10. The control method of the endoscope system according to claim 4, wherein An observation state is recognized by an inference process using the learned model from the captured image and the database, A liquid state is recognized by an inference process using the learned model from the captured image and the database, An illumination condition is corrected by an inference process using the learned model based on the recognized observation state and the liquid state.

11. An endoscope system characterized by comprising: provided with: an illumination section that irradiates illumination light whose spectrum is variable; a camera section that captures an observation target that is irradiated with the illumination light; and a control section that is configured to control the illumination section to switch a plurality of observation modes, the control of the illumination section to switch the plurality of observation modes including: determining whether or not there is a liquid around the observation target; when there is no liquid around the observation target, switching to a first observation mode, and irradiating the observation target with first illumination light; when there is a liquid around the observation target, when the transmittance of the liquid is relatively high, irradiating the observation target with second illumination light, and when the transmittance of the liquid is relatively low, irradiating the observation target with third illumination light.

12. The endoscope system according to claim 11, wherein the second illumination light is illumination light in which the ratio of the relatively long wavelength component in the spectrum is larger than that of the first illumination light, and the third illumination light is illumination light in which the ratio of the relatively long wavelength component in the spectrum is larger than that of the second illumination light.

13. The endoscope system according to claim 12, wherein at least one of the second illumination light and the third illumination light includes light in a waveband of amber color, the waveband of amber color exists between 576 nm and 730 nm.

14. The endoscope system according to claim 11, wherein the endoscope system further includes: a processing section that performs image processing on a captured signal from the camera section to generate a captured image; a supply section that supplies a liquid to the observation target; and a storage section that stores a setting table indicating a light amount balance and a brightness correction value corresponding to each observation mode, the control section switches the observation mode according to an operation input by a user, selects a prescribed light source control condition corresponding to each of the observation modes from the setting table, and controls the illumination section to irradiate the observation target with corresponding illumination light according to the selected light source control condition. ​

Images