Endoscope device and control method thereof, information storage medium, and processing device
By combining the light source, imaging and processing unit in the endoscope device, the lighting characteristics are dynamically adjusted, and the problem of insufficient display accuracy of AI auxiliary information in the prior art is solved, and more efficient diagnosis and treatment assistance is achieved.
Patent Information
- Application Number
- CN201980092951.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-02-26
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2039-02-26
AI Technical Summary
In diagnostic assistance, existing endoscopic devices are difficult to display AI auxiliary information with high accuracy based on image information, and fail to optimize the lighting light, affecting the diagnostic assistance effect.
By combining the light source part, the imaging part and the processing part in the endoscope device, the illumination characteristics, including the spectroscopic characteristics, the light amount and the irradiation timing, to generate auxiliary information suitable for diagnosis or treatment.
It improves the accuracy of diagnostic auxiliary information, provides AI auxiliary information that is more suitable for observation purposes, and enhances the diagnostic and handling auxiliary capabilities of the endoscopy.
Smart Images

Figure CN113473899B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an endoscope device and a control method thereof, an information storage medium, a processing device, etc. Background Art
[0002] In the field of medical endoscopes, the application of AI technology has been studied. For example, observation auxiliary information that is helpful for the discovery or diagnosis of lesions is extracted by AI technology, and the observation auxiliary information is displayed on the endoscope monitor. For example, such a technology is disclosed in Patent Document 1. In Patent Document 1, when the diagnosis auxiliary mode is set to on in the fluorescence observation mode, the state of fluorescence emission in the reference area and the area of interest is visualized, the state is monitored and displayed, and diagnosis assistance is performed through the time-dependent changes in the monitored state.
[0003] In addition, it is known that a technology is used to control a light source so as to be able to take images that meet the purpose in an endoscope device. For example, such technology is disclosed in Patent Documents 2 and 3. In Patent Document 2, the light distribution angle of the illumination light is changed by using a liquid crystal lens, thereby reducing the error in obtaining biological function information such as oxygen saturation. In Patent Document 3, a spectral transmittance distribution is provided near the pupil of the imaging optical system, and the spectral transmittance distribution is a distribution in which the area of the pupil becomes larger relative to the wavelength band that is desired to be emphasized in the return light from the subject. As a result, the desired deep tissue information can be observed with good contrast.
[0004] Prior art literature
[0005] Patent Literature
[0006] Patent Document 1: International Publication No. 2016 / 084504
[0007] Patent Document 2: Japanese Patent Application Publication No. 2006-181387
[0008] Patent Document 3: Japanese Patent Application Publication No. 2003-215469 Summary of the invention
[0009] Problems to be solved by the invention
[0010] In the endoscope device that performs the above-mentioned diagnosis assistance, there is a problem of displaying AI auxiliary information with high accuracy or appropriately based on image information. For example, the observation purposes of the endoscope are divided into multiple types, such as diagnosis of the presence or qualitative diagnosis of lesions, range diagnosis, treatment assistance, etc. Preferably, by controlling the appropriate lighting based on the image information, AI auxiliary information that is more suitable for the observation purpose of the endoscope can be generated. In the prior art such as Patent Document 1, monitoring and displaying auxiliary information is disclosed, but optimizing the auxiliary information based on image information is not disclosed.
[0011] According to some aspects of the present invention, it is possible to provide an endoscope apparatus, a program, and the like that can improve the accuracy of auxiliary diagnosis information or present appropriate auxiliary diagnosis information by optimizing illumination light based on image information.
[0012] Means for solving problems
[0013] One embodiment of the present invention relates to an endoscope device, which includes: a light source unit that is capable of changing illumination characteristics, wherein the illumination characteristics include at least one of the spectral characteristics and light amount, light distribution characteristics, and irradiation timing of the emitted illumination light; an imaging unit that captures a subject illuminated by the illumination light and outputs an image signal; and a processing unit that sets the illumination characteristics based on auxiliary information that assists diagnosis or treatment, and when the illumination characteristics are first illumination characteristics, the processing unit changes the illumination characteristics to second illumination characteristics based on the first auxiliary information generated based on the image signal.
[0014] In addition, another embodiment of the present invention relates to a program that causes a computer to execute the following steps: obtaining first auxiliary information when the lighting characteristic is a first lighting characteristic, wherein the lighting characteristic includes at least one of the spectral characteristics and light amount, light distribution characteristics, and irradiation timing of the illumination light, the first auxiliary information is generated based on an image signal obtained by photographing a subject illuminated by the illumination light, and the first auxiliary information assists in diagnosis or treatment; and changing the lighting characteristic to a second lighting characteristic based on the first auxiliary information. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 This is a structural example of an endoscope device.
[0016] Figure 2 1 is a diagram showing the light emission timing and the imaging timing in the diagnosis mode.
[0017] Figure 3 This is a diagram for explaining the process of generating a display image.
[0018] Figure 4 It is a diagram showing the light emission timing and the imaging timing in the NBI mode.
[0019] Figure 5 It is a diagram showing the light emission timing and the imaging timing in the pseudo dyeing mode.
[0020] Figure 6 This is a flowchart showing the procedure of processing performed by the processing unit when switching from the existence diagnosis mode to the qualitative diagnosis mode.
[0021] Figure 7 This is a diagram for explaining a method of determining whether to shift from the existence diagnosis mode to the qualitative diagnosis mode.
[0022] Figure 8 This is an example of automatic classification in qualitative diagnosis mode.
[0023] Fig. 9 It is a diagram showing the light emission timing and the imaging timing in the UC inflammation diagnosis mode.
[0024] Fig.10 This is an example of display of auxiliary information in the UC inflammation diagnosis mode.
[0025] Fig.11 : is a diagram showing the light emission timing and the imaging timing in the range diagnosis mode.
[0026] Fig.12 This is an example of a display image in range diagnosis mode.
[0027] Fig.13 It is a diagram showing the light emission timing and the imaging timing in the bleeding point recognition mode.
[0028] Fig.14 This is an example of a display image in the bleeding point identification mode.
[0029] Fig.15 This is a structural example of an endoscope system. DETAILED DESCRIPTION
[0030] The present embodiment is described below. It should be noted that the present embodiment described below does not unduly limit the contents of the present invention as described in the claims. In addition, all the structures described in the present embodiment are not necessarily necessary technical features of the present invention. For example, an endoscope for the digestive tract is described in this specification, but the application objects of the method disclosed in this specification are not limited to this. That is, the endoscope device mentioned in this specification refers to a device having an insertion portion for observing the inner surface of a recessed portion of various observation objects. For example, the endoscope device includes a surgical endoscope used for surgical operations such as the abdominal cavity or joints.
[0031] 1. Endoscopic device
[0032] Figure 1 This is a structural example of the endoscope device 10 in this embodiment. The endoscope device 10 includes a control device 100, a mirror body 200, a display unit 300, and an operating unit 600. It should be noted that the control device 100 is also referred to as a main body. In addition, the mirror body 200 is also referred to as a mirror body, a camera unit, and a camera device. In addition, the display unit 300 is also referred to as a display and a display device. In addition, the operating unit 600 is also referred to as an operating panel and an operating device.
[0033] First, the configuration of the endoscope device 10 will be described. Then, a method of optimizing illumination light according to the purpose of observation will be described.
[0034] The mirror body 200 is composed of an insertion portion 210 and a connector 240. The insertion portion 210 is flexible and can be inserted into a body cavity of a living body. The body cavity of a living body is a subject in this embodiment. The connector 240 is provided at one end of the mirror body 200 so that the control device 100 and the mirror body 200 can be attached and detached.
[0035] The front end of the insertion part 210 is provided with: an illumination lens 211 that emits illumination light toward the subject; an imaging unit 213 that captures an image by receiving illumination light reflected or scattered from the surface of the subject; and a distance sensor 215 that measures the distance from the front end of the insertion part 210 to the subject. In addition, the insertion part 210 includes a light guide 214 that guides the illumination light emitted from the light source unit 140 to the illumination lens 211.
[0036] The illumination lens 211 expands the illumination light guided by the light guide 214 into a desired radiation angle. Figure 1 In the embodiment, one illumination lens is provided at the front end of the insertion portion 210, but a plurality of illumination lenses may be provided at the front end of the insertion portion 210. The light guide 214 is, for example, an optical fiber bundle. The imaging portion 213 has an imaging optical system and an imaging element. The imaging element is, for example, a primary color type or a complementary color type CMOS imager. The distance measuring sensor 215 is, for example, a distance measuring sensor of a TOF (Time Of Flight) method, or an ultrasonic distance measuring sensor. It should be noted that the distance may be measured by making the imaging portion 213 a stereoscopic optical system instead of a distance measuring sensor. When a distance measuring sensor of a TOF method is used, light with a long wavelength such as infrared light is sometimes used.
[0037] It should also be noted that various functions or mechanisms that are omitted from the figure can be mounted in the insertion portion 210. For example, in the insertion portion 210, a scope operating portion for operating the scope portion 200, or a bending mechanism for bending the front end portion, or a forceps hole for inserting forceps, or a treatment instrument such as an electric scalpel for treatment, or an air and water supply pipe for ejecting and sucking liquids and gases, etc. can be mounted. In addition, a part of the structural elements included in the insertion portion 210 can also be omitted. For example, in an embodiment where the ranging function is not used, the ranging sensor 215 can also be omitted.
[0038] The control device 100 controls each unit of the endoscope device 10 or performs image processing on an image captured by the scope unit 200. The control device 100 includes a light source unit 140, a processing unit 110, and a storage unit 160.
[0039] The light source unit 140 generates illumination light and causes the illumination light to enter the light guide 214. The light source unit 140 can generate illumination light having various spectral characteristics. Specifically, the light source unit 140 includes: a light source emitting red light; a light source LDA emitting amber light; a light source LDG emitting green light; a light source LDB emitting blue light; a light source LDV emitting violet light; lenses LN1 to LN6; dichroic mirrors DC1 to DC4; optical sensors SS1 to SS5; a light source driving unit 141; and a filter unit 142. In addition, the structure of the light source unit 140 is not limited to Figure 1 For example, the optical sensor may be omitted. Alternatively, a light combining unit using optical fibers may be provided instead of the dichroic mirrors DC1 to DC4.
[0040] The light sources LDR, LDA, LDG, LDB, and LDV are, for example, LEDs (Light Emitting Diodes) or semiconductor lasers. The peak wavelength of the red light emitted by the light source LDR is 635 nm. The peak wavelength of the amber light emitted by the light source LDA is 600 nm. The peak wavelength of the green light emitted by the light source LDG is 532 nm. The peak wavelength of the blue light emitted by the light source LDB is 445 nm. The peak wavelength of the violet light emitted by the light source LDV is 405 nm. These lights are, for example, narrow-band lights with a half-value width of tens of nm. It should also be noted that the light emitted by each light source is not limited to this. That is, red light only needs to have a peak wavelength in the wavelength range of 615nm to 680nm, amber light only needs to have a peak wavelength in the wavelength range of 586nm to 615nm, green light only needs to have a peak wavelength in the wavelength range of 495nm to 585nm, blue light only needs to have a peak wavelength in the wavelength range of 440nm to 495nm, and violet light only needs to have a peak wavelength in the wavelength range of 400nm to 440nm. In addition, each light may be broadband light.
[0041] The lens LN1 allows the light emitted from the light source LDR to enter the dichroic mirror DC1. Similarly, the lenses LN2, LN3, LN4, and LN5 allow the light emitted from the light sources LDA, LDG, LDB, and LDV to enter the dichroic mirrors DC1, DC2, DC3, and DC4.
[0042] The dichroic mirror DC1 allows light emitted from the light source LDR to pass through, and reflects light emitted from the light source LDA. Similarly, the dichroic mirrors DC2, DC3, and DC4 allow light from the dichroic mirrors DC1, DC2, and DC3 to pass through, and reflect light emitted from the light sources LDG, LDB, and LDV.
[0043] The filter unit 142 is configured to be able to switch filters and pass-through parts. A plurality of filters may also be provided. The pass-through part allows the light from the dichroic mirror DC4 to pass directly without changing the spectral characteristics. The filter has a spectral transmittance characteristic, and allows the wavelength region corresponding to the spectral transmittance characteristic in the light from the dichroic mirror DC4 to pass. The light sources LDR, LDA, LDG, LDB, and LDV are configured so that any one or more of them can emit light. By selecting the light source to be emitted from these five light sources and selecting the filter of the filter unit 142, illumination light having a desired spectral characteristic is realized.
[0044] The lens LN6 allows the light that has passed through the filter unit 142 to enter the light guide 214 .
[0045] The light source driving unit 141 is controlled by the light source control unit 151, and drives the light sources LDR, LDA, LDG, LDB, and LDV based on the control. The light source control unit 151 will be described later. For example, when the light sources LDR, LDA, LDG, LDB, and LDV are LEDs or semiconductor lasers, the light source driving unit 141 outputs a driving current to each light source to make each light source emit light.
[0046] The optical sensors SS1, SS2, SS3, SS4, and SS5 are respectively arranged at positions capable of detecting leakage light from the light sources LDR, LDA, LDG, LDB, and LDV. The leakage light is light emitted to a portion that is not on the optical path incident on the lenses LN1 to LN5.
[0047] The processing unit 110 includes an image generating unit 111, an auxiliary information generating unit 121, and a light source control unit 151. It is conceivable that various hardwares realize the processing unit 110. For example, the image generating unit 111, the auxiliary information generating unit 121, and the light source control unit 151 may be composed of one processor or ASIC (Application Specific Integrated Circuit), or may be composed of separate processors or ASICs. Alternatively, two structural elements such as the image generating unit 111 and the auxiliary information generating unit 121 may be composed of one processor or ASIC.
[0048] The image generation unit 111 generates a display image based on the image signal from the camera unit 213. Specifically, the light source unit 140 emits display illumination light for capturing the display image and auxiliary illumination light for generating auxiliary information in a time-division manner. The image generation unit 111 generates a display image based on the image signal captured when the display illumination light is irradiated on the subject. The image generation unit 111 performs image processing such as interpolation processing of the image signal to generate an RGB image, white balance processing, or grayscale conversion processing. In addition, the image generation unit 111 superimposes the display content based on the auxiliary information generated by the auxiliary information generation unit 121 on the display image, and outputs the display image to the display unit 300.
[0049] The display unit 300 is, for example, a display device such as a liquid crystal display. The display unit 300 displays the display image output by the image generation unit 111. Thus, the image captured by the display illumination light and the auxiliary information superimposed thereon are presented to the user of the endoscope device 10.
[0050] The auxiliary information generating unit 121 generates auxiliary information based on the image signal captured when the auxiliary illumination light is irradiated on the subject. The auxiliary information is information used to assist the physician using the endoscope device 10 in diagnosis or treatment. The auxiliary information generating unit 121 performs reasoning processing based on AI (Artificial Intelligence). That is, the auxiliary information generating unit 121 extracts auxiliary information from the input image signal through reasoning processing. As AI, various image recognition methods or machine learning methods can be used. Machine learning is a process of performing various reasoning based on learning results. A representative AI is a neural network, but it is not limited to this. As the AI in this embodiment, various known machine learning methods can be used.
[0051] The storage unit 160 is, for example, a storage device such as a semiconductor memory or a hard disk drive. The storage unit 160 stores information about the learned model. The auxiliary information generation unit 121 performs the above-mentioned AI-based reasoning processing based on the learned model. Various hardware can be conceived as the hardware of the auxiliary information generation unit 121 that performs reasoning processing. For example, the auxiliary information generation unit 121 is a general-purpose processor such as a CPU. In this case, the storage unit 160 stores a program that describes an reasoning algorithm and parameters for the reasoning algorithm as information about the learned model. Alternatively, the auxiliary information generation unit 121 may also be a dedicated processor that hardwareizes the reasoning algorithm. In this case, the storage unit 160 stores the parameters for the reasoning algorithm as information about the learned model.
[0052] The inference algorithm is a neural network. The neural network has: an input layer that inputs data; an intermediate layer that performs calculations on the data input through the input layer; and an output layer that outputs data based on the calculation results output from the intermediate layer. The nodes included in the input layer are connected to the nodes included in the intermediate layer, and the nodes included in the intermediate layer are connected to the nodes included in the output layer. The weight coefficient of the connection between the nodes is equivalent to the above-mentioned parameter.
[0053] The light source control unit 151 controls the spectral characteristics of the illumination light. That is, the light source control unit 151 makes each light source emit or extinguish, or controls the light emission of each light source by outputting a control signal to the light source driving unit 141. In addition, the light source control unit 151 controls the spectral characteristics of the illumination light by controlling the filter unit 142. In addition, the light source control unit 151 controls the light emission timing of each light source according to a prescribed light emission timing. In addition, the light source control unit 151 performs feedback control based on the output signals of the light sensors SS1 to SS5 so that the light emission of the light sources LDR, LDA, LDG, LDB, and LDV becomes the desired value.
[0054] The light source control unit 151 causes the light source unit 140 to emit display illumination light and auxiliary illumination light in a time-sharing manner through the above-mentioned control. The endoscope device 10 has an existence diagnosis mode, a qualitative diagnosis mode, and a treatment assistance mode. The details of each mode will be described later. In the existence diagnosis mode, the qualitative diagnosis mode, and the treatment assistance mode, the spectral characteristics of the auxiliary illumination light are different. That is, the light source control unit 151 controls the light source unit 140 to generate auxiliary illumination light with spectral characteristics corresponding to the set mode. The light source control unit 151 includes a mode determination unit 152, which determines the switching of the mode based on the auxiliary information generated by the auxiliary information generation unit 121. In addition, in Figure 1 In the embodiment, the mode determination unit 152 is included in the light source control unit 151, but the mode determination unit 152 may be provided outside the light source control unit 151. Alternatively, the determination mode may be switched based on information input from the operation unit 600. The operation unit 600 is a device for a user such as a physician to operate the endoscope device 10. For example, the operation unit 600 is a button or a dial, a foot switch, a touch panel, or the like.
[0055] The judgment mode is a mode corresponding to the judgment criterion when the auxiliary information generating unit 121 generates auxiliary information. That is, the auxiliary information generating unit 121 of the first judgment mode generates auxiliary information according to the first judgment criterion, and the auxiliary information generating unit 121 of the second judgment mode different from the first judgment mode generates auxiliary information according to the second judgment criterion different from the first judgment criterion. The judgment criterion of each judgment mode can be any criterion corresponding to the observation purpose of the judgment mode. That is, the auxiliary information required to be prompted is different according to the observation purpose, but it is sufficient to use the judgment criterion of extracting the auxiliary information corresponding to the observation purpose from the image information. For example, the processing unit 110 can also infer the judgment mode based on the image information and set the inferred judgment mode. By setting the judgment mode, the judgment criterion when generating auxiliary information is set. The following is an example of a case where the first judgment mode is an existence diagnosis mode and the second judgment mode is a qualitative diagnosis mode. However, the existence diagnosis mode or the qualitative diagnosis is an example of a judgment mode set according to the doctor's observation mode during endoscopic observation, but it can also be other predetermined judgment modes. In the following, although the learned models corresponding to the respective judgment modes are used, the present invention is not limited thereto, and any processing may be performed as long as the judgment criteria differ according to the judgment mode.
[0056] Auxiliary information is generated by a learned model corresponding to each judgment mode. That is, the storage unit 160 stores information of a learned model corresponding to an existence diagnosis mode, information of a learned model corresponding to a qualitative diagnosis mode, and information of a learned model corresponding to a treatment assistance mode. Then, the auxiliary information generation unit 121 generates auxiliary information from the image signal through inference processing, wherein the inference processing is based on processing of the learned model corresponding to the set judgment mode. The learned model of each judgment mode is learned, for example, in the following manner. The following description will be made using the existence diagnosis mode as an example, but the qualitative diagnosis mode and the treatment assistance mode are also learned in the same way.
[0057] Teacher data is an image captured using auxiliary illumination light in the presence diagnosis mode and annotation information added to the image by experts such as doctors. Annotation information is information that is intended to be displayed as auxiliary information, and in the presence diagnosis mode, for example, information indicating the location or outline of the lesion. A plurality of combinations of images and annotation information are prepared, and they become teacher data. The images in the teacher data are input into the inference algorithm, and the inference algorithm infers the auxiliary information based on the images. Then, the parameters of the inference algorithm are fed back so that the auxiliary information is close to the annotation information. Learning is performed by repeatedly performing the above feedback using a plurality of combinations of images and annotation information.
[0058] Various environments for performing the learning process can be envisioned. For example, the learning process may be performed by the processing unit 110 of the endoscope device 10, and the information of the learned model generated thereby may be written to the storage unit 160. Alternatively, the learning process may be performed by an information processing device such as a PC (Personal Computer), and the information of the learned model generated thereby may be written to the storage unit 160 of the endoscope device 10.
[0059] In addition, in this specification, "lesion" refers to a part detected by the auxiliary information generating unit 121 as having a possible lesion, and the doctor determines whether it is actually a lesion. That is, the "lesion" detected by the auxiliary information generating unit 121 is a lesion candidate. In this specification, the lesion candidate is simply referred to as a lesion or a lesion part.
[0060] 2. First Implementation
[0061] The following describes a method for optimizing illumination light according to the purpose of observation. In the first embodiment, the presence diagnosis mode is automatically switched to the qualitative diagnosis mode based on auxiliary information. As described above, the display illumination light and the auxiliary illumination light are emitted in a time-sharing manner, but in each of the presence diagnosis mode and the qualitative diagnosis mode, the auxiliary illumination light is set to the optimal spectral characteristics.
[0062] First, the light emission timing and the image capture timing in each mode are described. In addition, the content of the auxiliary information and the display method of the auxiliary information are described.
[0063] Figure 2 1 is a diagram showing the light emission timing and the shooting timing in the diagnosis mode. In the driving timing of the imaging element, the reading period Trd for reading the image signal and the exposure period TE for shooting the subject are repeated. When the display image is displayed at a frame rate of 60 Hz, the repetition rate of the period Trd and TE is 120 Hz.
[0064] The light source control unit 151 causes the light source unit 140 to emit illumination light in the periods TH and TAI corresponding to the exposure period TE. In the period TH, the light source control unit 151 causes the light source unit 140 to emit white light W as display illumination light. White light is display illumination light. In the period TAI, the light source control unit 151 causes the light source unit 140 to emit amber light A and purple light V as auxiliary illumination light. The light source control unit 151 causes the light source unit 140 to emit white light, amber light A, and purple light V alternately.
[0065] Purple light is suitable for obtaining the characteristics of the surface blood vessels or glandular duct structure of the mucosa. Amber light is suitable for obtaining the characteristics of the deep blood vessels or redness, inflammation, etc. of the mucosa. That is, the auxiliary information generation unit 121 detects lesions that can be detected based on the characteristics of the surface blood vessels or glandular duct structure of the mucosa, or lesions that can be detected based on the characteristics of the deep blood vessels or redness, inflammation, etc. of the mucosa as auxiliary information. In the presence diagnosis mode, by using purple light and amber light, the presence of a wide range of lesions such as cancer and inflammatory diseases can be detected.
[0066] Figure 3 is a diagram for explaining the generation process of the display image. The image generation unit 111 generates a display image according to the Figure 2 The image signal captured during the period TH generates a normal image. The normal image is a white light image.
[0067] In addition, the image generation unit 111 Figure 2 The auxiliary image is generated by the image signal captured by TAI during the period. The auxiliary image is an image obtained by capturing the subject illuminated by amber light and purple light. The auxiliary information generating unit 121 generates auxiliary information based on the auxiliary image. Figure 3 In the figure, the shaded area in the lower left corner of the auxiliary image is a lesion, and its outline is detected as auxiliary information. Figure 3 In FIG, the outline of the lesion is indicated by a dotted line. In addition, the image generating unit 111 may directly input the image signal captured during the period TAI to the auxiliary information generating unit 121 without generating the auxiliary image.
[0068] The image generation unit 111 uses the normal image as the display image, and further superimposes a display indicating the outline of the lesion on the display image. The display image to which the auxiliary information is added is displayed on the display unit 300. A physician or the like using the endoscope device 10 can recognize the position or outline of the lesion candidate by observing the display image. The physician or the like understands how the lesion looks in the normal image through experience or learning. Therefore, by setting the normal image as the display image and adding the auxiliary information to it, the physician can easily observe the lesion and make a diagnosis.
[0069] Next, the timing of light emission in the qualitative diagnosis mode is described. As the qualitative diagnosis mode, a plurality of diagnosis modes can be set. In the first embodiment, an NBI (Narrow Band Imaging) mode and a pseudo staining mode are set.
[0070] Figure 4 is a diagram showing the lighting timing and the shooting timing in the NBI mode. Figure 2 The description of the same contents is omitted.
[0071] In the NBI mode, during the period TAI, the light source control unit 151 causes the light source unit 140 to emit the purple light V and the green light G. The combination of the purple light V and the green light G is used for NBI. However, the light intensity ratio of the purple light V and the green light G only needs to be an appropriate light intensity ratio for the AI-based reasoning process, and does not need to be the light intensity ratio in the normal NBI.
[0072] Auxiliary information in qualitative diagnosis modes such as NBI mode is qualitative auxiliary information related to the lesions detected in the presence diagnosis mode. Qualitative auxiliary information can be various information used in the diagnosis of lesions, such as the progress of the lesion, the degree of symptoms, the range of the lesion, or the boundary between the lesion and the normal part. For example, the learned model can also be made to learn the classification that follows the classification benchmark established by the society, and the classification result based on the learned model can be used as auxiliary information.
[0073] In the NBI mode, the auxiliary information generation unit 121 generates qualitative auxiliary information by processing based on the learned model corresponding to the NBI mode. The qualitative auxiliary information in the NBI mode is a classification result classified according to various NBI classification criteria. As NBI classification criteria, for example, there are VS classification as a classification criteria for gastric lesions, or JNET, NICE classification, EC classification, etc. as classification criteria for large intestine lesions.
[0074] Figure 5 is a diagram showing the light emission timing and the shooting timing in the simulated dyeing mode. Figure 2 The same contents are omitted.
[0075] In the simulated dyeing mode, the processing unit 110 uses the distance measuring sensor 215 to measure the distance of the object during TAI. Specifically, the processing unit 110 obtains the concave and convex information of the surface of the object by distance measurement. The concave and convex information is, for example, a depth map. The auxiliary information generation unit 121 generates auxiliary information based on the distance measurement result. The auxiliary information is information indicating simulated dyeing, for example, information indicating the depth of dyeing in each pixel. In dyeing performed by drug spreading, the concave parts on the surface of the object are dyed more densely. The auxiliary information generation unit 121 generates auxiliary information of simulated dyeing, wherein the simulated dyeing is a dyeing that reproduces dyeing performed by drug spreading.
[0076] The image generation unit 111 generates a display image by coloring a normal image captured using display illumination light based on the auxiliary information. That is, the image generation unit 111 adds color to each pixel according to the shade of each pixel indicated by the auxiliary information. For example, in the case of simulated coloring of indigo carmine, simulated coloring is performed using a blue color that simulates indigo carmine.
[0077] The auxiliary information generation unit 121 generates qualitative auxiliary information based on the display image that has been stained based on the auxiliary information. The learned model corresponding to the simulated staining pattern has learned the image after the staining process and the annotation information added to the image by experts such as doctors as teacher data. The auxiliary information generation unit 121 uses the learned model to generate auxiliary information based on the simulated staining image. When annotating, various classification criteria using staining based on pigment dispersion can be used. The auxiliary information generated by the auxiliary information generation unit 121 is a classification result along the classification criteria used in learning.
[0078] Next, a method of automatically switching from the existence diagnosis mode to the qualitative diagnosis mode based on the auxiliary information will be described. Figure 6 1 is a flowchart showing the procedure of processing performed by the processing unit 110 when switching from the existence diagnosis mode to the qualitative diagnosis mode.
[0079] In step S1 , the mode determination unit 152 sets the existence diagnosis mode. The light source control unit 151 causes the light source unit 140 to emit white light, amber light A, and purple light V in a time-division manner. The auxiliary information generation unit 121 generates auxiliary information by processing based on a learned model corresponding to the existence diagnosis mode.
[0080] Next, in step S2, the mode determination unit 152 determines whether the lesion indicated by the auxiliary information is larger than a predetermined area in the region of interest. If the lesion is larger than the predetermined area, in step S3, the mode determination unit 152 sets the NBI mode in the qualitative diagnosis mode. If the lesion is not larger than the predetermined area, the process returns to step S1.
[0081] Figure 7 This is a diagram for explaining a method for determining whether to shift from the existence diagnosis mode to the qualitative diagnosis mode. IMG is an image captured in the existence diagnosis mode, and the shadows in the image schematically represent a tubular object. The upper right corner of IMG is the deep part of the lumen, and the closer to the wall and the imaging unit, the lower left it is. It is assumed that the auxiliary information generation unit 121 detects a lesion BYH in the lower part of IMG, that is, on the wall of the lumen. The solid ellipse is the auxiliary information generated by the auxiliary information generation unit 121, which is the outline of the lesion BYH here.
[0082] ATA is an area of interest. For example, a doctor operating the endoscope device 10 can also set the area of interest using the operation unit 600. The area of interest ATA can be set to any position and area in the image IMG. Figure 7, the area of interest ATA is set in the lower area of the image IMG. The mode determination unit 152 finds the area of the lesion BYH in the area of interest ATA, and determines whether the area is greater than a specified value. For example, the mode determination unit 152 counts the number of pixels of the portion where the area of interest ATA overlaps with the lesion BYH. The mode determination unit 152 determines whether the number of pixels is greater than a specified value. Alternatively, the mode determination unit 152 finds the ratio of the counted number of pixels to the number of pixels in the area of interest ATA, and determines whether the ratio is greater than a specified value. When the mode determination unit 152 determines that the area of the lesion BYH is greater than a specified value, the mode determination unit 152 shifts from the existence diagnosis mode to the NBI mode of the qualitative diagnosis mode. For example, the physician operating the endoscope device 10 may also use the operation unit 600 to set the specified value for area determination.
[0083] In the NBI mode of step S3, the light source control unit 151 causes the light source unit 140 to emit white light W, purple light V, and green light G in a time-division manner. The auxiliary information generation unit 121 generates auxiliary information by processing based on a learned model corresponding to the NBI mode. The auxiliary information generation unit 121 generates qualitative auxiliary information related to the lesion detected in the presence diagnosis mode.
[0084] Next, in step S4, the mode determination unit 152 further determines whether a detailed examination is required based on the qualitative auxiliary information generated by the auxiliary information unit 121. If it is determined that a detailed examination is not required, the process returns to step S2. If it is determined that a detailed examination is required, in step S5, the mode determination unit 152 sets the simulated staining mode in the qualitative diagnosis mode.
[0085] In the simulated staining mode, illumination and distance measurement based on white light W are performed in a time-sharing manner. The auxiliary information generation unit 121 generates auxiliary information by processing based on the learned model corresponding to the simulated staining mode. The image generation unit 111 performs staining processing on the display image based on the auxiliary information. The auxiliary information generation unit 121 generates qualitative auxiliary information based on the display image that has been stained based on the auxiliary information. That is, the classification result of the lesion along the classification criterion using staining is generated as auxiliary information.
[0086] Next, in step S6, it is determined whether the lesion detected in step S5 is larger than a predetermined area in the region of interest. The determination method is the same as step S2. If the lesion is larger than the predetermined area, the process returns to step S5. If the lesion is not larger than the predetermined area, the process returns to step S1.
[0087] Figure 8 This figure shows an example of automatic classification in the qualitative diagnosis mode. Figure 8 This process shows the process of automatic classification using the JNET classification standard. Figure 6Steps S3 to S5 are performed.
[0088] In the learning phase, the doctor labels the image classification results based on JNET. That is, the doctor uses a pointing device to surround the lesion part contained in the image, thereby marking the position or outline of the lesion and the classification of the lesion. Then, the label information and the image are used as teacher data to generate a learned model. When using the endoscope device for diagnosis, the following automatic classification is achieved by performing inference processing using the above-mentioned learned model. That is, through inference processing, the position or outline of the lesion and the classification of the lesion are generated as auxiliary information.
[0089] like Figure 8 As shown, in the NBI mode, the auxiliary information generation unit 121 classifies the lesions detected in the presence diagnosis mode into Type 1, Type 2A, Type 2B, and Type 3. These types are classifications characterized by the vascular pattern of the mucosa and the surface structure of the mucosa. The auxiliary information generation unit 121 outputs the probability that the lesion is Type 1, the probability that the lesion is Type 2A, the probability that the lesion is Type 2B, and the probability that the lesion is Type 3. It should also be noted that not only the final classification result, but also the classification result of this stage can be displayed on the display image through text, etc.
[0090] The auxiliary information generating unit 121 determines whether or not it is difficult to distinguish a lesion based on the classification result in the NBI mode.
[0091] That is, when the probability of being type 1 and type 2A is the same, the auxiliary information generation unit 121 determines that it is difficult to distinguish. In this case, the mode determination unit 152 sets a simulated staining mode that simulates the reproduction of indigo carmine staining. In this simulated staining mode, the auxiliary information generation unit 121 classifies the lesion as a hyperplastic polyp or a low-atypia mucosal tumor based on the simulated staining image. These classifications are classifications characterized by the pit pattern in the indigo carmine staining image. On the other hand, when the probability of being type 1 is above the threshold, the auxiliary information generation unit 121 classifies the lesion as a hyperplastic polyp, and the mode determination unit 152 does not transfer to the simulated staining mode. In addition, when the probability of being type 2A is above the threshold, the auxiliary information generation unit 121 classifies the lesion as a hyperplastic polyp, and the mode determination unit 152 does not transfer to the simulated staining mode.
[0092] When the probability of being type 2A and type 2B is the same, the auxiliary information generation unit 121 determines that it is difficult to distinguish. In this case, the mode determination unit 152 sets a simulated staining mode that simulates the reproduction of crystal violet staining. In this simulated staining mode, the auxiliary information generation unit 121 classifies the lesion into a low-atypia mucosal tumor or a high-atypia mucosal tumor or a submucosal mild invasive cancer based on the simulated staining image. These classifications are classifications characterized by the pockmark image in the crystal violet staining image.
[0093] Hereinafter, although detailed description is omitted, in the case where it is judged to be type 2B, a simulated staining mode for simulating the reproduction of crystal violet staining is set, and the lesion is classified as a high-atypia intramucosal tumor or a submucosal mild invasive cancer. In the case where it is difficult to distinguish Type 2B and Type 3, a simulated staining mode for simulating the reproduction of crystal violet staining is set, and the lesion is classified as a high-atypia intramucosal tumor or a submucosal mild invasive cancer or a submucosal deep invasive cancer. In the case where it is judged to be type 2B, the lesion is classified as a submucosal deep invasive cancer, and is not transferred to the simulated staining mode.
[0094] The image generating unit 111 adds the classification result and the position or outline of the detected lesion to the display image as auxiliary information, and causes the display unit 300 to display the display image.
[0095] In addition, in the first embodiment, the spectral characteristics of the auxiliary illumination light are changed according to the mode, but it is not limited to this. The light amount, light distribution characteristics, and irradiation timing of the auxiliary illumination light may also be changed according to the mode. For example, the light amount may be set in the diagnosis assistance mode so that the lesion detected in the existence diagnosis mode is detected with appropriate brightness. Alternatively, the light distribution characteristics may be set in the diagnosis assistance mode so that the lesion detected in the existence diagnosis mode is appropriately illuminated. For example, a plurality of illumination lenses are provided at the front end of the mirror body 200, and the light distribution can be changed by controlling the amount of light emitted from each illumination lens. An example of changing the irradiation timing is, for example, in the second embodiment. Fig. 9 In the following. Fig. 9 In the embodiment, auxiliary illumination light is emitted at two timings of periods TAI1 and TAI2.
[0096] The endoscope device 10 of the embodiment described above includes a light source unit 140, an imaging unit 213, and a processing unit 110. The light source unit 140 is capable of changing the illumination characteristics including the spectral characteristics of the emitted illumination light and at least one of the light quantity, the light distribution characteristics, and the irradiation timing. The imaging unit 213 captures the subject illuminated by the illumination light and outputs an image signal. The processing unit 110 sets the illumination characteristics based on auxiliary information that assists diagnosis or treatment. When the illumination characteristics are the first illumination characteristics, the processing unit 110 changes the illumination characteristics to the second illumination characteristics based on the first auxiliary information generated based on the image signal.
[0097] Thus, the illumination light can be optimized to obtain the best illumination characteristics for the reasoning process of the AI, so that the auxiliary information can be estimated with high accuracy, and the high-accuracy auxiliary information can be presented to the doctor or the like. Specifically, it is assumed that an AI that outputs auxiliary information necessary for the purpose of observation is used. In this case, the light source unit 140 can emit the best illumination light for the reasoning process of the AI. Thus, the auxiliary information corresponding to the purpose of observation can be estimated with high accuracy.
[0098] It should be noted that, in the first embodiment, the first lighting characteristic is amber light A and purple light V in the presence diagnosis mode, the second lighting characteristic is purple light V and green light G in the NBI mode, the first auxiliary information is the position or outline of the lesion detected in the presence diagnosis mode, and the second auxiliary information is the classification result of the lesion in the NBI mode.
[0099] In addition, in the present embodiment, the processing unit 110 generates auxiliary information based on the image signal. That is, when the illumination characteristic is the first illumination characteristic, the processing unit 110 generates the first auxiliary information based on the image signal, and when the illumination characteristic is the second illumination characteristic, the processing unit 110 generates the second auxiliary information based on the image signal, the content of the diagnosis or treatment being different from that of the first auxiliary information.
[0100] However, if Fig.15 As shown, the auxiliary information generating unit that generates the auxiliary information may be provided outside the endoscope device 10 . Fig.15 The endoscope system includes an endoscope device 10 and an information processing system 400. Fig.15 In the embodiment, the processing unit 110 of the endoscope device 10 does not include the auxiliary information generating unit 121 , and the endoscope device 10 further includes the interface 170 . The information processing system 400 includes an interface 470 , a storage unit 460 , and a processing unit 410 . The processing unit 410 includes the auxiliary information generating unit 421 .
[0101] The auxiliary information generating unit 421 and the storage unit 460 correspond to Figure 1The auxiliary information generating unit 121 and the storage unit 160 are connected to the auxiliary information generating unit 421. That is, the image signal is input from the camera unit 213 to the auxiliary information generating unit 421 via the interface 170 and the interface 470. The auxiliary information generating unit 421 generates auxiliary information based on the image signal by performing inference processing based on the learned model stored in the storage unit 460. The auxiliary information is input to the mode determining unit 152 via the interface 470 and the interface 170. The mode determining unit 152 switches the determination mode based on the auxiliary information, and the information of the determination mode is input to the auxiliary information generating unit 421 via the interface 170 and the interface 470.
[0102] Various communication interfaces can be envisioned as interfaces 170 and 470. For example, interfaces 170 and 470 may be interfaces for network connection based on LAN or WAN, etc. Alternatively, interfaces 170 and 470 may also be interfaces for wired communication such as USB, or interfaces for wireless communication such as short-range wireless communication. Information processing system 400 may also be, for example, an information processing device such as a PC or a server. Alternatively, information processing system 400 may also be a cloud system in which multiple information processing devices connected via a network perform information processing. In this case, the functions of processing unit 410 are realized by the information processing performed by multiple information processing devices included in the cloud system.
[0103] In addition, in the present embodiment, when the judgment mode in the auxiliary information generation process is the first judgment mode, the processing unit 110 sets the first lighting characteristic and generates the first auxiliary information. The processing unit 110 switches the judgment mode to the second judgment mode based on the first auxiliary information. The processing unit 110 sets the second lighting characteristic in the second judgment mode and generates the second auxiliary information.
[0104] Thus, based on the first auxiliary information, it is possible to automatically switch from the first judgment mode to the second judgment mode, and then emit illumination light with illumination characteristics corresponding to each judgment mode. In this way, it is possible to emit illumination light with the best illumination characteristics for each judgment mode, generate high-precision or best auxiliary information in each judgment mode, and present the auxiliary information to doctors and the like.
[0105] In addition, in this embodiment, the processing unit 110 determines whether the first auxiliary information satisfies a predetermined condition. When the processing unit 110 determines that the first auxiliary information satisfies the predetermined condition, the processing unit 110 switches from the first determination mode to the second determination mode.
[0106] Thus, when the first auxiliary information satisfies a predetermined condition in the first determination mode, it is possible to automatically shift from the first determination mode to the second determination mode.
[0107] In this embodiment, the first judgment mode is an existence diagnosis mode for diagnosing the existence of a lesion candidate included in the subject. The first auxiliary information is information indicating the existence of the lesion candidate. The second judgment mode is a qualitative diagnosis mode for diagnosing the state of the lesion candidate. The second auxiliary information is information indicating the state of the lesion candidate.
[0108] Thus, in the presence diagnosis mode, it is possible to automatically switch from the presence diagnosis mode to the qualitative diagnosis mode based on the first auxiliary information. By automatically switching from the presence diagnosis mode to the qualitative diagnosis mode, it is possible not only to automatically prompt the physician, etc., of the presence of the lesion, but also to automatically prompt the physician, etc., of the qualitative information of the lesion. Thus, it is possible to automatically monitor and display the qualitative auxiliary information that is useful when the physician diagnoses the lesion.
[0109] In addition, in the present embodiment, the processing unit 110 determines whether the first auxiliary information satisfies a prescribed condition, wherein the prescribed condition refers to that the area of the lesion candidate included in the first auxiliary information is greater than a prescribed value, or the observation magnification of the imaging unit is greater than a prescribed value. When the processing unit 110 determines that the first auxiliary information satisfies the prescribed condition, it switches from the first determination mode to the second determination mode.
[0110] When the doctor focuses on the lesion and brings the scope 200 close to the lesion, it can be assumed that the area of the candidate lesion becomes larger than the specified value. Alternatively, when the doctor wants to observe the lesion, the lesion is observed by magnification. Therefore, when the area of the candidate lesion is larger than the specified value, or when the observation magnification of the imaging unit is larger than the specified value, the diagnosis auxiliary mode is switched to, so that qualitative auxiliary information related to the lesion that the doctor is concerned about can be provided to the doctor.
[0111] In addition, in the present embodiment, the lesion candidate is a tumor candidate.
[0112] According to this embodiment, a tumor candidate is detected in the presence diagnosis mode, and when the tumor candidate satisfies the above-mentioned specified conditions, it is possible to switch to the qualitative diagnosis mode. Thus, for example, qualitative auxiliary information of the tumor candidate, such as the type of tumor, can be presented to the doctor. In addition, the lesion is not limited to the tumor, as long as it is an abnormal area in the organism. For example, the lesion can be an inflammation or bleeding area.
[0113] In the first determination mode, the illumination light of the first illumination characteristic includes at least one light of a plurality of colors, that is, a first group of lights. In the second determination mode, the illumination light of the second illumination characteristic includes at least one light of a plurality of colors, a second group of lights different from the first group.
[0114] In this way, the spectral characteristics of the illumination light in the first judgment mode can be made different from the spectral characteristics of the illumination light in the second judgment mode. As a result, the illumination light with the best spectral characteristics can be used in each judgment mode to generate auxiliary information. It should also be noted that the group also includes the case of light of one color. In addition, the light of the first group and the light of the second group only need to be partially different, and there can be overlapping parts. For example, in Figure 2 and Figure 4 In the example of , the first group is A, V, and the second group is V, G. In this example, V is repeated, but the color combination is different.
[0115] In addition, in the present embodiment, the plurality of colors of light include purple light, amber light, and green light.
[0116] In this way, the information of the blood vessels in the surface layer of the mucosa can be obtained by purple light or green light, and auxiliary information can be extracted from the information. However, purple light can obtain information of shallower blood vessels than green light. In addition, the information of the blood vessels in the deep layer of the mucosa or the information of the hemoglobin concentration in the blood clot can be obtained by amber light, and auxiliary information can be extracted from this information.
[0117] In addition, in this embodiment, the judgment mode is an existence diagnosis mode for diagnosing the existence of a lesion candidate contained in the subject, or a qualitative diagnosis mode for diagnosing the state of the lesion candidate. Alternatively, the judgment mode may also be a treatment assistance mode for assisting treatment. The treatment assistance mode will be described in the embodiment described later.
[0118] The illumination light in the qualitative diagnosis mode includes light for observing inflammation or middle-layer blood vessels and deep-layer blood vessels, which are not included in the illumination light in the existence diagnosis mode. The illumination light includes light with a longer wavelength than the illumination light in the existence diagnosis mode. Specifically, the illumination light in the existence diagnosis mode includes purple light and amber light. The illumination light in the qualitative diagnosis mode includes purple light and green light. Alternatively, as described in the embodiment described later, the illumination light in the qualitative diagnosis mode may also include green light, amber light, red light, or illumination light for distance measurement.
[0119] In this way, by using purple light or green light, the blood vessels in the surface layer of the mucosa can be photographed with good contrast. In addition, by using amber light, the blood vessels in the deep part of the mucosa, or the thickness of the blood in the bleeding area, etc. can be photographed with good contrast. In the presence diagnosis mode, by using purple light and amber light, various types of lesion candidates can be extracted. In addition, by combining purple light and green light in the qualitative diagnosis mode, auxiliary information along the NBI diagnosis can be obtained. In addition, by combining green light, amber light and red light in the qualitative diagnosis mode, auxiliary information related to inflammatory diseases such as ulcerative colitis can be obtained based on the information of the surface blood vessels and deep blood vessels. In addition, by using the illumination light for ranging in the qualitative diagnosis mode, auxiliary information related to lesions such as the state of the mucosa or cancer can be obtained based on the information of the shape or structure of the mucosa.
[0120] In addition, as described in the embodiment described later, the illumination light in the treatment assistance mode may also include light used to extract the lesion range or extract the bleeding point in the generation process of the auxiliary information. Specifically, the illumination light in the treatment assistance mode may also include illumination light for distance measurement and purple light and amber light as narrow-band light. Alternatively, the illumination light in the treatment assistance mode may also include red light and amber light as narrow-band light.
[0121] By combining purple light, amber light, and distance-finding illumination light in the treatment assistance mode, the lesion range can be extracted as auxiliary information from the information of the surface blood vessels and deep blood vessels and the shape or structure of the mucosa. Thus, the lesion range can be indicated to doctors, etc., and treatment such as lesion removal can be assisted. In addition, by combining red light and amber light in the treatment assistance mode, information on bleeding points in blood clots, etc. can be obtained based on the concentration information of hemoglobin.
[0122] In addition, in the present embodiment, the endoscope device 10 includes a display unit 300 for displaying a display image. The light source unit 140 irradiates display illumination light for generating a display image and auxiliary illumination light having illumination characteristics different from those of the display illumination light in a time-division manner. The processing unit 110 generates a display image based on an image signal when the display illumination light is emitted. The processing unit 110 generates auxiliary information based on the image signal when the auxiliary illumination light is emitted. Then, the processing unit 110 performs image processing to add display contents based on the auxiliary information to the display image.
[0123] The auxiliary illumination light is optimized to be optimal for AI-based reasoning processing, and therefore may not be suitable for observation. In the present embodiment, the display image is captured using display illumination light that is different from the auxiliary illumination light, so that the display image suitable for observation can be presented to the doctor. In addition, according to the present embodiment, based on the auxiliary information generated by the auxiliary information generating unit 121, the display content corresponding to the auxiliary information is displayed on the display unit 300. Since the illumination characteristics are optimized according to the purpose of observation, the high-precision auxiliary information generated based on the image captured using the illumination light can be presented to the doctor to assist in diagnosis or treatment.
[0124] In the present embodiment, the display content based on the auxiliary information is at least one of the position and the outline of the target site indicated by the auxiliary information.
[0125] Thus, at least one of the position and the contour of the target site detected by the auxiliary information generating unit 121 can be displayed on the display unit 300. For example, in the presence diagnosis mode, by displaying at least one of the position and the contour on the display unit 300, the presence of a lesion candidate can be presented to the doctor.
[0126] In addition, in the present embodiment, when the illumination characteristic of the auxiliary illumination light is the first illumination characteristic, the processing unit 110 generates the first auxiliary information, and performs image processing to add the first display content to the display image based on the first auxiliary information. The processing unit 110 changes the illumination characteristic of the auxiliary illumination light to the second illumination characteristic based on the first auxiliary information, and generates the second auxiliary information. The processing unit 110 performs image processing to add the second display content different from the first display content to the display image based on the second auxiliary information.
[0127] Thus, high-precision auxiliary information obtained by using illumination light with optimized illumination characteristics can be displayed on a display image captured by using display illumination light. In addition, auxiliary information with different contents can be obtained according to each illumination characteristic, and the auxiliary information can be displayed on the display image. Thus, various auxiliary information corresponding to the purpose of observation can be superimposed on a display image suitable for observation and presented to the doctor.
[0128] In the present embodiment, the display illumination light is white light, and the auxiliary illumination light includes purple light and at least one of amber light and green light.
[0129] In this way, a white light image can be displayed as a display image on the display unit 300, and auxiliary information can be added to the white light image and prompted to the doctor. In addition, by making the auxiliary illumination light include purple light and at least one of amber light and green light, it is possible to achieve illumination characteristics corresponding to the purpose of observation. That is, by using purple light or green light, the blood vessels on the surface of the mucosa can be photographed with good contrast. In addition, by using amber light, the blood vessels deep in the mucosa, or the thickness of the blood in the bleeding area, etc. can be photographed with good contrast. Alternatively, it is possible to consider combining these lights according to the purpose of observation. For example, in the case of wanting to obtain auxiliary information along the NBI diagnosis, it is sufficient to combine purple light and green light.
[0130] In the present embodiment, the endoscope device 10 includes a storage unit 160 that stores information of a learned model. The learned model is a model that is learned to output auxiliary information for an image signal. The processing unit 110 generates auxiliary information from the image signal by processing based on the learned model.
[0131] Thus, auxiliary information can be generated from image signals through AI reasoning processing. By learning the learned model through teacher data generated by experts such as doctors, a learned model that reflects the expert's insights can be generated. By using such a learned model, auxiliary information based on the expert's insights can be provided to the user of the endoscope device 10.
[0132] As described above, the learned model can include a neural network. As a neural network, various well-known AI (Artificial Intelligence) technologies can be used. In order to use a neural network, software development for executing learning and reasoning algorithms is required, but currently, multiple commercially available and free software packages are available and can also be used. In addition, as a machine learning algorithm in a neural network, various well-known learning algorithms can be used, for example, a teacher learning algorithm using an error back propagation method can be used.
[0133] In addition, in the present embodiment, the storage unit 160 stores information of the first learned model and the second learned model. The first learned model is a model that is learned to output the first auxiliary information for the image signal when the illumination characteristic is the first illumination characteristic. The second learned model is a model that is learned to output the second auxiliary information for the image signal when the illumination characteristic is the second illumination characteristic. When the illumination characteristic is the first illumination characteristic, the processing unit 110 generates the first auxiliary information by processing based on the first learned model. When the illumination characteristic is the second illumination characteristic, the processing unit 110 generates the second auxiliary information by processing based on the second learned model.
[0134] According to this embodiment, by preparing a learned model corresponding to each lighting characteristic, auxiliary information can be generated using the learned model corresponding to the observation purpose. That is, not only the lighting characteristics but also the AI is optimized according to the observation purpose. Thus, it is possible to present the doctor with high-precision auxiliary information corresponding to the observation purpose.
[0135] The functions and actions of the endoscope device 10 described above can also be realized by a program. That is, the functions and actions of the endoscope device 10 can also be realized by storing a program describing the functions and actions of the endoscope device 10 in a memory and executing the program by a computer. A computer is a device having an input device, a processing unit, a storage unit, and an output unit, such as an information processing device such as a PC. The program causes the computer to execute the following steps: generating first auxiliary information based on an image signal captured when the illumination characteristic is the first illumination characteristic, changing the illumination characteristic to the second illumination characteristic based on the first auxiliary information, and generating second auxiliary information based on an image signal captured when the illumination characteristic is the second illumination characteristic.
[0136] In addition, the program of the present embodiment may also be stored in an information storage medium. The information storage medium is a medium that can be read by a computer. As the information storage medium, various storage media such as optical disks such as DVDs or CDs, hard disks, non-volatile memories, or semiconductor memories such as ROMs can be envisioned. The computer performs various processes in the present embodiment based on the program and data stored in the information storage medium. That is, the information storage medium stores a program for causing the computer to function as the endoscope device 10 of the present embodiment.
[0137] 3. Second Implementation
[0138] In the second embodiment, the qualitative diagnosis mode is the UC inflammation diagnosis mode. UC is ulcerative colitis. The method of transitioning from the presence diagnosis mode to the qualitative diagnosis mode is the same as that of the first embodiment, so the UC inflammation diagnosis mode will be described below.
[0139] Fig. 9 is a diagram showing the timing of light emission and the timing of imaging in the UC inflammation diagnosis mode. Figure 2 The same contents are omitted.
[0140] The light source control unit 151 causes the light source unit 140 to emit white light in the period TE. In the UC inflammation diagnosis mode, the auxiliary illumination light is red light R, green light G, and amber light A. The light source control unit 151 causes the light source unit 140 to emit red light R and green light G in the period TAI1, and causes the light source unit 140 to emit amber light A in the period TAI2. By repeating the periods TE, TAI1, and TAI2, the above-mentioned illumination light is emitted in a time-sharing manner. It should also be noted that the red light R and the amber light A both belong to the red region, for example, they pass through a red filter in a primary color Bayer type imager. Therefore, the red light R and the amber light A are emitted in a time-sharing manner. The emission order of the red light R and the amber light A can also be reversed.
[0141] The combination of red light R, green light G, and amber light A is an illumination light suitable for photographing blood vessels in the deep layer of the mucosa. The qualitative auxiliary information in the UC inflammation diagnosis mode is a diagnosis result or classification result based on the visibility of deep blood vessels. It should be noted that visibility does not actually mean that people can see it, for example, it refers to the contrast of deep blood vessels in the image. For example, the auxiliary information generation unit 121 generates the inflammation degree after the contrast of deep blood vessels is digitized as auxiliary information.
[0142] Fig.10 This is an example of display of auxiliary information in the UC inflammation diagnosis mode. The image generation unit 111 superimposes the information SJH of the inflammation degree generated by the auxiliary information generation unit 121 on the normal image IMG2 captured using white light. For example, when the numerical value of the inflammation degree is 50, the image generation unit 111 displays the text "Inflammation degree 50" at the bottom of the normal image IMG2. The numerical value of the inflammation degree changes according to the numerical value determined by the auxiliary information generation unit 121. Doctors and the like can refer to the displayed normal image and the numerical value of the inflammation degree to diagnose the inflammation degree of UC.
[0143] 4. Third Implementation
[0144] In the third embodiment, a treatment assistance mode can be set. The treatment assistance mode in the third embodiment is a range diagnosis mode for estimating the range of resection of a lesion. For example, the range diagnosis mode is selected based on input from the operation unit 600. Alternatively, the presence diagnosis mode may be switched to the range diagnosis mode in the same manner as the automatic switch from the presence diagnosis mode to the qualitative diagnosis mode in the first embodiment. Alternatively, the qualitative diagnosis mode may be switched to the range diagnosis mode. For example, Figure 6 If it is determined in step S6 that the lesion area is greater than the specified value, the mode is shifted to the range diagnosis mode.
[0145] Fig.11 is a diagram showing the lighting timing and the shooting timing in the range diagnosis mode. Figure 2The same contents are omitted.
[0146] During period TAI, the light source control unit 151 causes the light source unit 140 to emit amber light A and purple light V. During period TAI, the processing unit 110 measures the distance to the subject using the distance measuring sensor 215. That is, the processing unit 110 generates a depth map or the like indicating uneven information on the surface of the subject.
[0147] The amber light A and the purple light V as auxiliary illumination light are the same as those in the presence diagnosis mode. The auxiliary information generation unit 121 detects the position or contour of a lesion using the same learned model as in the presence diagnosis mode.
[0148] Fig.12 This is an example of a display image in the range diagnosis mode. The image generation unit 111 displays the position or contour of the lesion BYH detected by the auxiliary information generation unit 121 on the display image IMG3.
[0149] In addition, the auxiliary information generating unit 121 estimates the resection range SJH of the lesion based on the concavo-convex information of the object surface obtained by ranging and the position or contour of the lesion BYH detected above. In this estimation, a learned model that has learned the resection range is used. That is, the learned model uses the image of the lesion and the annotation information of the resection range annotated by the doctor or other experts on the image as teacher data for learning. The image generating unit 111 displays the resection range SJH estimated by the auxiliary information generating unit 121 in the display image IMG3. As described above, the position or contour of the lesion BYH and the resection range SJH are displayed as auxiliary information in the display image IMG3.
[0150] A physician or the like using the endoscope device 10 can determine the resection range SJH by referring to the display image IMG3 showing the lesion BYH and the resection range SJH. As a method of lesion resection using an endoscope, for example, endoscopic submucosal dissection can be envisioned. In this method, a mark is made on the periphery of the resection range, but when making the mark, the resection range SJH shown on the display image IMG3 can be referred to.
[0151] 5. Fourth embodiment
[0152] In the fourth embodiment, the treatment assistance mode is a bleeding point recognition mode for automatically recognizing bleeding points. The bleeding point recognition mode is used during hemostasis treatment. For example, the bleeding point recognition mode is selected based on the input from the operating unit 600. Alternatively, it is also possible to automatically transfer from the existence diagnosis mode to the bleeding point recognition mode. For example, the learned model used in the existence diagnosis mode is made to pre-learn an image of a bleeding area such as a blood clot. When the submucosal layer is peeled off in the above-mentioned ESD, a blood clot is generated in the concave part after peeling. Such an image can also be used as teacher data to perform learning. Thus, when a blood clot is generated in an image in the ESD, it is also possible to automatically transfer from the existence diagnosis mode to the bleeding point recognition mode.
[0153] Fig.13 FIG. 1 is a diagram showing the light emission timing and the shooting timing in the bleeding point recognition mode. Figure 2 The same contents are omitted.
[0154] In the period TAI1 , the light source control section 151 causes the light source section 140 to emit the red light R, and in the period TAI2 , the light source control section 151 causes the light source section 140 to emit the amber light A. The order of emitting the red light R and the amber light A may be reversed.
[0155] Fig.14 This is an example of a display image in the bleeding point identification mode. In a normal image, which is a white light image, a blood clot appears red or dark red. The shaded portion indicates a blood clot. A bleeding point exists at the bottom of a blood clot. In a state where blood and water are mixed due to water supply from the air and water supply pipes, the hemoglobin concentration becomes higher near the bleeding point, but since the contrast between the hemoglobin concentration and the light concentration is difficult to see in white light, it is difficult to visually identify the bleeding point in a normal image.
[0156] In the auxiliary image captured using red light R and amber light A, the blood clot appears orange, etc. In the wavelength region of red light R, hemoglobin is hardly absorbed, and amber light A around 600nm is absorbed by hemoglobin to some extent. Therefore, the contrast of hemoglobin density is easy to appear in the auxiliary image. The hemoglobin concentration in the blood flow from the bleeding point becomes higher, and the orange color of this part becomes darker than the surrounding area. Fig.14 In the figure, the area where the orange color becomes darker is indicated by SKT. The bleeding point can be estimated from the SKT in this area.
[0157] The learned model in the bleeding point identification mode is made to learn using the image of the bleeding area photographed by the red light R and the amber light A and the annotation information of the bleeding point annotated by the doctor or other experts on the image as the teacher data. The auxiliary information generation unit 121 detects the position of the bleeding point based on the auxiliary image obtained by photographing the blood clot and the above-mentioned area SKT by using the above-mentioned learned model. The position of the bleeding point is the auxiliary information in the bleeding point identification mode. The image generation unit 111 displays the position of the detected bleeding point in the display image using, for example, an arrow JYS. The doctor or the like using the endoscope device 10 can determine the bleeding point by referring to the arrow JYS or the like displayed in the display image.
[0158] The above describes the embodiments and variations of the present invention, but the present invention is not limited to the embodiments and variations thereof, and the constituent elements can be deformed and concretized in the implementation stage without departing from the scope of the purpose of the invention. In addition, various inventions can be formed by appropriately combining multiple technical features disclosed in the above-mentioned embodiments or variations. For example, several technical features can be deleted from all the technical features described in the embodiments and variations. In addition, the technical features described in different embodiments and variations can also be appropriately combined. In this way, various modifications and applications can be achieved without departing from the scope of the main purpose of the invention. In addition, in the specification or the drawings, a term described at least once together with a different term in a broader sense or synonymous can be replaced with the different term at any position in the specification or the drawings.
[0159] Description of symbols
[0160] 10 endoscope device, 100 control device, 110 processing unit, 111 image generating unit, 121 auxiliary information generating unit, 140 light source unit, 141 light source driving unit, 142 filtering unit, 151 light source control unit, 152 mode determination unit, 160 storage unit, 170 interface, 200 scope unit, 210 insertion unit, 211 illumination lens, 213 camera unit, 214 light guide, 215 distance measuring sensor, 240 connector, 300 display unit, 400 information processing system, 410 processing unit, 421 auxiliary information generating unit, storage unit 460, 600 operation unit, ATA area of interest, BYH lesion, DC1 to DC4 dichroic mirrors, LDA, LDB, LDG, LDV light sources, LN1 to LN6 lenses, SS1 to SS5 optical sensors
Claims
1. An endoscope device, characterized in that: The endoscopic device comprises: a light source unit capable of changing the illumination characteristics of the illumination light and emitting display illumination light and auxiliary illumination light; and a processing unit that generates a display image based on an image signal when the display illumination light is emitted, and generates auxiliary information based on the image signal when the auxiliary illumination light is emitted, The light source emits the display illumination light and the auxiliary illumination light in a partially timed manner. When the illumination characteristic is a first illumination characteristic, the processing unit generates first auxiliary information based on the image signal, the first auxiliary information being information indicating the presence of a lesion candidate. the processing unit superimposes first display content based on the first auxiliary information on the display image, Furthermore, when the processing unit determines that the first auxiliary information satisfies a predetermined condition, the processing unit switches the illumination characteristics of the auxiliary illumination light from the first illumination characteristics to the second illumination characteristics. When the illumination characteristic is the second illumination characteristic, the processing unit generates second auxiliary information based on the image signal, the second auxiliary information being information indicating a state of the lesion candidate, the processing unit superimposes second display content based on the second auxiliary information on the display image, The endoscope device further includes a storage unit for storing information of the learned model. The learned model is a model that has been learned to output the auxiliary information for the image signal. The processing unit generates the auxiliary information based on the image signal by processing based on the learned model. The storage unit stores the information of the first learned model and the second learned model, The first learned model is a model that has been learned to output the first auxiliary information for the image signal when the illumination characteristic is the first illumination characteristic. The second learned model is a model that is learned to output the second auxiliary information for the image signal when the illumination characteristic is the second illumination characteristic. When the lighting characteristic is the first lighting characteristic, the processing unit generates the first auxiliary information by processing based on the first learned model. When the illumination characteristic is the second illumination characteristic, the processing unit generates the second auxiliary information by processing based on the second learned model.
2. The endoscope device according to claim 1, characterized in that When the illumination characteristic is the second illumination characteristic, the processing unit generates the second auxiliary information based on the image signal, wherein the content of the diagnosis or treatment of the second auxiliary information is different from that of the first auxiliary information.
3. The endoscope device according to claim 2, characterized in that: When the determination mode in the auxiliary information generation process is the first determination mode, the processing unit sets the first illumination characteristic to generate the first auxiliary information, The processing unit switches the first determination mode to a second determination mode based on the first auxiliary information, The processing unit sets the second illumination characteristics and generates the second auxiliary information in the second determination mode.
4. The endoscope device according to claim 3, characterized in that: The first determination mode is an existence diagnosis mode for diagnosing the existence of a lesion candidate included in the object. The second determination mode is a qualitative diagnosis mode for diagnosing the state of the lesion candidate.
5. The endoscope device according to claim 4, characterized in that: The processing unit determines whether the first auxiliary information satisfies the specified condition, wherein the specified condition refers to that the area of the lesion candidate included in the first auxiliary information is greater than a specified value, or the observation magnification of the imaging unit is greater than a specified value. The processing unit switches from the first determination mode to the second determination mode when it is determined that the first auxiliary information satisfies the predetermined condition.
6. The endoscope device according to claim 4, characterized in that: The lesion candidate is a tumor candidate.
7. The endoscope device according to claim 3, characterized in that: In the first determination mode, the illumination light of the first illumination characteristic includes a first group of lights, wherein the first group of lights is at least one light of a plurality of colors. In the second determination mode, the illumination light of the second illumination characteristic includes a second group of lights, wherein the second group of lights is at least one of the plurality of colors of lights and is different from the first group.
8. The endoscope device according to claim 7, characterized in that: The multiple colors of light include purple light, amber light, and green light.
9. The endoscope device according to claim 3, characterized in that: The determination mode is any one of an existence diagnosis mode for diagnosing the existence of a lesion candidate included in the subject, a qualitative diagnosis mode for diagnosing the state of the lesion candidate, or a treatment assistance mode for assisting treatment.
10. The endoscope device according to claim 9, characterized in that: The illumination light in the qualitative diagnosis mode includes light for observing inflammation or middle-layer blood vessels and deep-layer blood vessels, which are not included in the illumination light in the presence diagnosis mode. or the illumination light in the presence diagnosis mode includes purple light and amber light, Alternatively, the illumination light in the qualitative diagnosis mode includes purple light and green light, or green light, amber light and red light, or illumination light for distance measurement.
11. The endoscope device according to claim 9, characterized in that: The illumination light in the treatment assistance mode includes illumination light for distance measurement, and purple light and amber light as narrow-band light, which are used as light for extracting the lesion range or the bleeding point in the generation process of the auxiliary information. Alternatively, the illumination light in the treatment assisting mode includes red light and amber light as narrow-band light.
12. The endoscope device according to claim 2, characterized in that: The first display content is at least one of a position and a contour of a target site indicated by the first auxiliary information.
13. The endoscope device according to claim 2, characterized in that: The display illumination light is white light, The auxiliary illumination light includes at least one of purple light, amber light, and green light.
14. An information storage medium storing a program, wherein the program causes a computer to execute the following steps: The display illumination light and the auxiliary illumination light are emitted from the light source part in a timely manner; generating a display image based on an image signal when the display illumination light is emitted; generating first auxiliary information based on the image signal when the illumination characteristic of the auxiliary illumination light is a first illumination characteristic, the first auxiliary information being information indicating the presence of a lesion candidate; Overlaying first display content based on the first auxiliary information on the display image; Furthermore, when it is determined that the first auxiliary information satisfies a predetermined condition, the illumination characteristic of the auxiliary illumination light is switched from the first illumination characteristic to a second illumination characteristic; When the illumination characteristic is the second illumination characteristic, generating second auxiliary information based on the image signal, the second auxiliary information being information indicating a state of the lesion candidate; Overlaying the second display content based on the second auxiliary information on the display image, The program causes the computer to execute the following steps: When the lighting characteristic is the first lighting characteristic, generating the first auxiliary information by processing based on a first learned model; Based on the first auxiliary information, changing the lighting characteristic to the second lighting characteristic; as well as When the lighting characteristic is the second lighting characteristic, the second auxiliary information is generated by processing based on a second learned model, The first learned model is a model that has been learned to output the first auxiliary information for the image signal when the illumination characteristic is the first illumination characteristic. The second learned model is a model that is learned to output the second auxiliary information for the image signal when the illumination characteristic is the second illumination characteristic.
