ENDOSCOPE PROCESSOR, ENDOSCOPE DEVICE, AND METHOD FOR OPERATION OF ENDOSCOPE DEVICE - Patent application
The endoscope system uses multiple illumination lights and AI to improve lesion detection and visualization, addressing the issue of overlooked lesions in white light endoscopy by enhancing detection and diagnosis accuracy.
Patent Information
- Application Number
- JP2023532905
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-07-05
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2041-07-05
AI Technical Summary
Existing endoscopic systems using white light illumination may overlook lesions due to the difficulty in distinguishing them from the surrounding tissue, leading to inadequate detection and diagnosis.
An endoscope system that utilizes multiple illumination lights with different spectra to detect lesions, combining image processing with artificial intelligence to identify and highlight potential lesion areas, allowing for improved detection and differentiation.
Enhances the detection of lesions by reducing the likelihood of overlooking them, providing accurate and detailed visualization of lesions at various depths, and enabling precise computer-aided diagnosis.
Smart Images

Figure 0007592165000001 
Figure 0007592165000002 
Figure 0007592165000003
Abstract
Description
[Technical field]
[0001] The present invention relates to an endoscope processor for detecting a lesion area from image information, an endoscope device, and Method for operating an endoscope device Regarding. [Background technology]
[0002] Conventionally, endoscopes have been widely used in the medical and industrial fields. For example, when an endoscope is used in the medical field, an operator can find and identify a lesion by looking at an endoscopic image of the inside of a subject displayed on a display device, and perform treatment on the lesion using a treatment tool.
[0003] In recent years, computer-aided image diagnosis (CAD: Computer Aided Detection / Diagnosis) has been developed to prevent operators from overlooking lesions and to facilitate diagnosis by quantitatively analyzing endoscopic images using a computer and displaying the location of possible lesions and differential diagnosis information on the endoscopic image. When a lesion is detected by CAD, an endoscopic device has been proposed that notifies the operator of the presence and location of the lesion by highlighting it (for example, displaying a marker such as a frame) on the endoscopic image.
[0004] For example, WO2019 / 087971 describes a medical image processing device and an endoscopic device that detect a lesion area using a first illumination light, identify the type of lesion using a second illumination light, and identify the degree of progression using a third illumination light.
[0005] However, in WO2019 / 087971, the first illumination light for detecting the lesion area is specifically a differentiation illumination light of pattern 1 as shown in Figure 5 of the publication. This differentiation illumination light is white light (= normal light) close to the color observed by the naked eye, and is not a special light that makes it relatively easy to recognize the lesion, so there is a possibility that the lesion will be overlooked.
[0006] The present invention has been made in view of the above circumstances, and provides an endoscope processor, an endoscope device, and Method for operating an endoscope device The purpose is to provide. DISCLOSURE OF THEINVENTION [Means for solving the problem]
[0007] An endoscope processor according to one aspect of the present invention includes a processor, the processor detects a first region of a lesion candidate from first image information acquired by irradiating a first illumination light, and detects a second region of the lesion candidate from second image information acquired by irradiating a second illumination light having a spectrum different from that of the first illumination light; Depending on the part of the subject to be observed, The first region and the second region From among A display area of the lesion candidate is selected, and display image information is generated by superimposing the display area on the first image information.
[0008] An endoscopic device according to one aspect of the present invention includes an endoscope including a light source device capable of irradiating a plurality of types of illumination light including a first illumination light and a second illumination light having a spectrum different from that of the first illumination light, and an imaging device that irradiates the first illumination light from the light source device to obtain first image information and irradiates the second illumination light from the light source device to obtain second image information; and an endoscope that detects a first region of a lesion candidate from the first image information and detects a second region of the lesion candidate from the second image information; Depending on the part of the subject to be observed, The first region and the second region From among The endoscope apparatus includes an endoscope processor having a processor that selects a display area of a lesion candidate and generates display image information by superimposing the display area on the first image information, and a monitor that displays the display image information.
[0009] According to one aspect of the present invention Method for operating an endoscope device teeth, A method for operating an endoscope apparatus including a first classifier, a second classifier, a lesion area selection unit, and a display processing unit, the first classifier comprising: detecting a first region of a lesion candidate from first image information acquired by irradiating the first illumination light; The second discriminator is detecting a second region of the lesion candidate from second image information acquired by irradiating the subject with a second illumination light having a spectrum different from that of the first illumination light; The lesion area selection unit, selecting a display area for the lesion candidate from among the first area and the second area according to an observation target site of the subject; The display processing unit,Display image information is generated by superimposing the display area on the first image information. [Brief description of the drawings]
[0010] [Figure 1] 1 is a perspective view showing an example of the appearance of an endoscope apparatus according to a first embodiment of the present invention. [Diagram 2] FIG. 2 is a block diagram showing an example of a configuration of an endoscope apparatus according to the first embodiment. [Diagram 3] FIG. 2 is a block diagram showing an example of the electrical configuration of the endoscope processor according to the first embodiment. [Figure 4] FIG. 2 is a block diagram showing an example of the configuration of a classifier according to the first embodiment. [Diagram 5] 11 is a table showing an example of image information on each classifier and a monitor in the first embodiment. [Figure 6] 5 is a flowchart showing processing of the endoscope processor according to the first embodiment. [Figure 7] FIG. 11 is a block diagram showing an example of the configuration of a classifier according to a second embodiment of the present invention. [Figure 8] 10 is a flowchart showing processing of the endoscope processor according to the second embodiment. [Figure 9] FIG. 13 is a block diagram showing an example of the configuration of a classifier according to a third embodiment of the present invention. [Figure 10] 13 is a flowchart showing processing of an endoscope processor according to the third embodiment. BEST MODE FOR CARRYING OUT THEINVENTION
[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the present invention is not limited to the embodiment described below. In the description of the drawings, the same or corresponding elements are appropriately designated by the same reference numerals. [First embodiment]
[0012] 1 to 6 show a first embodiment of the present invention, and Fig. 1 is a perspective view showing an example of the configuration of an endoscope device 1. The endoscope device 1 of this embodiment includes an endoscope 2, a light source device 3, an endoscope processor 4, and a monitor 5.
[0013] The endoscope 2 comprises a long and thin insertion section 9 that is inserted into the subject, an operation section 10 for performing various operations related to the endoscope 2, and a universal cable 17 for connecting the endoscope 2 to the light source device 3 and the endoscope processor 4.
[0014] The insertion section 9 includes, in order from the tip to the base end, a tip section 6, a bending section 7, and a flexible tube section 8. The tip section 6 includes an illumination window for emitting illumination light to the subject and an observation window into which return light from the subject enters, although not shown. The endoscope 2 of this embodiment is configured as an electronic endoscope, and an imaging device 21 (see FIG. 2) is provided at the tip section 6. The imaging device 21 includes an imaging optical system and an imaging element. The imaging optical system forms light incident from the observation window on the imaging element as an optical image of the subject (subject image). The imaging element is an image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The imaging element photoelectrically converts the subject image to generate an imaging signal, and outputs it. The imaging signal is transmitted to the endoscope processor 4 via a signal line.
[0015] The bending section 7 is a bendable section connected to the base end side of the tip section 6, and by bending it changes the direction in which the tip section 6 faces. By changing the direction of the tip section 6, the observed part of the subject can be changed or the insertability of the endoscope 2 can be improved.
[0016] The flexible tube portion 8 is connected to the base end side of the bending portion 7. deflection It is a part that has sexual activity.
[0017] A bending wire for bending the bending portion 7 and a treatment tool channel for inserting a treatment tool are disposed in the insertion portion 9 and the operation portion 10. In addition, the above-mentioned signal line connected to the imaging element and a light guide for transmitting illumination light are disposed in the insertion portion 9, operation portion 10, and universal cable 17 of the endoscope 2.
[0018] The operation unit 10 is provided with a bending operation unit 14 for bending the bending portion 7 via the bending wire, various switches including a focus switch 15, and the like. When the imaging optical system is equipped with a variable magnification optical system, the focus position is changed to a near point or a far point by operating the focus switch 15, and the magnification of the subject image is changed. The bending operation unit 14 is provided with a UD bending operation knob 12 for bending the bending portion 7 in the up-down direction, and a RL bending operation knob 13 for bending the bending portion 7 in the left-right direction. The bending portion 7 can also bend in an oblique direction by combining bending in the up-down direction and bending in the left-right direction.
[0019] On the distal end side of the operation section 10, there are provided a gripping section 11 for an operator to grip the endoscope 2 with his / her hand, and a treatment instrument channel insertion port 16 which is an opening on the proximal end side of the treatment instrument channel described above.
[0020] The universal cable 17 extends from, for example, a side surface on the base end side of the operation unit 10. A scope connector 17a is provided at the base end of the universal cable 17. The scope connector 17a detachably connects the endoscope 2 to the light source device 3. By connecting the scope connector 17a to the light source device 3, it becomes possible to transmit illumination light through a light guide.
[0021] A coiled electric cable 18 extends from the side of the scope connector 17a. An electric connector 18a provided at the extending end of the electric cable 18 is detachably connected to the endoscope processor 4. When the electric connector 18a is connected to the endoscope processor 4, the imaging element is electrically connected to the endoscope processor 4.
[0022] The endoscope processor 4 is electrically connected to a monitor 5 which is a display device. The endoscope processor 4 processes an imaging signal output from an imaging element of the endoscope 2 to generate image information for display. The image information for display is output from the endoscope processor 4 to the monitor 5 and displayed on the monitor 5 as a display image including an endoscopic image. The monitor 5 also includes a speaker 5a for outputting sound.
[0023] FIG. 2 is a block diagram showing an example of the configuration of the endoscope device 1. As shown in FIG.
[0024] As described above, the endoscope 2 includes the imaging device 21. The imaging device 21 acquires first image information (white light image information, described later) by imaging the subject illuminated with the first illumination light from the light source device 3, and acquires second image information (first special light image information and second special light image information, described later) by imaging the subject illuminated with the second illumination light from the light source device 3.
[0025] The light source device 3 can emit a plurality of types of illumination light including a first illumination light and a second illumination light having a spectrum different from that of the first illumination light. The light source device 3 of this embodiment includes a white light source 31, a first special light source 32, and a second special light source 33. The white light source 31 emits white light for observation. The first special light source 32 emits first special light having a spectrum different from that of the white light. The second special light source 33 emits second special light having a spectrum different from that of the white light and the first special light. The white light is the first illumination light, and the first special light and the second special light are the second illumination light.
[0026] Specifically, the light source device 3 has a plurality of light sources that emit light of various colors, such as R (red), G (green), B (blue), V (violet), and A (amber), and by combining the light sources of each color, the above-mentioned white light source 31, first special light source 32, and second special light source 33 are formed.
[0027] The light source device 3 includes a light emitting device such as an LED (Light Emitting Diode) or an LD (Laser Diode). As an example, the light source device 3 includes a V-LED that emits purple (V) light with a central wavelength of about 405 nm, a B-LED that emits blue (B) light with a central wavelength of about 445 nm, a G-LED that emits green (G) light with a central wavelength of about 540 nm, and an R-LED that emits red (R) light with a central wavelength of about 630 nm. The light source device 3 also includes a prism, a mirror, an optical fiber, or an optical filter that adjusts the wavelength band or the amount of light, etc., as necessary.
[0028] The light source device 3 of this embodiment is configured to emit, for example, white light, first special light, and second special light in sequence for each frame. As a result, the imaging device 21 acquires white light image information (hereinafter referred to as white light image), first special light image information (hereinafter referred to as first special light image), and second special light image information (hereinafter referred to as second special light image) in sequence, and after the second special light image, it acquires a white light image again, repeating this process. However, it is not limited to acquiring each image in one frame, and any image may be acquired in multiple frames. For example, the white light image and the first special light image are each acquired in one frame, while the second special light image is acquired in two frames, etc.
[0029] The endoscope processor 4 includes an image processing unit 41, a white light discriminator 42, a first special light discriminator 43, a second special light discriminator 44, a lesion area selection unit 45, a display processing unit 46, a bus 47, and a control unit 48.
[0030] 2 shows three identifiers, namely, white light identifier 42, first special light identifier 43, and second special light identifier 44, but two, four or more identifiers may be provided. Also, a single identifier having multiple identifier functions may be provided and configured to function as any of white light identifier 42, first special light identifier 43, and second special light identifier 44 by switching the identifier functions.
[0031] Fig. 3 is a block diagram showing an example of the electrical configuration of the endoscope processor 4. Fig. 2 shows the functional configuration of the endoscope processor 4, and the endoscope processor 4 includes, as an electrical configuration, a processor 4a and a memory 4b, for example.
[0032] The processor 4a includes, for example, an ASIC (Application Specific Integrated Circuit) including a CPU (Central Processing Unit) or an FPGA (Field Programmable Gate Array). The memory 4b is a storage medium such as a RAM (Random Access Memory), a flash memory, or a disk storage medium. The memory 4b includes a non-transitory storage medium readable by a computer that records a processing program.
[0033] The processor 4a loads and executes a processing program stored in the memory 4b to perform the functions of each unit shown in Fig. 2. However, the present invention is not limited to this configuration, and the processor 4a may be configured as a dedicated electronic circuit that performs the functions of each unit.
[0034] 4 is a block diagram showing an example configuration of classifiers 42, 43, and 44. White light classifier 42, first special light classifier 43, and second special light classifier 44 each include lesion classifier 4c that detects a region of a lesion candidate from image information acquired by irradiating the lesion with illumination light. Lesion classifier 4c includes, for example, artificial intelligence (AI) that has learned lesion images.
[0035] The image processing unit 41 performs various processes such as demosaicing, noise correction, color correction, contrast correction, and gamma correction on the image information output from the imaging device 21, and converts it into an image signal (display image information) in a format that can be output to the monitor 5.
[0036] The white light discriminator 42 is a first discriminator, and the first special light discriminator 43 and the second special light discriminator 44 are second discriminators.
[0037] White light discriminator 42 includes an AI that has learned the lesion image captured as a white light image by machine learning, deep learning, etc. White light discriminator 42 detects a lesion candidate region (a first region of a lesion candidate) from an endoscopic image (white light image) acquired by imaging device 21 by irradiating a subject with white light emitted from white light source 31. White light discriminator 42 also calculates a reliability score for the detected lesion candidate region. The reliability score indicates the probability (certainty) that the lesion candidate region is actually a lesion.
[0038] First special light classifier 43 includes an AI that has learned the lesion image captured as the first special light image by machine learning, deep learning, etc. First special light classifier 43 detects a lesion candidate region (second lesion candidate region) from an endoscopic image (first special light image) acquired by imaging device 21 by irradiating the subject with the first special light emitted from first special light source 32. In addition, first special light classifier 43 calculates a reliability score of the detected lesion candidate region.
[0039] The second special light discriminator 44 includes an AI that has learned the lesion image captured as the second special light image by machine learning, deep learning, etc. The second special light discriminator 44 detects a lesion candidate region (a second region of a lesion candidate) from an endoscopic image (a second special light image) acquired by the imaging device 21 by irradiating the subject with the second special light emitted from the second special light source 33. The second special light discriminator 44 also calculates a reliability score of the detected lesion candidate region.
[0040] The first special light discriminator 43 and the second special light discriminator 44 can be configured as discriminators that detect lesion candidate regions at different depths (superficial, intermediate, and deep layers) from the mucosal surface. The detection of lesion candidate regions is performed, for example, based on an image that emphasizes blood vessel information at a target depth. As the special light that emphasizes blood vessel information, a set of narrowband light wavelengths that reach sufficiently deep to the target depth and have a difference in the absorption coefficient μa of oxygenated and reduced hemoglobin and narrowband light wavelengths at an isosbestic point with no difference (a narrowband light wavelength set that is optimal when calculating the oxygen saturation of blood vessels at each depth of the superficial, intermediate, and deep layers) is selected, for example. Illumination light of such a wavelength set is known as illumination light for NBI (Narrow Band Imaging).
[0041] The superficial lesion classifier classifies superficial lesions that are located on the mucosal surface or at a relatively shallow depth from the mucosal surface (a depth of about several tens of μm from the mucosal surface). The special light that emphasizes superficial vascular information is, for example, light that includes violet light (405 nm) as a reference light and blue light (445 nm) as a measurement light, and in which the amount of violet light is greater than the amount of blue light. The superficial lesion classifier is a classifier that is trained using lesion images captured with special light that emphasizes superficial vascular information.
[0042] The classifier for medium-layer lesions classifies medium-layer lesions that are at a medium depth (a depth of several tens to several hundreds of μm) from the mucosal surface. The special light that emphasizes medium-layer vascular information includes, for example, blue light (473 nm) as measurement light, green light as reference light, and red light as reference light, and the amount of blue light is greater than the amount of green light, and the amount of green light is greater than the amount of red light. The classifier for medium-layer lesions is a classifier that is trained using lesion images captured with special light that emphasizes medium-layer vascular information.
[0043] The deep-layer lesion classifier classifies deep-layer lesions that are deep from the mucosal surface (from the muscularis mucosa to the submucosal tissue layer). The special light that emphasizes deep-layer vascular information is, for example, a classifier trained on lesion images captured with special light that includes blue light as a reference light, green light as a reference light, and red light (630 nm) as a measurement light, in which the amount of green light is greater than the amount of blue light and the amount of blue light is greater than the amount of red light.
[0044] Although the classifiers for lesions at different depths from the mucosal surface are given as examples of the second classifier (first special light classifier 43, second special light classifier 44), the present invention is not limited to this. The second classifier may be a classifier trained on a lesion image captured with special light (illumination light for RDI (Red Dichromatic Imaging)) that penetrates objects that would be obstacles for detecting a lesion candidate region with white light, such as residue or bile on the mucosa, or blood on the mucosa due to bleeding. The illumination light for RDI uses light of specific wavelengths of three colors, for example, green, amber, and red. Therefore, the multiple types of second illumination light may include at least one of illumination light for NBI and illumination light for RDI.
[0045] In this embodiment, as an example, the first special light discriminator 43 is a discriminator for superficial lesions, and the second special light discriminator 44 is a discriminator for intermediate lesions.
[0046] Lesion area selection unit 45 selects an area for displaying the lesion candidate based on the lesion candidate area detected by white light discriminator 42, the lesion candidate area detected by first special light discriminator 43, and the lesion candidate area detected by second special light discriminator 44.
[0047] The method of selecting the display area by lesion area selecting unit 45 may be, for example, any of the following methods (1) to (3). All of the following methods (1) to (3) are methods of calculating a reliability score for the lesion candidate area (first area) in the white light image, a reliability score for the lesion candidate area (second area) in the first special light image, and a reliability score for the lesion candidate area (second area) in the second special light image, and selecting the display area based on the multiple reliability scores.
[0048] In the following, the lesion candidate region detected by white light discriminator 42 will be referred to as region 1, the lesion candidate region detected by first special light discriminator 43 as region 2, and the lesion candidate region detected by second special light discriminator 44 as region 3, as appropriate. With regard to region 1, there are cases where no lesion candidate is detected and region 1 does not exist, and there are also cases where one or more lesion candidates are detected in one image and one or more region 1s exist. As with region 1, there are also cases where 0, 1, or more than one region 2 and 3 exist. (1) First, among the regions 1 to 3, regions with a reliability score equal to or greater than a predetermined threshold (prescribed value) are extracted, and if there are multiple regions among the extracted regions that overlap, all of the regions are merged and selected as the display region. In addition, if there are no overlapping regions among the extracted regions, they are selected as the display region as they are. (2) First, among the regions 1 to 3, regions with a reliability score equal to or greater than a predetermined threshold (prescribed value) are extracted, and among the extracted regions that overlap, the region with the highest reliability score is selected as the display region. In addition, among the extracted regions that do not overlap, they are selected as the display region as they are. (3) The reliability score is weighted according to the observation area of the subject (the organ that is the subject of the endoscopic examination), and the display area is selected by applying the method of (2) based on the weighted reliability score.
[0049] Method (3) will be further explained. In the following, the reliability score of area 1 will be called score 1, the reliability score of area 2 will be called score 2, and the reliability score of area 3 will be called score 3. The weighting coefficient multiplied by score 1 will be called weighting coefficient 1, the weighting coefficient multiplied by score 2 will be called weighting coefficient 2, and the weighting coefficient multiplied by score 3 will be called weighting coefficient 3.
[0050] A diagnostic method is being established for organs such as the esophagus and large intestine, using images captured with special light that emphasizes superficial blood vessel information. Therefore, when the organ is the esophagus or large intestine, weighting factor 2 is made larger than weighting factor 1 and weighting factor 3, and method (2) is applied to select the display region. At this time, if weighting factor 1 and weighting factor 3 are set to 0, only region 2 detected by first special light classifier 43, which is a classifier for superficial lesions, is selected, which is the same as selecting first special light classifier 43 as the classifier.
[0051] Furthermore, when the organ is the stomach and the lesion is scirrhous gastric cancer, a diagnostic method is being established that uses images captured with special light that emphasizes mid-layer vascular information. Therefore, when the organ is the stomach, weighting factor 3 is made larger than weighting factor 1 and weighting factor 2 to apply method (2) and select the display region. In this case, if weighting factor 1 and weighting factor 2 are set to 0, only region 3 detected by second special light classifier 44, which is a classifier for mid-layer lesions, will be selected, which is the same as selecting second special light classifier 44 as the classifier.
[0052] Therefore, depending on the organ, which of the suspected lesion area (first area) of the white light image, the suspected lesion area (second area) of the first special light image, and the suspected lesion area (second area) of the second special light image is selected as the display area will differ.
[0053] The weighting according to the organ may be switched according to a user instruction (i.e. manually), or may be automatically determined by the control unit 48 based on the characteristics of the endoscopic image, or may be automatically performed by detecting the insertion length of the insertion portion 9 of the endoscope 2 and the control unit 48 determining based on the detection results, or a position detection sensor may be incorporated in the tip portion 6 of the endoscope 2 and the control unit 48 determining based on the detection results of the position detection sensor.
[0054] The white light image is transmitted from image processing unit 41 to display processing unit 46 via white light discriminator 42 and lesion area selection unit 45. Furthermore, the display area selected by lesion area selection unit 45 is transmitted to display processing unit 46.
[0055] The bus 47 is a transmission path through which each unit in the endoscope processor 4 transmits and receives commands and information.
[0056] The control unit 48 is connected to the image processing unit 41, the white light discriminator 42, the first special light discriminator 43, the second special light discriminator 44, the lesion area selection unit 45, and the display processing unit 46 via the bus 47 and controls these.
[0057] The control unit 48 includes a motion detection unit 48a. The motion detection unit 48a detects the motion between the white light image, the first special light image, and the second special light image. The control unit 48 may detect motion by image analysis, for example, or may incorporate an acceleration sensor or the above-mentioned position detection sensor in the tip 6 of the endoscope 2 and perform motion detection based on the detection results of the sensor.
[0058] Image movement occurs due to movement of imaging device 21 provided at tip 6 of endoscope 2 relative to the subject. Types of image movement include, for example, vertical and horizontal movement on the screen, rotational movement of imaging device 21 relative to the subject, and image enlargement / reduction movement due to tip 6 moving closer to / away from the subject (or due to magnification change if imaging device 21 is equipped with a variable magnification optical system). Control unit 48 transmits image movement information detected by movement detection unit 48a to lesion area selection unit 45 and display processing unit 46.
[0059] Lesion area selection unit 45 aligns the images (corrects positional misalignment) based on image movement information transmitted from control unit 48 (thus aligning lesion candidate areas in the images), and then selects a display area from areas 1 to 3.
[0060] The display processing unit 46 aligns the display area with the white light image based on the image motion information transmitted from the control unit 48. However, the alignment of the display area with respect to the white light image may be performed by the lesion area selection unit 45. Then, the display processing unit 46 superimposes the display area on the white light image to generate display image information, and outputs it to the monitor 5. Note that the display area superimposed on the white light image may be, for example, the contour of the merged area, or it may be a marker (for example, a rectangular frame) indicating the range of the merged area.
[0061] The monitor 5 displays a display image on the monitor screen according to the display image information input from the display processing unit 46 .
[0062] FIG. 5 is a diagram showing an example of image information in each of the classifiers 42, 43, and 44 and the monitor 5. In FIG.
[0063] White light discriminator 42 detects lesion candidate region 51 a from white light image 51 .
[0064] First special light discriminator 43 detects lesion suspected region 52 a from first special light image 52 .
[0065] Second special light discriminator 44 detects lesion suspected region 53 a from second special light image 53 .
[0066] The monitor 5 displays the display image 50 on which the display area 50a is superimposed on the monitor screen. Here, the display area 50a is, for example, an area obtained by merging the lesion candidate area 51a, the lesion candidate area 52a, and the lesion candidate area 53a. The display of the discrimination result 50b and the progression degree 50c shown in FIG. 5 will be described in a later embodiment.
[0067] Fig. 6 is a flowchart showing the processing of the endoscope processor 4. Fig. 6 shows an example in which a white light image, a first special light image, and a second special light image are acquired in this order every three frames.
[0068] For example, in the n-th frame (n is an integer), white light is emitted from the light source device 3, and a white light image is acquired by the imaging device 21. The acquired white light image is input to the endoscope processor 4 (step S1). The input white light image is processed by the image processing unit 41, and then sent to the white light identifier 42 and the display processing unit 46.
[0069] White light classifier 42 detects lesion candidate regions from the white light image using lesion classifier 4c within white light classifier 42, and calculates a reliability score for each detected lesion candidate region (step S2).
[0070] When a lesion candidate region is detected, control unit 48 determines whether or not the reliability score calculated by white light discriminator 42 is equal to or greater than a specified value (threshold value) for each lesion candidate region (step S3). When a lesion candidate region is not detected, control unit 48 skips the process of step S3 and proceeds to step S4.
[0071] If control unit 48 determines that no lesion candidate region is detected or that the reliability scores of all lesion candidate regions are less than a specified value, it determines that there is no lesion (step S4) and does not transmit the lesion candidate region from white light discriminator 42 to lesion region selection unit 45 even if there is a lesion candidate region.
[0072] On the other hand, if it is determined that there is a lesion candidate region whose reliability score is equal to or greater than the specified value, control unit 48 causes white light discriminator 42 to transmit that lesion candidate region to lesion region selection unit 45 .
[0073] In the subsequent (n+1)th frame, the light source device 3 emits a first special light that emphasizes superficial blood vessels, and the imaging device 21 acquires a first special light image. The acquired first special light image is input to the endoscope processor 4 (step S5). The input first special light image is processed by the image processing unit 41 and then sent to the first special light identifier 43.
[0074] First special light classifier 43 detects lesion candidate regions from the first special light image using lesion classifier 4c within first special light classifier 43, and calculates a reliability score for each detected lesion candidate region (step S6).
[0075] When a lesion candidate region is detected, control unit 48 determines whether or not the reliability score calculated by first special light discriminator 43 is equal to or greater than a specified value (threshold value) for each lesion candidate region (step S7). When a lesion candidate region is not detected, control unit 48 skips the process of step S7 and proceeds to step S8.
[0076] If control unit 48 determines that no lesion candidate area is detected or that the reliability scores of all lesion candidate areas are less than a specified value, it determines that there is no lesion (step S8) and does not transmit the lesion candidate area from first special light identifier 43 to lesion area selection unit 45 even if there is a lesion candidate area.
[0077] On the other hand, if control unit 48 determines that there is a lesion candidate region whose reliability score is equal to or greater than the specified value, control unit 48 causes first special light discriminator 43 to transmit that lesion candidate region to lesion region selection unit 45 .
[0078] In the (n+2)th frame thereafter, the light source device 3 emits second special light that emphasizes the middle layer blood vessels, and the imaging device 21 acquires a second special light image. The acquired second special light image is input to the endoscope processor 4 (step S9). The input second special light image is processed by the image processing unit 41 and then sent to the second special light identifier 44.
[0079] Second special light classifier 44 detects lesion candidate regions from the second special light image using lesion classifier 4c therein, and calculates a reliability score for each detected lesion candidate region (step S10).
[0080] When a lesion candidate region is detected, control unit 48 determines whether or not the reliability score calculated by second special light discriminator 44 is equal to or greater than a specified value (threshold value) for each lesion candidate region (step S11). When a lesion candidate region is not detected, control unit 48 skips the process of step S11 and proceeds to step S12.
[0081] If control unit 48 determines that no lesion candidate area is detected or that the reliability scores of all lesion candidate areas are less than a specified value, it determines that there is no lesion (step S12) and does not transmit the lesion candidate area from second special light identifier 44 to lesion area selection unit 45 even if there is a lesion candidate area.
[0082] On the other hand, if control unit 48 determines that there is a lesion candidate region whose reliability score is equal to or greater than the specified value, control unit 48 causes second special light discriminator 44 to transmit that lesion candidate region to lesion region selection unit 45 .
[0083] If it is determined in step S3 that the reliability score is equal to or greater than the specified value, if it is determined in step S7 that the reliability score is equal to or greater than the specified value, or if it is determined in step S11 that the reliability score is equal to or greater than the specified value, lesion area selection unit 45 selects a display area from the lesion candidate areas obtained based on the images of frames n to (n+2) (step S13). Here, lesion area selection unit 45, for example, aligns all of the input lesion candidate areas as described above and merges them to obtain a display area. That is, all of the lesion candidate areas processed by lesion area selection unit 45 are areas with reliability scores equal to or greater than the specified value.
[0084] Next, the display processing unit 46 aligns the display area and performs image synthesis to superimpose it on the white light image (first image information) to generate display image information (step S14). The generated display image information is output from the endoscope processor 4 to the monitor 5, and the display image is displayed on the monitor screen of the monitor 5.
[0085] After processing the (n+2)th frame, the (n+3), (n+4), and (n+5)th frames are processed in the same manner as the n, (n+1), and (n+2) frames described above, and so on.
[0086] Note that if display image information is generated once every three frames and a display image is displayed, the frame rate will decrease. Therefore, it is possible to prevent the frame rate from decreasing by sequentially displaying a display image based on images of frames n to (n+2), then displaying a display image based on images of frames (n+1) to (n+3), and then displaying a display image based on images of frames (n+2) to (n+4), and so on.
[0087] According to the first embodiment, a plurality of lesion candidate regions are detected from a plurality of image information acquired by irradiating a plurality of illumination lights having different spectra including white light, a display region is selected from the plurality of lesion candidate regions, and display image information is generated by superimposing the selected display region on the white light image, thereby realizing a function of detecting lesions that may be overlooked in white light. Also, the details of the subject that are dark and difficult to see in the special light image can be visually recognized in the white light image.
[0088] Furthermore, by using a plurality of types of second illumination light, that is, the first special light and the second special light, for example, lesions at different depths from the mucosal surface can be detected from each image.
[0089] In this case, if NBI (Narrow Band Imaging) illumination light is used as the second illumination light, blood vessels can be emphasized.In addition, if RDI (Red Dichromatic Imaging) illumination light is used as the second illumination light, lesions can be detected by penetrating obstacles such as residue, bile, and blood.
[0090] Then, the display area of the lesion candidate is selected based on the reliability score, thereby improving the accuracy of lesion detection. At this time, by selecting the display area from the area with a reliability score higher than a threshold, the surgeon can concentrate on observing the area that is likely to be a lesion.
[0091] In addition, by selecting the lesion candidate region detected from the image information related to which illumination light is selected as the display region depending on the organ that is the subject's observation area, it is possible to perform accurate computer-aided image diagnosis (CAD: Computer Aided Detection / Diagnosis) corresponding to lesions for which diagnostic methods have been established.
[0092] Then, the suspected lesion areas detected from the multiple image information are aligned according to the results of motion detection, and then a display area is selected and image information for display is generated. Therefore, even if there is movement between the multiple images, a marker can be displayed in the correct position on the white light image.
[0093] In this way, by performing computer-aided image diagnosis in which a display area for a lesion candidate is selected based on a comprehensive judgment from multiple types of images, including a white light image and a special light image, and the display area is displayed on the white light image with a marker or the like, it becomes easier to find lesions by reducing oversight of lesions and to accurately grasp the scale of the lesion. This makes it easier for the surgeon to look at the displayed image and make a diagnosis. [Second embodiment]
[0094] 7 and 8 show a second embodiment of the present invention, and Fig. 7 is a block diagram showing an example of the configuration of discriminators 42, 43, and 44. In this second embodiment, parts that are the same as those in the above-mentioned first embodiment are given the same reference numerals, and the description will be omitted as appropriate, and mainly only the differences will be described.
[0095] In this embodiment, in addition to displaying the lesion candidate region in the above-described first embodiment, a discrimination result for the lesion candidate region is also displayed.
[0096] As shown in FIG. 7, each of the white light discriminator 42, the first special light discriminator 43, and the second special light discriminator 44 of this embodiment includes a discrimination discriminator 4d in addition to a lesion discriminator 4c.
[0097] The classifier 4d includes, for example, an AI that has learned lesion images. The classifier 4d classifies the type of each lesion for one or more lesion candidate regions detected by the lesion classifier 4c, and calculates a reliability score for the classified lesion type.
[0098] The discrimination classifier 4d discriminates the lesion candidate region as a "polyp", for example, and calculates the confidence score of the "polyp" to be, for example, "60%".
[0099] However, the discrimination classifier 4d is not limited to outputting one discrimination result, but may output discrimination results for multiple types of lesions, such as "confidence score for polyp is 60%", "confidence score for ulcerative colitis is 10%", and "confidence score for Crohn's disease is less than 1%".
[0100] In FIG. 7, the lesion classifier 4c and the discrimination classifier 4d are shown as separate blocks, but one AI may be configured to function as both blocks.
[0101] Fig. 8 is a flowchart showing the processing of the endoscope processor 4. In Fig. 8, the parts different from Fig. 6 will be described below.
[0102] For the white light image acquired in the nth frame, white light discriminator 42 performs the process of step S2, and also performs classification by discrimination discriminator 4d in white light discriminator 42 on the lesion candidate region detected by lesion discriminator 4c in white light discriminator 42. Discrimination discriminator 4d calculates the classification result of the lesion candidate region and a reliability score for the classification result (step S21). Note that, if no lesion candidate region is detected by lesion discriminator 4c, discrimination discriminator 4d skips the process of step S21.
[0103] For the first special light image acquired in the (n+1)th frame, first special light classifier 43 performs the process of step S6, and also performs classification by discrimination classifier 4d in first special light classifier 43 on the lesion candidate region detected by lesion classifier 4c in first special light classifier 43. Discrimination classifier 4d calculates the classification result of the lesion candidate region and a reliability score for the classification result (step S22). Note that, if no lesion candidate region is detected by lesion classifier 4c, discrimination classifier 4d skips the process of step S22.
[0104] With respect to the second special light image acquired in the (n+2)th frame, second special light classifier 44 performs the process of step S10, and also performs classification by discrimination classifier 4d in second special light classifier 44 on the lesion candidate region detected by lesion classifier 4c in second special light classifier 44. Discrimination classifier 4d calculates the classification result of the lesion candidate region and a reliability score for the classification result (step S23). Note that, if no lesion candidate region is detected by lesion classifier 4c, discrimination classifier 4d skips the process of step S23.
[0105] The lesion area selection unit 45 performs the processing of step S13 and compares the reliability scores for the discrimination results obtained in step S21, the reliability scores for the discrimination results obtained in step S22, and the reliability scores for the discrimination results obtained in step S23 to determine which discrimination result to display, and outputs the determined discrimination result to the display processing unit 46 (step S24).
[0106] Here, the lesion area selection unit 45 may determine, for example, the discrimination result with the highest reliability score as the discrimination result to be displayed. Alternatively, the lesion area selection unit 45 may determine several (i.e., multiple) discrimination results in descending order of reliability score as the discrimination results to be displayed, and display the reliability scores side by side with the discrimination results to be displayed.
[0107] The display processing unit 46 performs image synthesis to align the display area and superimpose it on the white light image (first image information), and further performs image synthesis to include the discrimination result in the vicinity of the white light image, generating display image information (step S14A). The generated display image information is output from the endoscope processor 4 to the monitor 5, and the display image is displayed on the monitor screen of the monitor 5. As a result, the discrimination result 50b as shown in FIG. 5 is displayed.
[0108] According to the second embodiment, effects similar to those of the first embodiment described above are achieved, and since the type of lesion is differentiated and display image information including the differentiation results is generated, it is possible to reduce variability in diagnosis of the type of lesion due to the surgeon's subjective judgment. [Third embodiment]
[0109] 9 and 10 show a third embodiment of the present invention, and Fig. 9 is a block diagram showing an example of the configuration of discriminators 42, 43, and 44. In this third embodiment, parts that are the same as those in the first and second embodiments described above are given the same reference numerals, and the description will be omitted as appropriate, and only the differences will be mainly described.
[0110] In this embodiment, in addition to displaying the lesion candidate region and the discrimination result in the second embodiment described above, the progression level of the lesion is also displayed.
[0111] As shown in FIG. 9, each of the white light discriminator 42, the first special light discriminator 43, and the second special light discriminator 44 of this embodiment includes a progression discriminator 4e in addition to a lesion discriminator 4c and a discrimination discriminator 4d.
[0112] The progression level classifier 4e includes, for example, an AI that has learned lesion images. The progression level classifier 4e classifies the progression level of the lesion classified by the classification classifier 4d for the lesion candidate region, and calculates a reliability score of the progression level.
[0113] In FIG. 9, the lesion classifier 4c, the discrimination classifier 4d, and the progression classifier 4e are illustrated as separate blocks, but one AI may be configured to function as the three blocks.
[0114] For example, endoscopic findings of polyps (including cancer) are classified by the Narrow-band imaging International Colorectal Endoscopic (NICE) classification and the Japan NBI (Narrow Band Imaging) Expert Team (JNET) classification.
[0115] For example, when identifying the stage of a polyp according to the NICE classification, it is classified into one of "NICE Type 1," "NICE Type 2," and "NICE Type 3." "NICE Type 1" represents hyperplastic lesions, "NICE Type 2" represents adenoma to intramucosal cancer (so-called M cancer), and "NICE Type 3" represents submucosal invasive cancer (so-called SM cancer).
[0116] In addition, endoscopic classifications of ulcerative colitis include the Mayo classification, the Matts classification, and the UCEIS (Ulcerative Colitis Endoscopic Index of Severity) classification.
[0117] For example, when identifying the degree of progression of ulcerative colitis according to the Mayo classification, it is classified into one of four grades: Mayo 0 (grade 0), Mayo 1 (grade 1), Mayo 2 (grade 2), and Mayo 3 (grade 3). Mayo 0 is a grade that indicates normal or inactive (including remission). Mayo 1 is a grade that indicates mild disease, and generally indicates a state in which redness, unclear vascular images, or mild bleeding tendency is observed. Mayo 2 is a grade that indicates moderate disease, and generally indicates a state in which significant redness, disappearance of vascular images, bleeding tendency, adhesion of purulent secretions, rough mucosa, erosion, or partial ulcers are observed. Mayo 3 is a grade that indicates severe disease (active stage), and generally indicates a state in which obvious spontaneous bleeding, edema, ulcers (including extensive ulcers), and the like are observed. Dysplasia associated with ulcerative colitis may also be identified.
[0118] The endoscopic classification of Crohn's disease includes the Simple Endoscopic Score for Crohn's Disease (SESCD). When identifying the stage of Crohn's disease according to the SESCD, it is classified as "SESCD 0," "SESCD 1," "SESCD 2," or "SESCD 3."
[0119] When the lesion area selection unit 45 determines that a plurality of discrimination results are to be displayed in descending order of reliability score, the degree of progression is also identified for each of the plurality of discrimination results.
[0120] Fig. 10 is a flowchart showing the processing of the endoscope processor 4. In Fig. 10, the parts different from Fig. 8 will be described below.
[0121] For the white light image acquired in the nth frame, white light classifier 42 performs the processes of steps S2 and S21, and also causes progression classifier 4e in white light classifier 42 to classify the progression of the lesion classified by classification classifier 4d in white light classifier 42. Progression classifier 4e calculates the classification result of the progression and a reliability score for the classified progression (step S31). Note that, if no lesion candidate region is detected by lesion classifier 4c, classification classifier 4d skips the process of step S31.
[0122] For the first special light image acquired in the (n+1)th frame, first special light classifier 43 performs the processes of steps S6 and S22, and also performs classification of the degree of progression by progression classifier 4e in first special light classifier 43 for the lesion classified by classification classifier 4d in first special light classifier 43. Progression classifier 4e calculates the classification result of the degree of progression and a reliability score for the classified degree of progression (step S32). Note that, if no lesion candidate region is detected by lesion classifier 4c, classification classifier 4d skips the process of step S32.
[0123] With respect to the second special light image acquired in the (n+2)th frame, second special light classifier 44 performs the processes of steps S10 and S23, and also performs classification of the degree of progression by progression classifier 4e in second special light classifier 44 for the lesion classified by classification classifier 4d in second special light classifier 44. Progression classifier 4e calculates the classification result of the degree of progression and a reliability score for the classified degree of progression (step S33). Note that, if no lesion candidate region is detected by lesion classifier 4c, classification classifier 4d skips the process of step S33.
[0124] The lesion area selection unit 45 performs the processing of steps S13 and S24, and determines the degree of progression corresponding to the discrimination result decided to be displayed in step S24 based on the reliability score, and outputs the determined degree of progression to the display processing unit 46 (step S34).
[0125] For example, when the displayed differentiation result is ulcerative colitis, the reliability score of the progression level of Mayo0 (grade 0) is 30%, the reliability score of Mayo1 (grade 1) is 60%, the reliability score of Mayo2 (grade 2) is 10%, and the reliability score of Mayo3 (grade 3) is less than 1%. In this case, the lesion area selection unit 45 determines the progression level to be, for example, “Mayo1” with the highest reliability score, and outputs it to the display processing unit 46.
[0126] The display processing unit 46 performs image synthesis to align the display area and superimpose it on the white light image (first image information), and further performs image synthesis to include the discrimination result and the progression degree near the white light image to generate display image information (step S14B). The generated display image information is output from the endoscope processor 4 to the monitor 5, and the display image is displayed on the monitor screen of the monitor 5. As a result, the discrimination result 50b and the progression degree 50c as shown in FIG. 5 are displayed.
[0127] According to the third embodiment, effects similar to those of the first and second embodiments described above are achieved, and since the degree of progression of the lesion is identified and display image information including the degree of progression is generated, it is possible to reduce variability in diagnosis of the degree of progression of the lesion due to the surgeon's subjective judgment.
[0128] In the above, the present invention has been described mainly in terms of an endoscope processor and an endoscope apparatus having an endoscope processor, but is not limited thereto. The present invention may also be a diagnostic image generating method that performs the same processing as an endoscope processor. Furthermore, the present invention may also be a computer program for causing a computer to perform the same processing as an endoscope processor, a non-transitory recording medium that can be read by a computer and that records the computer program, etc.
[0129] Furthermore, the present invention is not limited to the above-described embodiment as it is, and in the implementation stage, the components can be modified and embodied without departing from the gist of the invention. In addition, various aspects of the invention can be formed by appropriate combinations of multiple components disclosed in the above embodiments. For example, some components may be deleted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined. In this way, various modifications and applications are possible within the scope of the gist of the invention.
Claims
1. a processor, the processor comprising: detecting a first region of a lesion candidate from first image information acquired by irradiating the first illumination light; detecting a second region of the lesion candidate from second image information acquired by irradiating the subject with second illumination light having a spectrum different from that of the first illumination light; selecting a display area for the lesion candidate from among the first area and the second area according to an observation target site of the subject; generating display image information by superimposing the display area on the first image information; Endoscope processor.
2. The first illumination light is white light.
2. The endoscope processor of claim 1.
3. The processor, detecting a plurality of second regions from a plurality of pieces of second image information acquired by irradiating the plurality of types of second illumination light having different spectra; selecting the display area from the first area and the plurality of second areas; 2. The endoscope processor of claim 1.
4. The plurality of types of second illumination light include at least one of illumination light for Narrow Band Imaging (NBI) and illumination light for Red Dichromatic Imaging (RDI).
4. The endoscope processor of claim 3.
5. The processor, Calculating a plurality of confidence scores, the confidence score of the first region and the confidence score of the second region; selecting the region for display based on the plurality of confidence scores; 2. The endoscope processor of claim 1.
6. The processor, selecting the display region from among the first region and the second region, the reliability score of which is higher than a threshold; 6. The endoscope processor of claim 5.
7. The processor, When a position of the first region and a position of the second region overlap, a region having the highest reliability score among the first region and the second region is selected as the display region.
7. The endoscope processor of claim 6.
8. The processor, When a position of the first area and a position of the second area overlap, a region obtained by merging the first area and the second area is selected as the display area.
7. The endoscope processor of claim 6.
9. The processor, Calculating a plurality of confidence scores for the first region and a confidence score for each of the plurality of second regions; selecting the region for display based on the plurality of confidence scores; 4. The endoscope processor of claim 3.
10. The processor, multiplying the plurality of reliability scores by a plurality of weighting coefficients corresponding to the observation target site of the subject, respectively; selecting the display region based on the plurality of confidence scores multiplied by the plurality of weighting factors, respectively; 10. The endoscope processor of claim 9.
11. The observation target site is an organ, The processor sets the plurality of weighting coefficients according to the type of the organ. The endoscope processor of claim 10.
12. The processor automatically switches the weighting coefficients according to the observation target site. The endoscope processor of claim 10.
13. The processor, performing motion detection between the first image information and the second image information; aligning the first region and the second region in accordance with a result of the motion detection, and then selecting the display region and generating the display image information; 2. The endoscope processor of claim 1.
14. The processor, Differentiating the type of lesion in the display area; generating the display image information including the discrimination result; 2. The endoscope processor of claim 1.
15. The processor, Identifying the degree of progression of the lesion obtained as a result of the differentiation; generating the display image information including the progress level; 15. The endoscope processor of claim 14.
16. a light source device capable of emitting a plurality of types of illumination light including a first illumination light and a second illumination light having a spectrum different from that of the first illumination light; an endoscope including an imaging device that irradiates the first illumination light from the light source device to obtain first image information and irradiates the second illumination light from the light source device to obtain second image information; Detecting a first region of a suspected lesion from the first image information; Detecting a second region of a suspected lesion from the second image information; selecting a display area for the lesion candidate from among the first area and the second area according to an observation target site of the subject; an endoscope processor including a processor for generating display image information by superimposing the display area on the first image information; a monitor for displaying the display image information; An endoscope apparatus comprising:
17. A method for operating an endoscopic device including a first classifier, a second classifier, a lesion area selection unit, and a display processing unit, the first classifier detects a first region of a lesion candidate from first image information acquired by irradiating the first illumination light; the second classifier detects a second region of a lesion candidate from second image information acquired by irradiating the subject with second illumination light having a spectrum different from that of the first illumination light; the lesion area selection unit selects a display area for a lesion candidate from among the first area and the second area in accordance with an observation target site of a subject; the display processing unit generates display image information by superimposing the display area on the first image information; A method for operating an endoscopic device.
18. The second classifier detects a plurality of second regions from a plurality of second image information acquired by irradiating a plurality of types of second illumination light having different spectra from each other, the lesion area selection unit calculates a plurality of reliability scores, including a reliability score for the first area and a reliability score for each of the plurality of second areas; the lesion area selection unit multiplies the plurality of reliability scores by a plurality of weighting coefficients corresponding to the observation target site of the subject, the lesion area selection unit selects the display area from among the first area and the plurality of second areas based on the plurality of reliability scores each multiplied by the plurality of weighting coefficients; A method for operating an endoscopic device according to claim 17.
19. The observation target site is an organ, the lesion area selection unit sets the plurality of weighting coefficients according to the type of the organ.
20. A method for operating an endoscopic device according to claim 18.
20. The endoscopic device further includes a control unit, The control unit automatically switches the plurality of weighting coefficients depending on the observation target site.
20. A method for operating an endoscopic device according to claim 18.
Citation Information
Patent Citations
Image processing device and program
JP2011224038A
Image processor, endoscope apparatus, endoscope system, program, and image processing method
JP2012005512A
Endoscope device
WO2017104046A1
Medical image processing device, endoscope system, diagnosis assistance device, and medical work assistance device
WO2018221033A1
Medical image processing device and endoscope device
WO2019083019A1
