Medical image processing apparatus, operation method of the same, and program for medical image processing apparatus as well as endoscope system
The medical image processing device improves region of interest detection by classifying body parts and using tailored models with different datasets to enhance accuracy and stability, addressing issues of incorrect classification and false positives.
Patent Information
- Application Number
- JP2023080678
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-16
- Publication Date
- 2024-11-28
AI Technical Summary
Existing medical image processing systems face issues with false positives and unstable detection of regions of interest due to incorrect input of body part information, leading to incorrect classification and recognition processing.
A medical image processing device that classifies body parts into first and second parts using machine learning, employing two trained models with different datasets for accurate detection of regions of interest, one model having more data of the first region than the second, and selecting the appropriate model based on classification results or user input.
This approach stabilizes the detection of regions of interest by reducing false detections and ensuring accurate classification, even when incorrect body part information is input, by using models tailored to specific body parts.
Smart Images

Figure 2024164938000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a medical image processing apparatus, an operating method thereof, a program for the medical image processing apparatus, and an endoscope system. [Background technology]
[0002] In the medical field, when a doctor observes or diagnoses a medical image, various image processing is performed on the chronologically ordered medical images acquired by an endoscope system, an ultrasound diagnostic system, or the like. The medical image processing device described in Patent Document 1 acquires part information indicating the part inside the living body of the acquired medical image, and performs recognition processing such as detection of an attention area on the medical image. In the recognition processing, one of a plurality of trained models corresponding to the part information learned by deep learning or the like is used. These trained models are trained using an image set in which each part inside the living body is imaged according to the target part. Patent Document 1 also describes that the result of detecting the attention area is displayed on a display device in a manner corresponding to the imaged part.
[0003] The processor device (medical image processing device) described in Patent Document 2 also performs recognition processing to recognize the imaged part of a living body shown in the acquired medical image and to acquire information related to the imaged part from the medical image. A trained model trained by deep learning or the like is used for this recognition processing. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] International Publication No. 2020 / 162275 [Patent Document 2] JP 2022-137276 A Summary of the Invention [Problem to be solved by the invention]
[0005] As described in Patent Documents 1 and 2 above, a trained model used for recognition processing according to body part information is trained using an image set consisting of medical images of a target body part in a living body. If the acquired body part information is incorrect or if a user, i.e., a doctor, inputs body part information incorrectly, recognition processing will be performed using a trained model based on an image set consisting of medical images of a body part different from the actual body part. In this way, when recognition processing is performed using a trained model based on an image set of a body part different from the actual body part, there may be frequent false positives, or unstable behavior such as detecting a part that is not actually a region of interest as a region of interest, or conversely detecting a part that is actually a region of interest as not a region of interest.
[0006] The present invention aims to provide a medical image processing device, an operating method thereof, a program for a medical image processing device, and an endoscopic system that prevent erroneous detection in detection processing based on part information and enable stable detection. [Means for solving the problem]
[0007] The medical image processing device of the present invention includes a processor, which acquires medical images of a subject, classifies a part of the subject in the medical image into a first part or a second part, and for medical images in which the subject's part is classified as the first part, detects an area of interest from the medical image using a first trained model obtained by machine learning with a first dataset having data on the first part and data on the second part, and for medical images in which the subject's part is classified as the second part, detects an area of interest from the medical image using a second trained model obtained by machine learning with a second dataset having data on the second part, wherein the first dataset contains relatively more data on the first part than data on the second part.
[0008] It is preferable that the first trained model has a higher detection accuracy of the area of interest in the first region than the detection accuracy of the area of interest in the second region, and that the second trained model has a higher detection accuracy of the area of interest in the second region than the detection accuracy of the area of interest in the first region.
[0009] It is preferable that the first data set is data of a first region and is composed of images including an area of interest and images not including an area of interest, and data of a second region and is composed of images not including an area of interest.
[0010] It is preferable that the first trained model or the second trained model is a trained model trained on a first data set or a second data set, respectively, which have different data and / or combinations of data.
[0011] It is preferable that the processor analyzes the medical image to obtain a classification result indicating the part of the subject from which the medical image was captured, and selects the first trained model or the second trained model based on the classification result.
[0012] It is preferable that the processor, in response to user input, obtains a classification result indicating the part of the subject from which the medical image was captured, and selects the first trained model or the second trained model in response to the classification result.
[0013] The processor preferably controls to notify the detection result by the first trained model or the second trained model. The processor preferably controls display to display the acquired medical images in chronological order on the medical image display, and as a notification, the detection result is preferably superimposed on the medical image in which the region of interest is detected and displayed on the medical image display.
[0014] The processor is preferably configured to notify a selection status of the first trained model or the second trained model in addition to the detection result. The machine learning is preferably deep learning.
[0015] An endoscope system of the present invention includes an endoscope that captures medical images and the above-described medical image processing device.
[0016] The operating method of the medical image processing device of the present invention includes the steps of acquiring a medical image of a subject, classifying the part of the subject from which the medical image is captured as a first part or a second part, detecting an area of interest from the medical image in which the subject's part is classified as the first part using a first trained model obtained by machine learning with a first dataset having data on the first part and data on the second part, and detecting an area of interest from the medical image in which the subject's part is classified as the second part using a second trained model obtained by machine learning with a second dataset having data on the second part, wherein the first dataset contains relatively more data on the first part than data on the second part.
[0017] The program for the medical image processing device of the present invention causes a computer to execute the following processes: acquiring a medical image of a subject; classifying the part of the subject from which the medical image is captured as a first part or a second part; detecting an area of interest from the medical image using a first trained model obtained by machine learning with a first dataset having data on the first part and data on the second part for medical images in which the subject's part is classified as the first part; and detecting an area of interest from the medical image using a second trained model obtained by machine learning with a second dataset having data on the second part for medical images in which the subject's part is classified as the second part, wherein the first dataset contains relatively more data on the first part than data on the second part. Effect of the Invention
[0018] According to the present invention, erroneous detection can be prevented in detection processing according to body part information, and detection can be performed stably. [Brief description of the drawings]
[0019] [Figure 1]FIG. 1 is a schematic diagram of an endoscope system. [Diagram 2] FIG. 1 is a block diagram illustrating a configuration of a medical image processing apparatus. [Diagram 3] FIG. 2 is a block diagram showing the functions of the medical image processing apparatus. [Figure 4] FIG. 2 is an explanatory diagram for explaining a state of an examination by the endoscope system. [Diagram 5] 11 is an explanatory diagram illustrating the processing of a part classification unit and an attention area detection unit. FIG. [Figure 6] FIG. 2 is an image diagram showing a display mode of a medical image display. [Figure 7] FIG. 11 is an explanatory diagram illustrating learning by a learning unit to generate a trained model, showing (A) the case where a first dataset is input and (B) the case where a second dataset is input. [Figure 8] 4 is a flowchart showing the operation of the medical image processing apparatus. [Figure 9] 10A and 10B are diagrams illustrating a specific example of processing by a part classification unit and an attention area detection unit, in which the stomach is recognized as a part classification result (FIG. 10A) and the esophagus is recognized (FIG. 10B). [Figure 10] FIG. 11 is a block diagram showing a part of the configuration of a medical image processing apparatus according to a second embodiment. [Figure 11] FIG. 11 is an image diagram showing a display screen when a medical image processing apparatus according to the second embodiment accepts input of body part information through a user operation. [Figure 12] FIG. 11 is an explanatory diagram illustrating learning to generate a trained model in the first modified example. [Figure 13] FIG. 13 is an image diagram showing a display screen of a medical image display in a third modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0020] [First embodiment] 1, a medical image processing device 20 according to the present invention is included in an endoscope system 10. The endoscope system 10 captures an image of an observation target to obtain a medical image such as an endoscopic image.
[0021] The endoscope system 10 includes an endoscope 12, a light source device 13, a processor device 14, a display 15, a processor device-side input device 16, a medical image processing device 20, a medical image display 21, and a medical image processing device-side input device 22. The endoscope 12 is optically or electrically connected to the light source device 13, and is electrically connected to the processor device 14. The medical image processing device 20 is electrically connected to the light source device 13 and the processor device 14.
[0022] The endoscope 12 has an insertion section 12a, an operating section 12b, a bending section 12c, and a tip section 12d. The insertion section 12a is inserted into the body of the subject. The operating section 12b is provided at the base end portion of the insertion section 12a. The bending section 12c and the tip section 12d are provided on the tip side of the insertion section 12a. The bending section 12c is bent by operating an angle knob 12e of the operating section 12b. The tip section 12d is directed in a desired direction by the bending operation of the bending section 12c.
[0023] An imaging optical system for forming an image of the subject and an illumination optical system for irradiating the subject with illumination light are provided inside the endoscope 12. An angle knob 12e, a mode switch 12f, a still image acquisition instruction switch 12h, and a zoom operation unit 12i are provided in the operation unit 12b. The mode switch 12f is used to switch the observation mode. The still image acquisition instruction switch 12h is used to instruct acquisition of a still image of the subject. The zoom operation unit 12i is used to enlarge or reduce the observation target. In addition to the mode switch 12f and the still image acquisition instruction switch 12h, the operation unit 12b may also be provided with a scope side input device 19 for performing various operations on the processor device 14.
[0024] The light source device 13 generates illumination light. The processor device 14 performs system control of the endoscope system 10, and further performs image processing and the like on the image signal transmitted from the endoscope 12 to generate medical images. The processor device 14 transmits medical images to the medical image processing device 20 in addition to the display 15. The display 15 displays operation images and the like in addition to the medical images transmitted from the processor device 14. The processor device side input device 16 has a keyboard, mouse, foot switch, touch pen, and the like, and accepts input operations such as function settings by the user.
[0025] The medical image processing device 20 is connected to a medical image display 21. The medical image processing device 20 receives medical images transmitted from the processor device 14, and performs various processes based on the received medical images. The medical image processing device 20 transmits the results of various processes, in addition to the medical images, to the medical image display 21. The medical image display 21 displays the medical images transmitted from the medical image processing device 20.
[0026] The endoscope system 10 has a normal mode, an area of interest detection mode, and a part classification mode. In the normal mode, a normal color image obtained by illuminating an observation target with white light is displayed on the display 15 as a medical image. In the part classification mode, a part classification process is performed on the medical image, and a detector selection process is performed to select an area of interest detector according to the classification result. In the part classification mode, the type of classified part, etc. are displayed on the medical image display 21 as necessary. In the area of interest detection mode, an area of interest detection process is performed on the medical image, and when an area of interest is detected, the medical image display 21 displays the presence of the area of interest.
[0027] The normal mode, the area of interest detection mode, and the region classification mode can be set by the input device 22 on the medical image processing device side. In addition to sequential switching, the mode can also be set to execute two or more modes simultaneously (for example, execute both the region classification mode and the region of interest detection mode). Details of the region classification mode and the region of interest detection mode implemented by the medical image processing device 20 will be described later.
[0028] 2, the medical image processing device 20 of this embodiment has a hardware configuration of a computer in which a control unit 25, a communication unit 26, and a storage unit 27 are electrically connected to each other via a data bus 28. The medical image processing device 20 is connected to the above-mentioned medical image display 21 and a medical image processing device side input device 22. The medical image processing device 20 also includes a speaker 29 for making notifications.
[0029] The medical image processing apparatus side input device 22 is an input device having the same configuration as the above-mentioned processor apparatus side input device 16, i.e., a keyboard, a mouse, a foot switch, etc. However, without being limited thereto, the medical image processing apparatus side input device 22 may include a touch panel provided on the medical image display 21 and / or a scope side input device 19 of the endoscope 12, etc. Alternatively, the processor apparatus side input device 16 and the medical image processing apparatus side input device 22 may be configured from a common input device, and the operation of the processor apparatus 14 and the operation of the medical image processing apparatus 20 may be switched according to the purpose of use or the usage situation.
[0030] The computer constituting the medical image processing device 20 accepts various inputs of instructions from the input device 22 on the medical image processing device side. The medical image processing device 20 receives medical images transmitted from the processor device 14, and performs various processes such as body part classification processing or attention area detection processing based on the received medical images. The medical image processing device 20 transmits the results of various processes to the medical image display 21 in addition to the medical images. The medical image display 21 may display various operation screens in response to the operation of the input device 22 on the medical image processing device side in addition to the medical images acquired from the processor device 14. For example, the operation screen is provided with an operation function using a GUI (Graphical User Interface). The computer constituting the medical image processing device 20 can accept input from a user through the operation screen.
[0031] The control unit 25 includes a processor such as a CPU (Central Processing Unit) 31, a RAM (Random Access Memory) 32, and a ROM (Read Only Memory) 33. The CPU 31 loads a program stored in the storage unit 27, etc., into the RAM 32 or ROM 33, and executes processing according to the program, thereby controlling each part of the computer in an integrated manner. The communication unit 26 is a network interface that controls the transmission of various information via the network 30. The RAM 32 or ROM 33 may have the function of the storage unit 27.
[0032] The storage unit 27 is an example of a memory, and is, for example, a hard disk drive, a solid state drive, or a disk array including a plurality of hard disk drives or the like, which are built into the computer constituting the medical image processing device 20 or connected via a cable or the network 30. The storage unit 27 stores control programs, various application programs, various data to be used for these programs, and display data for various operation screens associated with these programs.
[0033] The storage unit 27 of this embodiment stores various data such as a program 34 for a medical image processing device and data 35 for a medical image processing device. The program 34 for a medical image processing device or the data 35 for a medical image processing device are programs or data for implementing various functions of the medical image processing device 20. The program 34 for a medical image processing device and the data 35 for a medical image processing device realize the functions of the medical image processing device 20. The data 35 for a medical image processing device includes, for example, a temporary storage unit 35a which is an area where data is temporarily stored, and a data storage unit 35b where program data and the like are stored.
[0034] The computer constituting the medical image processing device 20 may be a dedicated device, a general-purpose server device, a PC (Personal Computer), etc. Also, as long as it can perform the functions of the medical image processing device 20, the computer may be shared with devices performing other functions, and the functions of the medical image processing device 20 may be incorporated into an endoscope processor device and / or a medical information management device, etc. In the medical image processing device 20, a program related to medical image processing is stored in a storage unit 27, which is a program memory.
[0035] As shown in Figure 3, the medical image processing device 20 realizes the functions of a medical image acquisition unit 41, a part classification unit 42, a region of interest detection unit 43, a display control unit 44, a notification control unit 45, and a learning unit 46 by the control unit 25 operating the program 34 for the medical image processing device in the memory unit 27.
[0036] The medical image acquisition unit 41 acquires medical images of a subject by sequentially acquiring medical images from the processor device 14. In this embodiment, the medical images acquired by the medical image acquisition unit 41 from the processor device 14 are multiple medical images in chronological order. In this embodiment, as the medical images, endoscopic images obtained by the endoscope 12 are acquired, but this is not limiting, and other medical images such as ultrasound images, CT (Computed Tomography) images, and MRI (Magnetic Resonance Imaging) images may also be received.
[0037] The part classification unit 42 performs part classification processing to classify the part of the subject from which the medical image was captured by analyzing the medical image acquired by the medical image acquisition unit 41. The attention area detection unit 43 performs a detector selection processing to select an attention area detector based on the part classification processing performed by the part classification unit 42, and an attention area detection processing to detect an attention area from the medical image using the selected attention area detector. Specific contents of the part classification processing, the detector selection processing, and the attention area detection processing will be described later.
[0038] The region of interest detection process or the part classification process is preferably a process performed by a trained model obtained by deep learning using, for example, a neural network (NN), a convolutional neural network (CNN), a recurrent neural network (RNN), etc. Alternatively, the process may be a process performed by a trained model obtained by learning using other machine learning methods such as Adaboost and random forest, without being limited to deep learning.
[0039] The area of interest detected by the area of interest detection unit 43 in the area of interest detection process is, for example, an area including a lesion such as cancer, a treatment scar, a surgical scar, a bleeding area, a benign tumor, or an inflammatory area (including areas with changes such as bleeding or atrophy in addition to inflammation). That is, an area including a lesion, an area with a possibility of a lesion, or an area that requires detailed observation regardless of the possibility of a lesion, such as a dark area (behind a fold, or an area deep inside the lumen where observation light does not easily reach), may be an area of interest. The area of interest detection process detects an area including at least one of a lesion, a benign tumor, or an inflammatory area as an area of interest.
[0040] In the part classification process performed by the part classification unit 42, the part of the subject from which the medical image 50 is captured is classified into a first part or a second part. In order to recognize a plurality of parts of the subject set in advance, including the first part and the second part, the part classification unit 42 uses a trained model generated by learning the medical image as described above. Then, in the part classification process, it is determined whether the first part or the second part is included in the medical image using the trained model. In the part classification process, the parts of the subject to be examined are set according to the purpose of the examination, etc.
[0041] As shown in FIG. 4, in this embodiment, the endoscope system 10 performs an examination of the upper digestive tract 100. That is, in the case of imaging the upper digestive tract 100 by the endoscope 12, an example is shown in which the esophagus 101, stomach 102, and duodenum 103 are set as the multiple subject parts preset by the part classification unit 42. In this way, any of the multiple subject parts preset by the part classification unit 42 corresponds to the "first part" and the "second part" in the claims. However, without being limited thereto, the multiple subject parts preset by the part classification unit 42 may include at least the "first part" and the "second part", and may include three or more parts.
[0042] However, the examination performed by the endoscope system 10 is not limited to this, and may be an examination of other parts of the subject, such as an examination of the lower digestive tract. In the case of an examination of the lower digestive tract, the multiple preset subject parts are set to, for example, the large intestine and the small intestine. In addition, in the part classification process, the preset subject parts may be set to smaller parts. For example, when examining the inside of the stomach, the preset subject parts may be set to the fundus, upper part of the stomach, middle part of the stomach, lower part of the stomach, cardia, angle of the stomach, etc. In addition, when examining the inside of the large intestine, the preset subject parts may be set to the rectum, sigmoid colon, descending colon, transverse colon, etc.
[0043] As shown in FIG. 5, medical images 50 acquired by the medical image acquisition unit 41 in chronological order are input to the site classification unit 42. The site classification unit 42 performs a site classification process using the medical images 50. As described above, in this embodiment, the esophagus 101, stomach 102, and duodenum 103 are set as the sites of a plurality of subjects that have been set in advance. That is, the site classification unit 42 performs a site classification process by analyzing the medical image 50 to recognize and classify the site of the subject from which the medical image 50 is captured as being one of the esophagus 101, stomach 102, and duodenum 103. The classification result of the site classification process performed by the site classification unit 42 and the medical image 50 that has been subjected to the site classification process are input to the attention area detection unit 43.
[0044] Region-of-interest detection unit 43 includes a plurality of region-of-interest detectors, namely, esophageal lesion detector 43A, stomach lesion detector 43B, and duodenum lesion detector 43C. Esophageal lesion detector 43A, stomach lesion detector 43B, and duodenum lesion detector 43C correspond to esophagus 101, stomach 102, and duodenum 103, respectively, which are a plurality of predetermined subject parts.
[0045] The attention area detection unit 43 acquires the classification result by the part classification unit 42, and selects one of the esophageal lesion detector 43A, the stomach lesion detector 43B, and the duodenum lesion detector 43C according to the classification result. That is, when the classification result of the medical image 50 by the part classification unit 42 is the esophagus 101, the attention area detection unit 43 selects the esophageal lesion detector 43A, and similarly, when the classification result is the stomach 102, the attention area detection unit 43 selects the stomach lesion detector 43B, and similarly, when the classification result is the duodenum 103, the attention area detection unit 43 selects the duodenum lesion detector 43C. In this way, when the part of the subject is classified as the first part, the attention area detector selected by the attention area detection unit 43 corresponds to the "first learned model". Similarly, when the part of the subject is classified as the second part, the attention area detector selected by the attention area detection unit 43 corresponds to the "second learned model".
[0046] The area-of-interest detection unit 43 performs an area-of-interest detection process to detect an area of interest from the medical image 50 using one of the esophageal lesion detector 43A, the stomach lesion detector 43B, and the duodenum lesion detector 43C selected by the detector selection process. Information on the detected area of interest and the medical image 50 in which the area of interest has been detected are input to the display control unit 44. The display control unit 44 performs display control to display the medical image 50 on the medical image display 21 in chronological order. Furthermore, when an area of interest is detected, the display control unit 44 notifies the detection result.
[0047] The display control unit 44 controls the display of the medical images 50, which have been acquired in chronological order and have been subjected to the site classification process and the region-of-interest detection process, on the medical image display 21, and also controls the notification of the detection results of the region of interest. It is preferable that the detection results are notified by the display control unit 44 during the examination. Specifically, when the medical images 50 in chronological order are displayed on the medical image display 21, the notification control unit 45 displays the detection results of the region of interest on the medical image display 21 by superimposing them on the medical image 50 in which the region of interest has been detected.
[0048] For example, as shown in Fig. 6, when an attention area 55 is detected from a medical image 50 by the attention area detection unit 43, the display control unit 44 preferably highlights an outer frame 51A of a main screen 51 that displays the medical image 50 in real time as a moving image during examination, in a color for notifying the detection result. In addition, it is preferable that the color for notifying the detection result is different from the color when an attention area is not detected. For example, when the color when an attention area is not detected is yellow, it is preferable that the color for notifying the detection result is blue, which is complementary to yellow.
[0049] Furthermore, the notification of the detection results by the display control unit 44 is not limited to these, and the portion including the area of interest 55 in the medical image 50 displayed on the main screen 51 may be highlighted in a color different from the actual color (a color with a different shade, brightness, saturation, etc.), or a figure may be superimposed on the portion including the area of interest 55 or around the area of interest 55.
[0050] The notification control unit 45 controls notification of the detection result of the region of interest using a method different from the display on the medical image display 21. For example, the notification control unit 45 controls the speaker 29 to play a notification sound from the speaker 29 when the region of interest detection unit 43 detects a region of interest from the medical image 50. Also, for example, the notification sound is played from the speaker 29 in accordance with the highlighted display for notifying the detection result by the display control unit 44 described above.
[0051] The learning unit 46 generates the above-mentioned esophageal lesion detector 43A, stomach lesion detector 43B, and duodenal lesion detector 43C by deep learning. The esophageal lesion detector 43A, stomach lesion detector 43B, and duodenal lesion detector 43C are trained models obtained by deep learning performed by the learning unit 46. Note that the configuration for generating the attention area detector included in the attention area detection unit 43 by deep learning is not limited to this, and a device provided outside the medical image processing device 20 may perform deep learning similar to that of the learning unit 46, obtain the generated trained model, and use it as the attention area detector. For example, an external cloud server connected via the network 30 may be made to execute a function similar to that of the learning unit 46, and the medical image processing device 20 may obtain a trained model deep-learned by the learning unit on this cloud server, and use it as a plurality of attention area detectors included in the attention area detection unit 43.
[0052] As shown in FIG. 7(A) and FIG. 7(B), the deep learning by the learning unit 46 is performed by inputting a first data set DS1 or a second data set DS2 having a plurality of medical images. The first trained model 43X1 generated by the learning unit 46 and used as any one of the esophageal lesion detector 43A, the stomach lesion detector 43B, and the duodenum lesion detector 43C is a trained model obtained by the above-mentioned deep learning with the first data set DS1. The first data set DS1 has data D1 of a plurality of first regions and data D2 of a plurality of second regions. Specifically, the data D1 is data of a medical image obtained by imaging the first region, and the data D2 is data of a medical image obtained by imaging the second region.
[0053] The "first region" and the "second region" change depending on whether the first trained model corresponds to esophageal lesion detector 43A, stomach lesion detector 43B, or duodenum lesion detector 43C. Specifically, when the first trained model corresponds to esophageal lesion detector 43A, the "first region" is the region corresponding to esophageal lesion detector 43A, i.e., the esophagus, and the "second region" is the region of a plurality of subjects set in advance excluding the first region, the stomach, or the duodenum.
[0054] As shown in FIG. 7B, the second trained model 43X2 is a trained model obtained by the above-mentioned deep learning with the second data set DS2. The second data set DS2 has a plurality of second site data D2. As in the above-mentioned specific example, when the first site is the esophagus and the second site is the stomach, the first site data D1 is data of a medical image of the esophagus, and the second site data D2 is data of a medical image of the stomach.
[0055] In the above-mentioned specific example, the second region may be the duodenum. In this case, the data D2 of the second region is data of medical images of the duodenum, and the second data set DS2 includes a plurality of data D2 of the second region, i.e., data of medical images of the duodenum.
[0056] In addition, when the first trained model is the gastric lesion detector 43B, the "first site" is the site corresponding to the gastric lesion detector 43B, i.e., the stomach, and the "second site" is the site of a plurality of subjects set in advance, excluding the first site, the esophagus, or the duodenum. In addition, when the first trained model is the duodenal lesion detector 43C, the "first site" is the duodenum, and the "second site" is the site of a plurality of subjects set in advance, excluding the first site, the stomach, or the esophagus. In the above case, as in the above specific example, the learning unit 46 deep-learns the first data set DS1 having the data D1 of the first site and the data D2 of the second site, and the second data set DS2 having the data D2 of the second site, respectively, to obtain the first trained model and the second trained model.
[0057] As described above, the first dataset DS1 or the second dataset DS2 is input to the learning unit 46, and deep learning is performed to obtain the first trained model or the second trained model, that is, the esophageal lesion detector 43A, the stomach lesion detector 43B, and the duodenum lesion detector 43C.
[0058] The first data set DS1 includes a relatively large amount of data D1 of the first region compared to data D2 of the second region. For example, when the first region is the esophagus, the first data set DS1 includes a relatively large amount of data D1 of the first region, which is the data of the esophagus, compared to data D2 of the second region, which is the data of the stomach or the duodenum.
[0059] The esophageal lesion detector 43A, the stomach lesion detector 43B, and the duodenum lesion detector 43C are preferably first or second trained models trained with the first data set DS1 or the second data set DS2 having different data D1, D2, and / or different combinations of data D1 and D2, respectively. This generates a first or second trained model optimal for the esophageal lesion detector 43A, the stomach lesion detector 43B, and the duodenum lesion detector 43C, respectively.
[0060] Next, the flow of processing by the medical image processing device 20 when an examination is performed using the endoscope system 10 will be described with reference to the flowchart shown in Fig. 8 and the explanatory diagram shown in Fig. 9. The examples shown in Figs. 8 and 9 show the operation of the endoscope system 10 when two modes, an attention area detection mode and a part classification mode, are executed simultaneously. A doctor, who is a user, turns on the power of the endoscope system 10 and observes the inside of the subject. Then, the endoscope system 10 sequentially captures endoscopic images, which are medical images 50.
[0061] In the medical image processing device 20, first, the medical image acquisition unit 41 acquires a plurality of medical images from the processor device 14 in chronological order (step ST110). Next, the part classification unit 42 performs part classification processing of the acquired medical image 50 (step ST120). In this embodiment, the part classification unit 42 performs part classification processing to classify the part of the subject from which the medical image 50 was captured into a first part or a second part. Specifically, the part classification processing is performed to recognize and classify the part as being one of the esophagus 101, stomach 102, and duodenum 103.
[0062] Next, the area of interest detection unit 43 uses the classification result by the part classification unit 42 to perform a detector selection process to select one of the esophageal lesion detector 43A, the stomach lesion detector 43B, and the duodenum lesion detector 43C, i.e., the first learned model or the second learned model (step ST130), and then performs an area of interest detection process to detect an area of interest from the medical image 50 using one of the esophageal lesion detector 43A, the stomach lesion detector 43B, and the duodenum lesion detector 43C selected by the detector selection process, i.e., the first learned model or the second learned model (step ST140).
[0063] Based on the detection result of the area of interest by the area of interest detection unit 43 and the medical image 50, the display control unit 44 and the notification control unit 45 notify the detection result of the area of interest (step ST150). If the user wants to continue the examination (Y in step ST160), the process returns to acquisition of the medical image 50 by the medical image acquisition unit 41 (step ST110) and repeats the above process. If the user wants to end the examination (N in step ST160), the power of the endoscope system 10 is turned off.
[0064] As described above, when an examination is performed using the endoscope system 10, the medical image 50 may be classified into the second region in the region classification process performed by the region classification unit 42, even though the medical image 50 is actually acquired by imaging the first region. For example, the region classification process performed by the region classification unit 42 may give an erroneous classification result for the medical image 50 captured near the boundary between the regions of the subject. Alternatively, the region classification process performed by the region classification unit 42 may give an erroneous classification result due to a change in the shape of the lumen due to peristalsis of the digestive tract, which is the subject, water splashing when liquid or the like is sprayed on the subject, or the influence of defocusing of the imaging optical system of the endoscope 12. In the example shown in FIG. 4, the medical images 50 captured near the boundary 104 between the esophagus 101 and the stomach 102 and near the boundary 105 between the stomach 102 and the duodenum are included.
[0065] 9(A), when the classification result is stomach 102, region of interest detection unit 43 selects stomach lesion detector 43B and detects region of interest 55 in medical image 50. On the other hand, when the classification result is esophagus 101, region of interest detection unit 43 selects esophageal lesion detector 43A and detects region of interest 55 in medical image 50, as shown in FIG.
[0066] When medical image 50 is captured near boundaries 104 and 105 as described above, for example, there is a possibility that site classification section 42 will give a classification result indicating esophagus 101, even though medical image 50 is actually acquired by capturing an image of stomach 102. In this case, attention area detection section 43 will select esophageal lesion detector 43A based on the erroneous classification result.
[0067] However, in this embodiment, since the first data set DS1 having the data D1 of the first region and the data D2 of the second region is input to the learning unit 46, the data D2 of the second region, i.e., data of the stomach, is also included in the first data set DS1 with which the esophageal lesion detector 43A is trained. Therefore, in reality, the first trained model trained with the first data set DS1 including the data of the stomach is used to detect the medical image 50 acquired by imaging the stomach, so there is little false detection by the attention area detection unit 43. In addition, the attention area detection unit 43 rarely detects a portion that is not actually an attention area as an attention area, and rarely detects a portion that is actually an attention area as not an attention area, so that the attention area can be detected with stable behavior.
[0068] On the other hand, when the stomach 102 is imaged to obtain the medical image 50, the site classification unit 42 almost always correctly classifies the medical image 50 as the stomach 102. In this case, the region of interest detection unit 43 selects the gastric lesion detector 43B based on the correct classification result, and detects the region of interest 55 in the medical image 50. As in the above-mentioned specific example, when the first region is the esophagus and the second region is the stomach, the second data set DS2 with which the gastric lesion detector 43B is trained contains data on the medical image of the stomach, which is the second region. Therefore, there is little false detection by the region of interest detection unit 43.
[0069] In addition, in this embodiment, the first data set DS1 for performing deep learning contains a relatively large amount of data D1 of the first region compared to data D2 of the second region. For example, when the first region is the esophagus, the data D1 of the first region is relatively large compared to data D2 of the stomach or duodenum of the second region. The first trained model thus generated becomes the esophageal lesion detector 43A. As a result, when the classification result is the esophagus 101, the detection of the region of interest is performed by the esophageal lesion detector 43A trained with the first data set DS1 containing a large amount of esophageal data, so that the region of interest detection unit 43 can detect the region of interest with even higher accuracy, and the region of interest can be detected with stable behavior.
[0070] [Second embodiment] In the first embodiment, the part classification unit 42 analyzes the medical image to obtain a classification result indicating the part of the subject from which the medical image 50 was captured, but the present invention is not limited to this. As shown in FIG. 10, a classification result indicating the part of the subject from which the medical image 50 was captured may be obtained in response to a user's input. In this case, for example, the part classification unit 42 accepts part information S input by the user operating the medical image processing device side input device 22. Then, the part classification unit 42 uses the part information S input by the user as the classification result. Note that the configuration of the medical image processing device according to this embodiment is the same as that of the medical image processing device 20 according to the first embodiment, except for the part classification unit 42, and therefore a description thereof will be omitted.
[0071] When inputting the site information S, for example, as shown in FIG. 11, a medical image 50 is displayed in real time on the medical image display 21 as a moving image during the examination, and a schematic diagram PD showing the site of the subject is also displayed. In the example shown in FIG. 11, the esophagus, stomach, and duodenum are set as the site of the subject, as in the first embodiment. While observing the medical image 50, the user operates the medical image processing device side input device 22 to align the position of the pointer 56 with the position of the site (any one of the esophagus, stomach, and duodenum) in the schematic diagram PD. This allows the user to input the site information S.
[0072] In the process flow of the medical image processing device according to this embodiment, in the site classification process in the first embodiment (step ST120; see FIG. 8), a doctor, who is a user, operates the input device 22 on the medical image processing device to input site information. Then, using the site information input by the user as a classification result, a detector selection process is performed to select one of the esophageal lesion detector 43A, the stomach lesion detector 43B, and the duodenum lesion detector 43C (step ST130; see FIG. 8). The subsequent process is the same as in the first embodiment.
[0073] As described above, when an examination is performed using an endoscope system including a medical image processing device according to this embodiment, the medical image 50 may be classified as a second part due to incorrect part information input by the user, even though the medical image 50 is actually acquired by capturing an image of a first part. In particular, it is difficult to judge a medical image 50 captured near the boundary between the subject's parts, and an incorrect classification result may be obtained. Or, the user may forget to operate during the examination, making it impossible to switch to an appropriate classification result.
[0074] However, as in the first embodiment, the first data set DS1 having the data D1 of the first region and the data D2 of the second region is input to the learning unit 46, and the data D2 of the second region is also included in the first data set DS1 in which the first trained model is trained. Therefore, if the region information input by the user is incorrect and classified as the second region, for example, if the medical image 50 actually acquired by imaging the stomach is classified as the esophagus by the user's input, the detection is performed using the trained model trained on the first data set DS1 including stomach data, so there is little erroneous detection by the attention area detection unit 43. Also, as in the first embodiment, the attention area detection unit 43 can detect the attention area with stable behavior.
[0075] [First Modification] In each of the above embodiments, the data set for deep learning is not limited as to whether or not the data includes an area of interest. In the modified example shown in FIG. 12, the first data set DS1 is data D1 of the first region, and is composed of an image D11 including an area of interest and an image D12 not including an area of interest, and data D2 of the second region, and is composed of an image D21 not including an area of interest. As a result, when the classification result by the region classification unit 42 is the first region, the detection of the area of interest is performed by the first trained model 43X1 trained with the first data set DS1 composed of the image D11 including an area of interest as data of the first region. This makes it possible for the area of interest detection unit 43 to detect the area of interest with even higher accuracy, and the area of interest can be detected with stable behavior. In particular, when a medical image 50 of the second region is input to the first trained model, erroneous detection of the area of interest can be suppressed.
[0076] [Second modified example] In the above-described respective embodiments and the first modified example, in order to adjust the detection accuracy by the first trained model and the second trained model, the number and contents of the data D1 and D2 are made different in the first data set DS1 and the second data set DS2 for training the first trained model and the second trained model, but the present invention is not limited thereto, and adjustment may be made by making the weighting of each data D1 and D2 different during training by the training unit 46. In this case, by making the weighting of each data D1 and D2 different, it is preferable that the first trained model has a higher detection accuracy of the attention area in the first part than the detection accuracy of the attention area in the second part, and the second trained model has a higher detection accuracy of the attention area in the second part than the detection accuracy of the attention area in the first part.
[0077] In this specification, high detection accuracy preferably means high sensitivity and / or high specificity. High sensitivity means that there is a high possibility that an object that should be determined as an area of interest will be correctly determined as an area of interest, and high specificity means that there is a low possibility that an area of interest will be determined to exist when the object is normal (there is no area of interest).
[0078] [Third Modification] In each of the above embodiments, when the attention area detection unit 43 detects the attention area 55 from the medical image 50, the display control unit 44 displays the detection result of the attention area 55 on the medical image display 21 by superimposing it on the medical image 50 in which the attention area is detected, but the present invention is not limited to this, and as shown in Fig. 13, a message MS1 indicating the detection of the attention area 55 may be superimposed and displayed on the medical image display 21. Furthermore, as the message MS1 indicating the detection of the attention area 55, the selection status of the first trained model or the second trained model selected by the attention area detection unit 43 may be displayed.
[0079] 13, a message MS1 "Stomach" is displayed to indicate that stomach lesion detector 43B is selected as the first trained model or the second trained model selected by attention area detection unit 43. Note that the display mode for displaying the selection status of the first trained model or the second trained model selected by attention area detection unit 43 is not limited to this, and as shown in FIG. 11, a schematic diagram PD corresponding to a part of the subject may be displayed on medical image display 21 as the selection status of the first trained model or the second trained model.
[0080] Furthermore, in addition to the display form of displaying the detection result of the attention area 55 as in each of the above embodiments, the display control unit 44 may also display a message indicating the selection status of the first trained model or the second trained model as in the modified example shown in Fig. 13. This allows the user to compare the selection status of the first trained model or the second trained model with the medical image 50 displayed on the medical image display 21 and check whether the first trained model or the second trained model has been correctly selected.
[0081] In each of the above embodiments, the hardware structure of the processing unit that executes various processes, such as the medical image acquisition unit, the part classification unit, the region of interest detection unit, the display control unit, the notification control unit, and the learning unit, is various processors as shown below. The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) and functions as various processing units, a GPU (Graphical Processing Unit), a programmable logic device (PLD), which is a processor whose circuit configuration can be changed after manufacture such as an FPGA (Field Programmable Gate Array), and a dedicated electric circuit, which is a processor having a circuit configuration designed exclusively for executing various processes.
[0082] One processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same type or different types (for example, multiple FPGAs, a combination of a CPU and an FPGA, or a combination of a CPU and a GPU, etc.). Also, multiple processing units may be configured with one processor. As an example of configuring multiple processing units with one processor, first, as represented by a computer such as a client or a server, there is a form in which one processor is configured with a combination of one or more CPUs and software, and this processor functions as multiple processing units. Second, as represented by a system on chip (SoC), there is a form in which a processor is used that realizes the functions of the entire system including multiple processing units with one IC (Integrated Circuit) chip. In this way, the various processing units are configured using one or more of the above various processors as a hardware structure.
[0083] Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit (circuitry) in the form of a combination of circuit elements such as semiconductor elements, and the hardware structure of the memory unit is a storage device such as a hard disk drive (HDD) or a solid state drive (SSD).
[0084] [Additional note 1] A processor is provided. The processor, Acquire a medical image of the subject; classifying a region of a subject of the medical image into a first region or a second region; For the medical image in which the part of the subject is classified as the first part, a first trained model obtained by machine learning on a first dataset having data of the first part and data of the second part is used to detect an area of interest from the medical image; For the medical image in which the part of the subject is classified as the second part, detecting an area of interest from the medical image using a second trained model obtained by machine learning on a second dataset having data on the second part; A medical image processing device, wherein the first data set includes a relatively larger amount of data about the first region than data about the second region. [Additional note 2] The first trained model has a higher detection accuracy of a region of interest in the first part than a detection accuracy of a region of interest in the second part; The medical image processing device according to claim 1 , wherein the second trained model has a higher detection accuracy of a region of interest in the second region than a detection accuracy of a region of interest in the first region. [Additional note 3] 3. The medical image processing device according to claim 1, wherein the first data set is data of the first region and is composed of images including an area of interest and images not including an area of interest, and data of a second region and is composed of images not including an area of interest. [Additional note 4] The medical image processing device according to any one of claims 1 to 3, wherein the first trained model or the second trained model is a trained model trained with a first data set or a second data set having different data and / or combinations of data, respectively. [Additional note 5] 5. The medical image processing device according to claim 1, wherein the processor analyzes a medical image to obtain a classification result indicating the part of the subject from which the medical image was captured, and selects the first trained model or the second trained model according to the classification result. [Additional note 6] 6. The medical image processing device according to claim 1, wherein the processor obtains a classification result indicating the part of the subject from which the medical image was captured in response to a user input, and selects the first trained model or the second trained model in response to the classification result. [Additional note 7] 7. The medical image processing device according to claim 1, wherein the processor performs control to notify a detection result based on the first trained model or the second trained model. [Additional note 8] The processor performs display control to display the acquired medical images in chronological order on a medical image display; 8. The medical image processing apparatus according to claim 7, wherein, as the notification, the detection result is superimposed on the medical image in which the region of interest is detected and displayed on the medical image display. [Additional note 9] The medical image processing device according to claim 7 , wherein the processor notifies a selection status of the first trained model or the second trained model in addition to the detection result. [Additional Note 10] The medical image processing device according to claim 1 , wherein the machine learning is deep learning. [Additional Note 11] an endoscope for capturing the medical image; 11. An endoscope system comprising the medical image processing device according to claim 1. [Additional Note 12] acquiring a medical image of a subject; classifying a region of a subject from which the medical image was captured into a first region or a second region; For the medical image in which the part of the subject is classified as a first part, detecting an area of interest from the medical image using a first trained model obtained by machine learning with a first dataset having data of the first part and data of the second part; A method for operating a medical image processing device, comprising: for the medical image in which the subject's part is classified as the second part, detecting an area of interest from the medical image using a second trained model obtained by machine learning with a second dataset having data on the second part, wherein the first dataset contains relatively more data on the first part than data on the second part. [Additional Note 13] A program for a medical image processing device, A process of acquiring a medical image of a subject; A process of classifying a part of a subject from which the medical image is captured into a first part or a second part; For the medical image in which the part of the subject is classified as a first part, a process of detecting an area of interest from the medical image using a first trained model obtained by machine learning with a first dataset having data of the first part and data of the second part; A program for a medical image processing device that causes a computer to execute a process of detecting an area of interest from the medical image in which the subject's part is classified as the second part using a second trained model obtained by machine learning with a second dataset having data on the second part, wherein the first dataset contains relatively more data on the first part than the data on the second part. [Explanation of symbols]
[0085] 10 Endoscope System 12 Endoscopy 12a Insertion part 12b Operation section 12c Curved section 12d Tip 12e Angle knob 12f Mode Switch 12h Still image acquisition switch 12i Zoom control section 13 Light source device 14 Processor unit 15 Display 16 Processor side input device 19 Scope side input device 20 Medical image processing device 21 Medical image display 22 Medical image processing device input device 25 Control Unit 26 Communications Department 27 Memory section 28 Data Bus 29 Speaker 30 Network 31 CPU(Central Processing Unit) 32 RAM (Random Access Memory) 33 ROM (Read Only Memory) 34 Program for medical image processing device 35 Data for medical image processing equipment 35a Temporary storage section 35b Data storage section 41 Medical Image Acquisition Department 42 Part classification section 43 Area of interest detection section 43A Esophageal Lesion Detector 43B Gastric Lesion Detector 43C Duodenal Lesion Detector 43X1 First trained model 43X2 Second trained model 44 Display control section 45 Notification control section 46 Learning Department 50 Medical Imaging 51 Main Screen 51A Outer frame 55 Areas of Interest 56 Pointer 100 Upper gastrointestinal tract 101 Esophagus 102 Stomach 103 Duodenum 104, 105 boundaries D1, D2 data D11 Image containing a region of interest D12 Image without a region of interest D21 Image without a region of interest DS1 First Data Set DS2 Second Data Set MS1 Messages PD schematic diagram
Claims
1. A processor is provided. The processor, Acquire a medical image of the subject; classifying a region of a subject of the medical image into a first region or a second region; For the medical image in which the part of the subject is classified as the first part, a first trained model obtained by machine learning on a first dataset having data of the first part and data of the second part is used to detect an area of interest from the medical image; For the medical image in which the part of the subject is classified as the second part, detecting an area of interest from the medical image using a second trained model obtained by machine learning on a second dataset having data on the second part; A medical image processing device, wherein the first data set includes a relatively large amount of data about the first region compared to data about the second region.
2. The first trained model has a higher detection accuracy of a region of interest in the first part than a detection accuracy of a region of interest in the second part; The medical image processing device according to claim 1 , wherein the second trained model has a higher detection accuracy of a region of interest in the second region than a detection accuracy of a region of interest in the first region.
3. The medical image processing device according to claim 1 , wherein the first data set is data of the first region and is composed of images including a region of interest and images not including a region of interest, and data of a second region and is composed of images not including a region of interest.
4. The medical image processing device according to claim 1 , wherein the first trained model or the second trained model is a trained model trained with a first data set or a second data set having different data and / or combinations of data, respectively.
5. The medical image processing device according to claim 1 , wherein the processor analyzes a medical image to obtain a classification result indicating the part of the subject from which the medical image was captured, and selects the first trained model or the second trained model according to the classification result.
6. The medical image processing device according to claim 1 , wherein the processor obtains a classification result indicating the part of the subject from which the medical image was captured in response to a user input, and selects the first trained model or the second trained model in response to the classification result.
7. The medical image processing device according to claim 1 , wherein the processor performs control to notify a detection result based on the first trained model or the second trained model.
8. The processor performs display control to display the acquired medical images in chronological order on a medical image display; The medical image processing apparatus according to claim 7 , wherein, as the notification, the detection result is superimposed on the medical image in which the region of interest is detected and displayed on the medical image display.
9. The medical image processing device according to claim 7 , wherein the processor notifies a selection status of the first trained model or the second trained model in addition to the detection result.
10. The medical image processing device according to claim 1 , wherein the machine learning is deep learning.
11. an endoscope for capturing the medical image; An endoscope system comprising the medical image processing device according to any one of claims 1 to 10.
12. acquiring a medical image of a subject; classifying a region of a subject from which the medical image is captured into a first region or a second region; For the medical image in which the part of the subject is classified as a first part, detecting an area of interest from the medical image using a first trained model obtained by machine learning on a first dataset having data of the first part and data of the second part; A method for operating a medical image processing device, comprising: for the medical image in which the subject's part is classified as the second part, detecting an area of interest from the medical image using a second trained model obtained by machine learning on a second dataset having data on the second part, wherein the first dataset contains relatively more data on the first part than data on the second part.
13. A program for a medical image processing device, A process of acquiring a medical image of a subject; A process of classifying a part of a subject from which the medical image is captured into a first part or a second part; For the medical image in which the part of the subject is classified as a first part, a process of detecting an area of interest from the medical image using a first trained model obtained by machine learning with a first dataset having data of the first part and data of the second part; A program for a medical image processing device that causes a computer to execute a process of detecting an area of interest from the medical image in which the subject's part is classified as the second part using a second trained model obtained by machine learning on a second dataset having data on the second part, wherein the first dataset contains relatively more data on the first part than the data on the second part.
Citation Information
Patent Citations
Endoscopy-related business system
JP2022137276A
Medical image processing device, endoscope system, and medical image processing method
WO2020162275A1
