Image processing device, image processing method, and image processing program
The image processing device accurately detects abnormalities in medical images by extracting and analyzing target organ shape information using trained models, addressing accuracy issues with special or incomplete organ shapes.
Patent Information
- Application Number
- JP2023085372
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-24
- Publication Date
- 2024-12-05
- Estimated Expiration
- 2043-05-24
AI Technical Summary
Existing methods for abnormality detection in medical images face accuracy issues when the shape of the target organ is incorrectly extracted or is a special shape, such as parts missing due to surgical resection.
An image processing device and method that extracts shape information of a target organ using trained models like CNNs, derives shape normality, and performs abnormality detection based on this information, displaying results accurately.
Enables high-accuracy detection of abnormalities in medical images by correctly handling variations in target organ shapes, including special shapes.
Smart Images

Figure 2024168571000001_ABST
Abstract
Description
[Technical field]
[0001] The present disclosure relates to an image processing device, an image processing method, and an image processing program. [Background technology]
[0002] Patent Document 1 discloses a technique for extracting a blood vessel region from a medical image, dividing the blood vessel region into blood vessel segments, and measuring the three-dimensional shape of each blood vessel segment to generate shape data. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] Republished Patent No. 2017 / 047819 Summary of the Invention [Problem to be solved by the invention]
[0004] When performing anomaly detection processing on medical images, it is possible to detect anomalies with high accuracy by using shape information of the target organ. However, if the shape of the target organ cannot be extracted correctly, or if the shape of the target organ is an unusual shape even if it can be extracted correctly, the accuracy of anomaly detection may decrease.
[0005] The present disclosure has been made in consideration of the above circumstances, and aims to provide an image processing device, an image processing method, and an image processing program capable of detecting abnormalities in medical images with high accuracy. [Means for solving the problem]
[0006] The image processing device of the first aspect is an image processing device having at least one processor, which extracts shape information of a target organ based on a medical image, derives a shape normality of the target organ based on the shape information, and performs an abnormality detection process on the medical image based on the medical image, the shape information, and the shape normality.
[0007] The image processing device of the second aspect is the image processing device of the first aspect, wherein the shape normality is a numerical value that is equal to or greater than a lower limit and equal to or less than an upper limit, which indicates the normality of the shape of the target organ.
[0008] In the image processing device of a third aspect, in the image processing device of the first aspect, the shape normality is classification information representing a classification of a result of extraction of the shape information.
[0009] An image processing device of a fourth aspect is an image processing device of any one of the first to third aspects, wherein the processor is a trained model that receives medical images, shape information, and shape normality as input and outputs abnormality information related to abnormalities in a target organ, and performs an abnormality detection process by inputting a medical image to be diagnosed, the shape information of the medical image, and the shape normality to the trained model that has been trained using multiple sets of medical images, shape information, shape normality, and abnormality information as training data.
[0010] The image processing device of the fifth aspect is an image processing device of any one of the first to third aspects, in which the processor is a trained model that receives medical images and shape information as input and outputs abnormality information related to abnormalities in a target organ, and performs an abnormality detection process by inputting the medical image of the diagnosis target and the shape information of the medical image to a trained model corresponding to the shape normality derived for the medical image of the diagnosis target, among a plurality of trained models trained for each shape normality using a plurality of sets of medical images, shape information, and abnormality information as training data.
[0011] In an image processing device of a sixth aspect, in an image processing device of any one of the first to fifth aspects, the processor performs control to display that abnormality detection processing cannot be performed when the shape normality indicates other than that the shape of the target organ is normal.
[0012] An image processing device of a seventh aspect is the image processing device of any one of the first to sixth aspects, wherein the processor performs control to display a result of execution of the abnormality detection process and a degree of shape normality.
[0013] The image processing method of the eighth aspect is performed by a processor provided in the image processing device to extract shape information of a target organ based on a medical image, derive shape normality of the target organ based on the shape information, and perform abnormality detection processing on the medical image based on the medical image, the shape information, and the shape normality.
[0014] The image processing program of the ninth aspect is intended to cause a processor provided in an image processing device to execute a process of extracting shape information of a target organ based on a medical image, deriving shape normality of the target organ based on the shape information, and performing abnormality detection processing on the medical image based on the medical image, the shape information, and the shape normality. Effect of the Invention
[0015] According to the present disclosure, abnormalities can be detected from medical images with high accuracy. [Brief description of the drawings]
[0016] [Figure 1] FIG. 1 is a block diagram showing a schematic configuration of a medical information system. [Diagram 2] FIG. 2 is a block diagram showing an example of a hardware configuration of the image processing device. [Diagram 3] FIG. 2 is a block diagram showing an example of a functional configuration of the image processing device. [Figure 4] FIG. 1 is a diagram for explaining a first trained model. [Diagram 5]FIG. 13 is a diagram for explaining a second trained model. [Figure 6] FIG. 13 is a diagram for explaining a third trained model. [Figure 7] FIG. 4 is a diagram showing an example of a display screen. [Figure 8] FIG. 4 is a diagram showing an example of a display screen. [Figure 9] FIG. 4 is a diagram showing an example of a display screen. [Figure 10] FIG. 4 is a diagram showing an example of a display screen. [Figure 11] 13 is a flowchart illustrating an example of a diagnosis support process. [Figure 12] FIG. 13 is a diagram for explaining a second trained model according to a modified example. [Figure 13] FIG. 11 is a diagram for explaining a third trained model according to a modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0017] Hereinafter, examples of embodiments for carrying out the technology of the present disclosure will be described in detail with reference to the drawings.
[0018] First, the configuration of a medical information system 1 according to this embodiment will be described with reference to Fig. 1. As shown in Fig. 1, the medical information system 1 includes an image processing device 10, an imaging device 12, and an image storage server 14. The image processing device 10, the imaging device 12, and the image storage server 14 are connected in a state in which they can communicate with each other via a wired or wireless network 18. The image processing device 10 is, for example, a computer such as a personal computer or a server computer.
[0019] The imaging device 12 is a device that generates a medical image representing a diagnostic target part of a subject by imaging the diagnostic target part. Examples of the imaging device 12 include a plain X-ray imaging device, an endoscope device, a CT (Computed Tomography) device, an MRI (Magnetic Resonance Imaging) device, and a PET (Positron Emission Tomography) device. In this embodiment, an example will be described in which the imaging device 12 is a CT device and the diagnostic target part is the abdomen. That is, the imaging device 12 according to this embodiment generates a CT image of the abdomen of the subject as a three-dimensional medical image consisting of multiple tomographic images. The medical image generated by the imaging device 12 is transmitted to the image storage server 14 via the network 18 and is stored by the image storage server 14.
[0020] The image storage server 14 is a computer that stores and manages various data, and is equipped with a large-capacity external storage device and database management software. The image storage server 14 receives medical images generated by the imaging device 12 via the network 18, and stores and manages the received medical images. The storage format of image data by the image storage server 14 and communication with other devices via the network 18 are based on protocols such as DICOM (Digital Imaging and Communication in Medicine).
[0021] Next, a hardware configuration of the image processing device 10 according to the present embodiment will be described with reference to Fig. 2. As shown in Fig. 2, the image processing device 10 includes a CPU (Central Processing Unit) 20, a memory 21 as a temporary storage area, and a non-volatile storage unit 22. The image processing device 10 also includes a display 23 such as a liquid crystal display, an input device 24 such as a keyboard and a mouse, and a network I / F (InterFace) 25 connected to a network 18. The CPU 20, the memory 21, the storage unit 22, the display 23, the input device 24, and the network I / F 25 are connected to a bus 27. The CPU 20 is an example of a processor according to the disclosed technology.
[0022] The storage unit 22 is realized by a hard disk drive (HDD), a solid state drive (SSD), a flash memory, or the like. The storage unit 22 as a storage medium stores an image processing program 30. The CPU 20 reads the image processing program 30 from the storage unit 22, expands it in the memory 21, and executes the expanded image processing program 30.
[0023] The storage unit 22 also stores a trained model 32, a trained model 34, and a trained model 36. The trained model 32, the trained model 34, and the trained model 36 will be described in detail later.
[0024] However, when extracting a target organ from a medical image, the shape of the target organ may not be extracted correctly, or may be an unusual shape even if it is extracted correctly. In this case, when an abnormality detection process is performed on the target organ, the accuracy of abnormality detection may decrease. An example of a case where the shape of the target organ is an unusual shape is a case where a part of the target organ is missing due to surgical resection.
[0025] Therefore, the image processing device 10 according to the present embodiment has a function of suppressing a decrease in accuracy of abnormality detection based on the result of extraction of the target organ from the medical image. In the following, an example in which the pancreas is applied as the target organ will be described.
[0026] Next, a functional configuration of the image processing device 10 according to this embodiment will be described with reference to Fig. 3. As shown in Fig. 3, the image processing device 10 includes an acquisition unit 40, an extraction unit 42, a derivation unit 44, a detection unit 46, and a display control unit 48. The CPU 20 executes the image processing program 30 to function as the acquisition unit 40, the extraction unit 42, the derivation unit 44, the detection unit 46, and the display control unit 48.
[0027] The acquisition unit 40 acquires medical images of a diagnosis target (hereinafter, referred to as “diagnosis target images”) from the image storage server 14 via the network I / F 25.
[0028] The extraction unit 42 extracts shape information of the target organ based on the diagnosis target image acquired by the acquisition unit 40 and the trained model 32. The trained model 32 is a trained model that receives a medical image as input and outputs shape information of the target organ, and is trained using a plurality of sets of medical images and shape information of the target organ in the medical images as learning data (sometimes referred to as teacher data). The trained model 32 is configured, for example, by a CNN (Convolutional Neural Network). The trained model 32 according to this embodiment outputs information that identifies the area of the target organ in the input medical image as the shape information of the target organ. This information may be any information that can identify the shape of the target organ, and may be, for example, information that represents the voxel position of the area of the target organ in the medical image, or an image in which the area of the target organ in the medical image is filled with a preset color.
[0029] As shown in FIG. 4, the extraction unit 42 inputs the diagnostic target image acquired by the acquisition unit 40 to the trained model 32. The trained model 32 outputs shape information of the target organ corresponding to the input diagnostic target image. As a result, the extraction unit 42 extracts the shape information of the target organ from the diagnostic target image. In the example of FIG. 4, the area of the pancreas represented by the shape information of the target organ is filled with diagonal lines. Note that, for the extraction of the target organ, a known method other than the method using the trained model 32, such as template matching or threshold processing on the CT value, may be applied.
[0030] The derivation unit 44 derives the shape normality of the target organ based on the shape information extracted by the extraction unit 42 and the trained model 34. The trained model 34 is a trained model that receives shape information of the target organ as input and outputs the shape normality of the target organ, and is trained using multiple sets of shape information and the shape normality of the target organ represented by the shape information as training data. The trained model 34 is configured by, for example, CNN. The trained model 34 according to this embodiment outputs a numerical value between a lower limit and an upper limit, which indicates the normality of the shape of the target organ, as the shape normality of the target organ. In this embodiment, the shape normality is a numerical value between 0 and 1, and the closer to 1 the value is, the closer the shape is to normal.
[0031] 5, the derivation unit 44 inputs the shape information extracted by the extraction unit 42 to the trained model 34. The trained model 34 outputs the shape normality of the target organ corresponding to the input shape information. In this way, the derivation unit 44 derives the shape normality of the target organ.
[0032] The detection unit 46 executes an anomaly detection process on the diagnostic target image based on the diagnostic target image acquired by the acquisition unit 40, the shape information extracted by the extraction unit 42, the shape normality derived by the derivation unit 44, and the trained model 36. The anomaly detection process in this embodiment is a process for detecting an anomaly in a target organ included in the diagnostic target image. The anomalies detected by the anomaly detection process in this embodiment include lesions that are directly treated, such as cancer, cysts, and inflammation. The anomalies detected include, in addition to lesions, parts that occur around the lesions and have at least one abnormal shape and property. These abnormal parts are also called indirect findings. For example, examples of indirect findings that may be indicative of pancreatic cancer include shape abnormalities, such as partial atrophy and swelling in the pancreas.
[0033] The trained model 36 is a trained model that receives a medical image, shape information of a target organ contained in the medical image, and a shape normality of the target organ represented by the shape information, and outputs information on an abnormality of the target organ (hereinafter referred to as "abnormality information"). The trained model 36 is trained using a plurality of sets of medical images, shape information, shape normality, and abnormality information as training data. The trained model 36 is configured by, for example, CNN. The trained model 36 according to the present embodiment outputs, as abnormality information, information that identifies an area (hereinafter referred to as "abnormal area") containing an abnormality in the target organ of the input medical image, and a certainty of the abnormality. The information that identifies the abnormal area may be information that can identify the abnormal area, and may be, for example, information that indicates the voxel position of the abnormal area in the medical image, or an image in which the abnormal area in the medical image is filled with a preset color.
[0034] 6, the detection unit 46 inputs the diagnostic target image acquired by the acquisition unit 40, the shape information extracted by the extraction unit 42, and the shape normality derived by the derivation unit 44 to the trained model 36. The trained model 36 outputs anomaly information corresponding to the input diagnostic target image, shape information, and shape normality. As a result, the detection unit 46 executes anomaly detection processing on the diagnostic target image.
[0035] The display control unit 48 controls the display of the results of the abnormality detection process performed by the detection unit 46 on the display 23. As shown in Fig. 7 as an example, when the abnormality information includes an abnormality whose certainty level is equal to or greater than a threshold value (e.g., 0.5), the display control unit 48 controls the display 23 to display, as the results of the abnormality detection process, the diagnosis target image, information indicating the abnormal area, and a message indicating that an abnormality exists. Fig. 7 shows an example in which the area inside the dashed line is detected as an abnormal area by the abnormality detection process.
[0036] As an example, as shown in FIG. 8, when the abnormality information does not include an abnormality whose certainty level is equal to or higher than a threshold, the display control unit 48 controls the display 23 to display the diagnostic target image and a message indicating that no abnormality was detected as a result of the abnormality detection process.
[0037] As shown in FIG. 9, the display control unit 48 may perform control to display on the display 23 the shape normality derived by the derivation unit 44 in addition to the execution result of the abnormality detection process.
[0038] 10, the display control unit 48 may perform control to display that the abnormality detection process cannot be executed when the shape normality derived by the derivation unit 44 indicates something other than that the shape of the target organ is normal. In the example of Fig. 10, a message is displayed to the effect that the abnormality detection process could not be executed because the target organ could not be correctly extracted. An example of a case in which the shape normality indicates something other than that the shape of the target organ is normal is when the shape normality is equal to or lower than a threshold value (e.g., 0.5).
[0039] Next, the operation of the image processing device 10 according to this embodiment will be described with reference to Fig. 11. The CPU 20 executes the image processing program 30 to execute the diagnostic support process shown in Fig. 11. The diagnostic support process shown in Fig. 11 is executed, for example, when a command to start execution is input by a user.
[0040] 11, the acquisition unit 40 acquires a diagnostic target image from the image storage server 14 via the network I / F 25. In step S12, the extraction unit 42 extracts shape information of the target organ based on the diagnostic target image acquired in step S10 and the trained model 32, as described above. In step S14, the derivation unit 44 derives a shape normality of the target organ based on the shape information extracted in step S12 and the trained model 34, as described above.
[0041] In step S16, the detection unit 46 executes anomaly detection processing on the diagnostic target image based on the diagnostic target image acquired in step S10, the shape information extracted in step S12, the shape normality derived in step S14, and the trained model 36, as described above. In step S18, the display control unit 48 controls the display of the execution result of the anomaly detection processing in step S16 on the display 23, as described above. When the processing of step S18 ends, the diagnostic support processing ends.
[0042] As described above, according to this embodiment, abnormalities can be detected from medical images with high accuracy.
[0043] In the above embodiment, the trained model 34 outputs a numerical value between a lower limit and an upper limit, which indicates the normality of the shape of the target organ, as the shape normality of the target organ. However, the present invention is not limited to this. For example, as shown in FIG. 12, the trained model 34 may output classification information indicating the classification of the shape information extraction result by the extraction unit 42 as the shape normality of the target organ. Examples of the classification information in this case include "normal" indicating that the extraction unit 42 was able to extract shape information of a normal target organ, "extraction failure" indicating that the extraction unit 42 failed to extract shape information, and "special shape" indicating that the extraction unit 42 extracted shape information of the target organ but the shape of the target organ was a special shape. In this embodiment, when the shape normality is "extraction failure" or "special shape", it is determined that the shape of the target organ is other than normal.
[0044] As shown in FIG. 13, the trained model 36 may be a plurality of trained models trained for each shape normality. In the example of FIG. 13, the trained model 36A is a trained model for a shape normality of "normal" trained using a plurality of sets of medical images, shape information, and abnormality information for which the shape normality is "normal". In the example of FIG. 13, the trained model 36B is a trained model for a shape normality of "extraction failure" trained using a plurality of sets of medical images and abnormality information for which the shape normality is "extraction failure". In the example of FIG. 13, the trained model 36C is a trained model for a shape normality of "special shape" trained using a plurality of sets of medical images, shape information, and abnormality information for which the shape normality is "special shape".
[0045] In this embodiment, the detection unit 46 executes an abnormality detection process by inputting the diagnosis target image and shape information to a trained model 36 corresponding to the shape normality derived by the derivation unit 44, among the multiple trained models 36. When the shape normality is a numerical value, the trained model 36 may be multiple trained models trained for different numerical ranges of shape normality, such as 0 or more and less than 1 / 3, 1 / 3 or more and less than 2 / 3, and 2 / 3 or more and less than 1.
[0046] In the above embodiment, the trained models 32, 34, and 36 are configured by CNN, but the present invention is not limited to this. The trained models 32, 34, and 36 may be configured by a machine learning method other than CNN.
[0047] In the above embodiment, the case where a CT image is applied as a diagnostic target image is described, but the present invention is not limited to this. As a diagnostic target image, a medical image other than a CT image, such as a radiological image taken by a plain X-ray imaging device and an MRI image taken by an MRI device, may be applied.
[0048] Furthermore, the processes of steps S10 to S16 of the diagnosis support process according to the above embodiment may be executed before the user inputs an instruction to start execution. In this case, when the user inputs an instruction to start execution, step S18 is executed and the screen is displayed.
[0049] In the above embodiment, the following various processors can be used as the hardware structure of the processing units that execute various processes, such as the acquisition unit 40, extraction unit 42, derivation unit 44, detection unit 46, and display control unit 48. As described above, the above various processors include a CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units, as well as a programmable logic device (PLD), which is a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array), a dedicated electric circuit, which is a processor having a circuit configuration designed specifically for executing specific processes, such as an ASIC (Application Specific Integrated Circuit), etc.
[0050] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, multiple processing units may be configured with a single processor.
[0051] As an example of configuring multiple processing units with one processor, first, there is a form in which one processor is configured with a combination of one or more CPUs and software, as typified by computers such as client and server, and this processor functions as multiple processing units. Second, there is a form in which a processor is used that realizes the functions of the entire system including multiple processing units with one IC (Integrated Circuit) chip, as typified by System on Chip (SoC), etc. In this way, the various processing units are configured using one or more of the above various processors as a hardware structure.
[0052] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements.
[0053] In the above embodiment, the image processing program 30 is stored (installed) in advance in the storage unit 22, but the present invention is not limited to this. The image processing program 30 may be provided in a form recorded on a recording medium such as a CD-ROM (Compact Disc Read Only Memory), a DVD-ROM (Digital Versatile Disc Read Only Memory), or a USB (Universal Serial Bus) memory. The image processing program 30 may be downloaded from an external device via a network. [Explanation of symbols]
[0054] 1 Medical Information System 10 Image processing device 12 Imaging Device 14 Image storage server 18 Network 20 CPU 21 Memory 22 Memory section 23 Display 24 Input Devices 25 Network I / F 27 Bus 30 Image Processing Programs 32, 34, 36, 36A, 36B, 36C Pre-trained models 40 Acquisition Department 42 Extraction part 44 Derivation part 46 Detection unit 48 Display control section
Claims
1. An image processing device comprising at least one processor, The processor: Extract shape information of the target organ based on the medical image, deriving a shape normality of the target organ based on the shape information; An abnormality detection process is performed on the medical image based on the medical image, the shape information, and the shape normality. Image processing device.
2. The shape normality is a numerical value between a lower limit and an upper limit, which indicates the normality of the shape of the target organ. The image processing device according to claim 1 .
3. The shape normality is classification information that indicates the classification of the extraction result of the shape information. The image processing device according to claim 1 .
4. The processor: The trained model receives the medical image, the shape information, and the shape normality and outputs abnormality information related to an abnormality in the target organ, and the trained model has been trained using a plurality of sets of the medical image, the shape information, the shape normality, and the abnormality information as training data. The abnormality detection process is performed by inputting a medical image to be diagnosed, the shape information of the medical image, and the shape normality of the medical image. The image processing device according to any one of claims 1 to 3.
5. The processor: The trained model receives the medical image and the shape information as input and outputs abnormality information related to an abnormality in the target organ, and is trained for each of the shape normalities using a plurality of sets of the medical image, the shape information, and the abnormality information as training data. The abnormality detection process is performed by inputting the medical image of the diagnosis target and the shape information of the medical image into a trained model corresponding to the shape normality derived for the medical image of the diagnosis target. The image processing device according to any one of claims 1 to 3.
6. The processor: If the degree of shape normality indicates that the shape of the target organ is not normal, control is performed to display that the abnormality detection process is not executable. The image processing device according to claim 1 .
7. The processor: Controlling the display of the execution result of the abnormality detection process and the shape normality The image processing device according to claim 1 .
8. Extract shape information of the target organ based on the medical image, deriving a shape normality of the target organ based on the shape information; An abnormality detection process is performed on the medical image based on the medical image, the shape information, and the shape normality. An image processing method in which processing is performed by a processor provided in an image processing device.
9. Extract shape information of the target organ based on the medical image, deriving a shape normality of the target organ based on the shape information; An abnormality detection process is performed on the medical image based on the medical image, the shape information, and the shape normality. An image processing program for causing a processor included in the image processing device to execute the processing.
Citation Information
Patent Citations
Image segmentation and object detection using fully convolutional neural network
US20200058126A1
Method and system for image analysis
US20210042930A1
Medical image diagnosis support device and method
WO2005011501A1
Generating a segmentation confidence map from node variations in a plurality of differently perturbed, model-based anatomical image segmentations
WO2021122248A1
Medical image processing device, method for operating medical image processing device, and operation program for medical image processing device
WO2022209889A1
