Image processing device, image processing method, and program

Abstract

To appropriately grasp a state of a fracture.SOLUTION: An image processing device 10 comprises: an extraction unit 12 which extracts a fracture portion from a medical image obtained by photographing a fracture suspect portion; and a detection unit 13 which recurrently calculates a fracture line from position information of the fracture portion. The extraction unit 12 extracts the fracture portion by using a depth learning model of estimating a fracture portion by classifying pixels of the medical image. With the medical image and the fracture line as teacher data, the depth learning model is learned so as to minimize an error between the calculated fracture line and the fracture line of the teacher data.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] The medical device in Non-Patent Document 1 marks areas where rib fractures are suspected from chest computed tomography (CT) images that include the entire rib. By having a doctor check the marked areas, it is possible to prevent oversights. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] "Rib fracture detection service," Fujifilm Corporation, [Retrieved October 3, 2023], Internet <URL: https: / / www.fujifilm.com / jp / ja / healthcare / healthcare-it / medical-cloud / rib-fracture-detection> Summary of the Invention [Problem to be solved by the invention]

[0004] It is believed that identifying not only the presence or absence of a fracture, but also the fracture line and fracture cross section can help determine the appropriate treatment plan. For example, in the case of a femoral neck fracture, the angle of the fracture line is important. If the fracture line is steep and the stability of the fracture is low, osteosynthesis using a plate is recommended.

[0005] The medical device of Non-Patent Document 1 can detect the presence or absence of a fracture, but cannot detect a fracture line or visualize a fracture cross section.

[0006] The present disclosure has been made in consideration of the above, and aims to enable the state of a fracture to be appropriately grasped. [Means for solving the problem]

[0007] An image processing device according to one aspect of the present disclosure includes an extraction unit that extracts a fractured portion from a medical image of a region suspected of being fractured, and a detection unit that recursively determines a fracture line from position information of the fractured portion. Effect of the Invention

[0008] According to the present disclosure, it becomes possible to appropriately grasp the state of a fracture. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of an image processing device. [Diagram 2] FIG. 2 is a flowchart showing an example of a process flow of the image processing device. [Diagram 3] FIG. 3 is a flowchart illustrating an example of a process flow of the extraction unit. [Figure 4] Figure 4 shows an example of a medical image in which the estimated fracture site and training data are superimposed. [Diagram 5] Figure 5 shows an example of a medical image in which the estimated fracture site and training data are superimposed. [Figure 6] FIG. 6 is a diagram showing an example of a three-dimensional model of a bone. [Figure 7] FIG. 7 is a diagram illustrating an example of the configuration of a medical diagnosis system. [Figure 8] FIG. 8 is a flowchart showing an example of a process flow of the diagnostic device. [Figure 9] FIG. 9 is a diagram showing an example in which a fracture line is superimposed on a medical image. [Figure 10] FIG. 10 is a diagram showing an example in which a fracture line and a bone axis are superimposed on a medical image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] [Image processing device configuration] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that, although a femur fracture will be described as an example, the present disclosure can also be applied to fractures in other locations.

[0011] An example of the configuration of an image processing device 10 of this embodiment will be described with reference to Fig. 1. The image processing device 10 shown in the figure includes an input unit 11, an extraction unit 12, a detection unit 13, and a visualization unit 14. Each unit included in the image processing device 10 may be configured with at least one computer including an arithmetic processing device, a storage device, etc., and the processing of each unit may be executed by a program. This program is stored in a storage device included in the image processing device 10, and can be recorded on a non-transitory computer-readable recording medium such as a magnetic disk, an optical disk, or a semiconductor memory, or can be provided via a network.

[0012] The input unit 11 inputs a medical image of a suspected fracture site. For example, a CT image composed of a plurality of cross-sectional images can be used as the medical image. The slice thickness of the CT image is preferably 5 mm or less. An MRI may be used as the medical image.

[0013] The extraction unit 12 extracts fractures from medical images. Specifically, the extraction unit 12 inputs the medical images into a trained deep learning model and extracts pixels that are estimated to be fractures. The deep learning model extracts features from the images and classifies each pixel in the image using a technique called segmentation to estimate the fractures.

[0014] The deep learning model was trained to minimize the error between the detected fracture line and the fracture line of the training data, using the medical image and the fracture line as training data. The trained parameters are stored in a storage device provided in the image processing device 10.

[0015] The deep learning model may be configured to perform a low-resolution segmentation followed by a high-resolution segmentation of the boundary, for example by inferring points adaptively selected from the low-resolution segmentation map at high resolution.

[0016] When detecting feature points (e.g., the end points of lines in bones) from the features of medical images and performing segmentation, the detection results of the feature points may be input to the deep learning model, which can improve the accuracy of detecting fracture lines.

[0017] The extraction unit 12 may also classify fractures using a deep learning model. For example, if the suspected fracture site is the femur, the extraction unit 12 classifies the medical image into no fracture, a neck fracture, or a trochanteric fracture. The extraction unit 12 may also determine the displacement of the fracture.

[0018] The detection unit 13 recursively determines the fracture line on the medical image from the coordinates of the pixels of the fractured portion extracted by the extraction unit 12.

[0019] The visualization unit 14 visualizes the fracture lines detected by the detection unit 13. For example, the visualization unit 14 superimposes the fracture lines on the medical image, which allows the position and shape of the fracture to be easily understood.

[0020] The visualization unit 14 may also identify the fracture cross section based on the fracture lines detected from the multiple tomographic images, generate a 3D model of the bone from the multiple tomographic images, and display the fracture cross section on the transparent 3D model of the bone. This 3D model can be freely enlarged, reduced, and rotated, allowing the doctor to observe the fracture state of the patient from various angles and make a diagnosis.

[0021] [Image processing device processing] An example of the flow of processing by the image processing device 10 will be described with reference to the flowchart of FIG.

[0022] In step S11, the input unit 11 inputs a medical image. In the case of a CT image, the input unit 11 inputs a plurality of tomographic images. The image processing device 10 repeats the following processing of steps S12 and S13 for each of the plurality of tomographic images. Note that the plurality of tomographic images may be collectively processed in units of 2D or 3D patches, or the entire image may be collectively processed.

[0023] In step S12, the extraction unit 12 inputs the tomographic image to the deep learning model to extract the fractured part. For a tomographic image that does not include a fractured part, the process of step S13 is not performed, and the next tomographic image is processed.

[0024] In step S13, the detection unit 13 recursively determines the fracture line from the coordinates of the pixels of the fractured portion. If there are any unprocessed tomographic images, the process returns to step S12 to process the next tomographic image.

[0025] In step S14, the visualization unit 14 identifies a fracture cross section based on the fracture lines detected from the multiple tomographic images, and visualizes the fracture cross section.

[0026] [Fracture extraction process] An example of the flow of processing by the extraction unit 12 will be described with reference to the flowchart of FIG.

[0027] In step S121, the extraction unit 12 extracts features from the input medical image. For the extraction of features, various methods such as Convolutional Neural Network (CNN), 3D CNN, Vision Transformer, and Multilayer Perceptron (MLP) can be used.

[0028] In step S122, the extraction unit 12 detects feature points from the features of the medical image. For example, the extraction unit 12 detects feature points indicating end points of straight lines within a bone.

[0029] In step S123, the extraction unit 12 performs segmentation to estimate the pixels of the fractured part. At this time, by inputting the result of the feature points detected in step S122, the extraction accuracy of the fractured part can be improved. For the segmentation, various methods such as U-Net, Fully Convolutional Networks (FCN), DeepLab, PSPNet, HRNet, PointRend, Mask R-CNN, etc. can be used.

[0030] In step S124, the extraction unit 12 calculates the coordinates of the feature points on the medical image. Note that if the feature points are not displayed, it is not necessary to calculate the coordinates of the feature points.

[0031] 4 and 5 show examples of medical images in which a fracture 110, feature points 120, and training data 200 are superimposed. The collection of dark dots on the figure is a collection of pixels estimated to be the fracture 110. The two dots on the figure are feature points 120. The training data 200 is a straight line input along the fracture line. FIG. 4 shows the extraction result of the fracture when segmentation is performed without inputting the information of the feature points detected in step S122, and FIG. 5 shows the extraction result of the fracture when segmentation is performed by inputting the information of the feature points detected in step S122. It can be seen that FIG. 5 shows the fracture 110 being detected along the training data 200 with greater accuracy.

[0032] By the above processing, the pixels of the fractured part in the medical image can be estimated.

[0033] In step S131, the detection unit 13 detects a regression line from the coordinates of the pixels of the fractured portion. The detected regression line is the fracture line.

[0034] In step S126, the extraction unit 12 may classify the fracture based on the characteristics of the medical image. For example, the extraction unit 12 classifies the presence or absence of a fracture, the fracture site, and the fracture state.

[0035] [3D model] The three-dimensional bone model generated by the visualization unit 14 will now be described.

[0036] The visualization unit 14 uses existing technology to reconstruct multiple tomographic images (e.g., about 100 images) and generate a 3D image (3D model). Figure 6 shows an example of a 3D model of a bone. The 3D model can be visually enlarged, reduced, and rotated, and can be observed from any angle. The visualization unit 14 connects the fracture lines detected from the multiple tomographic images to identify the fracture cross section, and displays it superimposed on the transparent or semi-transparent 3D model.

[0037] By visualizing the fracture cross section, it becomes possible to grasp fractures with large displacements and three-dimensional damage that require attention, and it can support the formulation of treatment plans.

[0038] [Study example] As an example, the deep learning model was trained on medical images of 204 cases (110 cases with fractures and 94 cases without fractures). Training data of fracture lines prepared by experts was added to the training medical images with fractures, and the deep learning model was trained to minimize the error between the fracture lines obtained by the image processing device 10 and the fracture lines in the training data.

[0039] After training, the system was tested on medical images of 37 cases (18 with fractures and 19 without fractures). The sensitivity, specificity, and accuracy of the system for determining the presence or absence of fractures on a case-by-case basis were 94.4% (17 / 18), 100.0% (19 / 19), and 97.3% (36 / 37), respectively.

[0040] [Medical diagnostic systems] Next, an example of the configuration of a medical diagnosis system including the image processing device 10 of this embodiment will be described.

[0041] The medical diagnosis system shown in Fig. 7 determines the presence or absence of a fracture in a proximal femur fracture, detects a fracture line, evaluates the stability of the fracture based on the angle of the fracture line, and displays the fracture type linked to a treatment plan. The medical diagnosis system in Fig. 7 includes an image processing device 10, a diagnostic device 20, an imaging device 30, and a terminal 40. Each device is connected to each other so as to be able to communicate with each other via a network.

[0042] The imaging device 30 is a CT scanner that images the area suspected of having a fracture.

[0043] The image processing device 10 inputs a medical image and detects the presence or absence of a fracture and a fracture line. The image processing device 10 may acquire a medical image from an imaging device 30, may acquire a medical image from a storage device on a network, or may input a medical image from a terminal 40. When the image processing device 10 inputs a medical image, it determines the presence or absence of a fracture and detects a fracture line, and transmits the processing result of the medical image to the diagnostic device 20.

[0044] The diagnostic device 20 displays the fracture type associated with the treatment plan based on the processing result of the image processing device 10.

[0045] The terminal 40 is a terminal operated by a doctor. The terminal 40 operates a medical diagnosis system by the doctor's operation, and displays medical images, fracture lines, and fracture types associated with treatment plans. The terminal 40 may display a three-dimensional model showing a fracture cross section.

[0046] The image processing device 10 and the diagnostic device 20 may be realized by one or more computers, or may be realized by a virtual machine on a cloud.

[0047] A program may be installed in the terminal 40 to cause the terminal 40 to function as the image processing device 10 and the diagnostic device 20. The terminal 40 may execute some of the functions of the image processing device 10 and the diagnostic device 20.

[0048] An example of the flow of processing by the medical diagnosis system for determining the type of fracture will be described with reference to the flowchart of FIG.

[0049] When a doctor operates the terminal 40 and inputs a medical image to the image processing device 10, the processing result of the medical image is transmitted to the diagnostic device 20. The processing result includes a classification of the fracture and information on the fracture line.

[0050] In step S101, the diagnostic device 20 receives the processing result of the image processing device 10, and performs processing corresponding to either no fracture, neck fracture, or trochanteric fracture according to the classification of the fracture.

[0051] If no fracture is found, the diagnostic device 20 notifies the terminal 40 that there is no fracture. The terminal 40 displays the medical image and also indicates that there is no fracture.

[0052] In the case of a trochanteric fracture, the diagnostic device 20 notifies the terminal 40 that it is a trochanteric fracture. The terminal 40 displays the medical image and indicates that it is a trochanteric fracture.

[0053] In the case of a cervical fracture, the process proceeds to step S102, where the diagnostic device 20 determines the presence or absence of a dislocation based on the processing result of the image processing device 10.

[0054] If the fracture is displaced, the diagnostic device 20 notifies the terminal 40 that it is a neck fracture and that the fracture is displaced. The terminal 40 displays the medical image and also indicates that it is a trochanteric fracture and that the fracture is displaced.

[0055] If the fracture is non-displaced, in step S103, the diagnostic device 20 notifies the terminal 40 that it is a neck fracture, that the fracture is non-displaced, and information on the fracture line. The terminal 40 displays the fracture line superimposed on the medical image. Fig. 9 shows an example of a fracture line 300 superimposed on a medical image.

[0056] In step S104, the terminal 40 accepts input of the bone axis (the center line along the longitudinal direction of the femur) from the doctor, and transmits the input information of the bone axis to the diagnostic device 20. Fig. 10 shows an example of input of a bone axis 400. The doctor inputs the bone axis by drawing a straight line on the medical image displayed on the terminal 40. The image processing device 10 may estimate the bone axis.

[0057] In step S105, the diagnostic device 20 classifies the type of fracture according to the Pauwels classification. The Pauwels classification is a classification based on the angle of the fracture line of the femoral neck fracture, and is a classification that evaluates the stability of the fractured part. The larger the angle between the horizontal line and the fracture line, the more unstable the fractured part is and the more likely it is to displace. An angle of less than 30 degrees is classified as Type I, less than 50 degrees as Type II, and 50 degrees or more as Type III.

[0058] If the Pauwels classification is type I or type II, the diagnostic device 20 notifies the terminal 40 that the Pauwels classification is type I or type II. The terminal 40 displays the Pauwels classification.

[0059] If the Pauwels classification is Type III, the diagnostic device 20 notifies the terminal 40 that the Pauwels classification is Type III. The terminal 40 displays the Pauwels classification.

[0060] Doctors create treatment plans based on the fracture type displayed on the medical diagnostic system. For example, treatment plans according to fracture type include osteosynthesis with intramedullary nail fixation for trochanteric fractures, total hip replacement for displaced neck fractures, osteosynthesis for non-displaced neck fractures classified as Pauwels type I or II, and osteosynthesis with plate for non-displaced neck fractures classified as Pauwels type III.

[0061] As described above, the image processing device 10 of this embodiment includes the extraction unit 12 that extracts a fractured portion from a medical image of a suspected fracture site, and the detection unit 13 that recursively determines a fracture line from position information of the fractured portion. This makes it possible to appropriately grasp the state of a fracture.

[0062] The extraction unit 12 extracts the fractured part using a deep learning model that classifies each pixel of the medical image and estimates the fractured part. Using the medical image and the fracture line as training data, the deep learning model is trained to minimize the error between the fracture line to be obtained and the fracture line of the training data. In addition, when performing segmentation, the deep learning model inputs the detection result of the feature points detected from the medical image. This makes it possible to determine the fracture line with higher accuracy.

[0063] The visualization unit 14 generates a fracture cross section from multiple fracture lines and displays the fracture cross section superimposed on a three-dimensional model generated from multiple medical images. This makes it possible to grasp the damage in three dimensions and supports the formulation of an appropriate treatment plan. [Explanation of symbols]

[0064] 10 Image processing device 11 Input section 12 Extraction part 13 Detection section 14 Visualization part 20 Diagnostic Equipment 30 Imaging Equipment 40 Terminals

Claims

1. an extraction unit that extracts a fractured portion from a medical image of a suspected fractured portion; A detection unit for recursively detecting a fracture line from the position information of the fractured portion is provided. Image processing device.

2. 2. The image processing device according to claim 1, The extraction unit extracts the fractured part using a deep learning model that classifies each region of the medical image and estimates the fractured part, The deep learning model was trained to minimize the error between the fracture line obtained and the fracture line in the training data using medical images and fracture lines as training data. Image processing device.

3. 3. The image processing device according to claim 2, The extraction unit detects feature points indicating end points of lines from the medical image, and extracts the fractured portion by using a detection result of the feature points in segmentation of the medical image. Image processing device.

4. 4. The image processing device according to claim 1, A visualization unit is provided that generates a fracture cross section from a plurality of fracture lines and superimposes the fracture cross section on a three-dimensional model generated from a plurality of medical images. Image processing device.

5. The computer The fracture site is extracted from medical images taken of the suspected fracture site, Recursively find the fracture line from the fracture position information Image processing methods.

6. On the computer, A process of extracting a fractured portion from a medical image of a suspected fractured portion; Execute a process to recursively find the fracture line from the position information of the fractured part. program.

Images