Image processing device, image processing method, and program
The image processing apparatus uses deep learning for fracture detection and visualization to address the limitations of existing devices, enabling accurate fracture analysis and treatment planning.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- FUTURE CORP
- Filing Date
- 2023-10-27
- Publication Date
- 2026-07-16
AI Technical Summary
Existing medical devices cannot accurately detect the fracture line and visualize the fracture cross-section, limiting the ability to formulate appropriate treatment plans.
An image processing apparatus that includes an extraction unit to identify fracture areas using a deep learning model for segmentation and a detection unit to determine fracture lines, with a visualization unit to superimpose these on a three-dimensional model for comprehensive fracture analysis.
Enables accurate understanding of fracture state and formulation of appropriate treatment plans by clearly visualizing fracture lines and cross-sections, supporting precise medical diagnosis.
Smart Images
Figure 0007891212000001 
Figure 0007891212000002 
Figure 0007891212000003
Abstract
Description
Technical Field
[0004] , , ,
[0006] , , , ,
[0005] , , , , ,
[0001] The present disclosure relates to an image processing apparatus, an image processing method, and a program.
Background Art
[0002] The medical device of Non-Patent Document 1 marks a location suspected of rib fracture from a chest computed tomography (CT) image that includes the entire rib. By having a doctor check the marked location, oversight can be prevented.
Prior Art Documents
Non-Patent Documents
[0003]
Non-Patent Document 1
Summary of the Invention
[0005] The medical device of Non-Patent Document 1 can detect the presence or absence of a fracture, but cannot detect the fracture line and visualize the fracture cross-section.
[0006] The present disclosure has been made in view of the above, and an object thereof is to enable an appropriate grasp of the state of a fracture.
Means for Solving the Problems
[0007] An image processing apparatus according to one aspect of the present disclosure includes an extraction unit that extracts a fractured area from a medical image taken of a suspected fracture site, and a detection unit that recursively determines a fracture line from the positional information of the fractured area. [Effects of the Invention]
[0008] According to this disclosure, it will become possible to properly understand the state of the fracture. [Brief explanation of the drawing]
[0009] [Figure 1] Figure 1 shows an example of the configuration of an image processing device. [Figure 2] Figure 2 is a flowchart showing an example of the processing flow of an image processing device. [Figure 3] Figure 3 is a flowchart showing an example of the processing flow in the extraction section. [Figure 4] Figure 4 is an example of a medical image in which the estimated fracture site and training data are superimposed. [Figure 5] Figure 5 is an example of a medical image in which the estimated fracture site and training data are superimposed. [Figure 6] Figure 6 shows an example of a three-dimensional model of bone. [Figure 7] Figure 7 shows an example of the configuration of a medical diagnostic system. [Figure 8] Figure 8 is a flowchart showing an example of the processing flow of a diagnostic device. [Figure 9] Figure 9 shows an example of a medical image with fracture lines superimposed on it. [Figure 10] Figure 10 shows an example of a medical image in which fracture lines and bone axes are superimposed. [Modes for carrying out the invention]
[0010] [Configuration of the image processing device] The embodiments of this disclosure will be described below with reference to the drawings. While a femoral fracture will be used as an example here, the method can also be applied to fractures in other locations.
[0011] Referring to Figure 1, an example of the configuration of the image processing apparatus 10 in this embodiment will be described. The image processing apparatus 10 shown in the figure comprises an input unit 11, an extraction unit 12, a detection unit 13, and a visualization unit 14. Each unit of the image processing apparatus 10 may be configured by at least one computer equipped with an arithmetic processing unit, a storage device, etc., and the processing of each unit may be executed by a program. This program is stored in the storage device of the image processing apparatus 10 and can be recorded on a computer-readable non-temporary recording medium such as a magnetic disk, optical disk, or semiconductor memory, or it can be provided via a network.
[0012] The input unit 11 receives medical images of the suspected fracture site. For example, CT images, which consist of multiple tomographic images, can be used as medical images. The slice thickness of the CT images is preferably 5 mm or less. MRI images may also be used as medical images.
[0013] The extraction unit 12 extracts fracture sites from medical images. Specifically, the extraction unit 12 inputs medical images into a trained deep learning model and extracts pixels that are estimated to be fracture sites. The deep learning model extracts features from the image and uses a technique called segmentation to classify each pixel in the image and estimate the fracture sites.
[0014] The deep learning model was trained to minimize the error between the detected fracture lines and the fracture lines in the training data, using medical images and fracture lines as training data. The trained parameters are stored in the memory device of the image processing device 10.
[0015] The deep learning model may be configured to perform segmentation at a low resolution, and then segment the boundaries at a high resolution. For example, points adaptively selected from the low-resolution segmentation map may be used for high-resolution inference.
[0016] When detecting feature points (for example, the endpoints of lines in bones) from the features of medical images and performing segmentation, the detection results of the feature points may be input to the deep learning model. Thereby, the detection accuracy of fracture lines can be improved.
[0017] In addition, the extraction unit 12 may classify fractures using a deep learning model. For example, when the suspected fracture site is the femur, the extraction unit 12 classifies the medical image as no fracture, with femoral neck fracture, or with trochanteric fracture. The extraction unit 12 may also discriminate the displacement of the fracture.
[0018] The detection unit 13 regressively obtains the fracture line on the medical image from the coordinates of the pixels of the fracture part extracted by the extraction unit 12.
[0019] The visualization unit 14 visualizes the fracture line detected by the detection unit 13. For example, the visualization unit 14 superimposes and displays the fracture line on the medical image. Thereby, the position and shape of the fracture can be easily understood.
[0020] In addition, the visualization unit 14 may specify the fracture cross-section based on the fracture lines detected from a plurality of tomographic images, generate a three-dimensional model of the bone from the plurality of tomographic images, and show the fracture cross-section on the transparent three-dimensional model of the bone. Since this three-dimensional model can be freely enlarged, reduced, and rotated, doctors can observe the patient's fracture state from various angles and make a diagnosis.
[0021] [Processing of the Image Processing Apparatus] Referring to the flowchart of FIG. 2, an example of the processing flow of the image processing apparatus 10 will be described.
[0022] In step S11, the input unit 11 inputs a medical image. In the case of a CT image, the input unit 11 inputs a plurality of tomographic images. The image processing apparatus 10 repeats the processing of the following steps S12 and S13 for each of the plurality of tomographic images. Note that the plurality of tomographic images may be processed in units of 2D or 3D patches, or the entire image may be processed collectively.
[0023] In step S12, the extraction unit 12 inputs the tomographic image into a deep learning model to extract the fracture site. For tomographic images that do not contain a fracture site, the process in step S13 is skipped, and the next tomographic image is processed.
[0024] In step S13, the detection unit 13 recursively determines the fracture line from the coordinates of the pixels at the fracture site. If there are any unprocessed tomographic images, the process returns to step S12 to process the next tomographic image.
[0025] In step S14, the visualization unit 14 identifies the fracture cross-section based on the fracture lines detected from multiple tomographic images and visualizes the fracture cross-section.
[0026] [Fracture extraction process] Referring to the flowchart in Figure 3, an example of the processing flow of the extraction unit 12 will be explained.
[0027] In step S121, the extraction unit 12 extracts features from the input medical image. Various methods can be used for feature extraction, such as Convolutional Neural Network (CNN), 3D CNN, Vision Transformer, and Multilayer Perceptron (MLP).
[0028] In step S122, the extraction unit 12 detects feature points from the features of the medical image. For example, the extraction unit 12 detects feature points that indicate the endpoints of straight lines within bone.
[0029] In step S123, the extraction unit 12 performs segmentation and estimates the pixels of the fracture site. At this time, the accuracy of extracting the fracture site can be improved by inputting the result of the feature points detected in step S122. Various methods can be used for segmentation, such as U-Net, Fully Convolutional Networks (FCN), DeepLab, PSPNet, HRNet, PointRend, and Mask R-CNN.
[0030] In step S124, the extraction unit 12 calculates the coordinates of feature points on the medical image. If feature points are not to be displayed, the calculation of the feature point coordinates is unnecessary.
[0031] Figures 4 and 5 show examples of medical images in which the fracture site 110, feature points 120, and training data 200 are superimposed. The cluster of dark dots in the figure represents the cluster of pixels estimated to be the fracture site 110. The two points in the figure are the feature points 120. The training data 200 is a straight line input along the fracture line. Figure 4 shows the result of extracting the fracture site when segmentation is performed without inputting the feature point information detected in step S122, and Figure 5 shows the result of extracting the fracture site when segmentation is performed with the feature point information detected in step S122 input. It can be seen that Figure 5 detects the fracture site 110 more accurately along the training data 200.
[0032] Through the above processing, it is possible to estimate the pixels of the fracture site in a medical image.
[0033] In step S131, the detection unit 13 detects a regression line from the coordinates of the pixels at the fracture site. The detected regression line is the fracture line.
[0034] In step S126, the extraction unit 12 may classify fractures based on the characteristics of the medical image. For example, the extraction unit 12 may classify the presence or absence of a fracture, the fracture site, and the state of the fracture.
[0035] [3D model] The 3D model of bone generated by the visualization unit 14 will be explained.
[0036] The visualization unit 14 uses existing technology to reconstruct multiple tomographic images (e.g., about 100 images) to generate a three-dimensional image (three-dimensional model). Figure 6 shows an example of a three-dimensional model of bone. The three-dimensional model can be visually enlarged, reduced, and rotated, and can be observed from any angle. The visualization unit 14 connects fracture lines detected from multiple tomographic images to identify fracture cross-sections and displays them superimposed on the transparent or semi-transparent three-dimensional model.
[0037] Visualizing fracture cross-sections makes it possible to identify fractures with significant displacement or three-dimensional injuries that require attention, supporting the formulation of treatment plans.
[0038] [Learning Example] As an example, the deep learning model was trained on medical images from 204 cases (110 cases with fractures and 94 cases without fractures). For the medical images with fractures used for training, expert-provided training data of fracture lines was added, and the deep learning model was trained to minimize the error between the fracture lines obtained by the image processing device 10 and the fracture lines in the training data.
[0039] After training, the system was tested with medical images from 37 cases (18 with fractures and 19 without). For fracture detection on a case-by-case basis, the sensitivity was 94.4% (17 / 18), specificity was 100.0% (19 / 19), and accuracy was 97.3% (36 / 37).
[0040] [Medical diagnostic system] Next, an example of the configuration of a medical diagnostic system equipped with the image processing device 10 of this embodiment will be described.
[0041] The medical diagnostic system shown in Figure 7 determines the presence or absence of a fracture in the proximal femur, detects the fracture line, evaluates the stability of the fracture site based on the angle of the fracture line, and displays the fracture type linked to the treatment plan. The medical diagnostic system in Figure 7 comprises an image processing device 10, a diagnostic device 20, a photography device 30, and a terminal 40. Each device is connected via a network for communication.
[0042] The imaging device 30 is a CT scanner used to image the suspected fracture site.
[0043] The image processing device 10 receives a medical image as input and detects the presence or absence of a fracture and the fracture line. The image processing device 10 may acquire a medical image from the imaging device 30, acquire a medical image from a storage device on the network, or receive a medical image as input from the terminal 40. Upon receiving a medical image as input, the image processing device 10 determines the presence or absence of a fracture, detects the fracture line, and transmits the processing results of the medical image to the diagnostic device 20.
[0044] The diagnostic device 20 displays the fracture type associated with the treatment plan based on the processing results of the image processing device 10.
[0045] Terminal 40 is a terminal operated by a physician. The physician operates the medical diagnostic system through Terminal 40, displaying medical images, fracture lines, and fracture types associated with treatment plans. Terminal 40 may also display a three-dimensional model showing the fracture cross-section.
[0046] The image processing device 10 and the diagnostic device 20 may be implemented using one or more computers, or they may be implemented as virtual machines on the cloud.
[0047] A program may be installed on terminal 40, allowing terminal 40 to function as an image processing device 10 and a diagnostic device 20. Terminal 40 may perform some of the functions of the image processing device 10 and the diagnostic device 20.
[0048] Referring to the flowchart in Figure 8, we will explain an example of the process flow for a medical diagnostic system to determine the fracture type.
[0049] When a physician operates terminal 40 and inputs a medical image into image processing device 10, the processing results of the medical image are transmitted to diagnostic device 20. The processing results include fracture classification and fracture line information.
[0050] In step S101, the diagnostic device 20 receives the processing result from the image processing device 10 and performs processing corresponding to the fracture classification, either no fracture, cervical fracture, or trochanteric fracture.
[0051] If no fracture is found, the diagnostic device 20 notifies the terminal 40 that there is no fracture. The terminal 40 displays the medical image and also indicates that there is no fracture.
[0052] In the case of a trochanteric fracture, the diagnostic device 20 notifies the terminal 40 that it is a trochanteric fracture. The terminal 40 displays a medical image and indicates that it is a trochanteric fracture.
[0053] In the case of a cervical fracture, the process proceeds to step S102, where the diagnostic device 20 determines whether or not there is displacement based on the processing results of the image processing device 10.
[0054] If the fracture is displaced, the diagnostic device 20 notifies the terminal 40 that it is a cervical fracture and that the fracture is displaced. The terminal 40 displays the medical image and also indicates that it is a trochanteric fracture and that the fracture is displaced.
[0055] If the fracture is not displaced, in step S103, the diagnostic device 20 notifies the terminal 40 that it is a cervical fracture, that the fracture is not displaced, and provides information about the fracture line. The terminal 40 overlays the fracture line onto the medical image. Figure 9 shows an example of the fracture line 300 being overlaid on the medical image.
[0056] In step S104, terminal 40 receives input of the bone axis (centerline along the longitudinal direction of the femur) from the physician and transmits the input bone axis information to the diagnostic device 20. Figure 10 shows an example of inputting the bone axis 400. The physician inputs the bone axis by drawing a straight line on the medical image displayed on terminal 40. The image processing device 10 may also estimate the bone axis.
[0057] In step S105, the diagnostic device 20 classifies the fracture type according to the Powells classification. The Powells classification is a classification based on the angle of the fracture line of femoral neck fractures and is used to evaluate the stability of the fracture site. The larger the angle between the horizontal line and the fracture line, the more unstable the fracture site is and the more likely it is to be displaced. Fractures with an angle of less than 30 degrees are classified as Type I, those with an angle of less than 50 degrees as Type II, and those with an angle of 50 degrees or more as Type III.
[0058] If the Powells classification is Type I or Type II, the diagnostic device 20 notifies the terminal 40 that the Powells classification is Type I or Type II. The terminal 40 displays the Powells classification.
[0059] If the Powells classification is type III, the diagnostic device 20 notifies the terminal 40 that the Powells classification is type III. The terminal 40 displays the Powells classification.
[0060] Physicians create treatment plans based on the fracture type displayed in the medical diagnostic system. For example, treatment plans according to fracture type include intramedullary nail fixation for trochanteric fractures, artificial hip replacement for displaced cervical fractures, osteosynthesis for non-displaced cervical fractures classified as Powells type I or II, and plate-assisted osteosynthesis for non-displaced cervical fractures classified as Powells type III.
[0061] As described above, the image processing device 10 of this embodiment includes an extraction unit 12 that extracts the fractured area from a medical image taken of the suspected fracture site, and a detection unit 13 that recursively determines the fracture line from the positional information of the fractured area. This makes it possible to appropriately understand the state of the fracture.
[0062] The extraction unit 12 extracts fracture areas using a deep learning model that classifies each pixel of a medical image to estimate fracture locations. The deep learning model is trained using the medical image and fracture lines as training data to minimize the error between the calculated fracture lines and the fracture lines of the training data. Furthermore, when performing segmentation, the deep learning model receives the detection results of feature points detected from the medical image. This allows for more accurate determination of fracture lines.
[0063] The visualization unit 14 generates fracture cross-sections from multiple fracture lines and displays these fracture cross-sections superimposed on a 3D model generated from multiple medical images. This enables a three-dimensional understanding of the injury and supports the appropriate formulation of treatment plans. [Explanation of Symbols]
[0064] 10 Image Processing Device 11 Input section 12 Extraction part 13 Detection unit 14 Visualization part 20 Diagnostic devices 30 Imaging device 40 devices
Claims
1. An extraction unit that extracts the fracture site from medical images taken of the suspected fracture site, The system includes a detection unit that recursively determines the fracture line from the positional information of the fracture site. Image processing device.
2. An image processing apparatus according to claim 1, The extraction unit extracts fracture sites using a deep learning model that classifies each region of a medical image and estimates the fracture site. The deep learning model was trained to minimize the error between the fracture lines obtained by the detection unit and the fracture lines in the training data, using medical images as input data and fracture lines as training data. Image processing device.
3. An image processing apparatus according to claim 2, The extraction unit detects feature points indicating the endpoints of straight lines within the bone from the medical image, and uses the detection results of the feature points to extract the fractured area for segmentation of the medical image. Image processing device.
4. An image processing apparatus according to any one of claims 1 to 3, The system includes a visualization unit that generates fracture cross-sections from multiple fracture lines and superimposes these fracture cross-sections onto a three-dimensional model generated from multiple medical images. Image processing device.
5. Computers The fracture site is extracted from medical images taken of the suspected fracture site. Regressively determine the fracture line from the location information of the fracture site. Image processing methods.
6. On the computer, The process involves extracting the fracture site from medical images taken of the suspected fracture site, The system performs a process to regressively determine the fracture line from the location information of the fracture site. program.