Automatic rib fracture detection from expanded scan images
Through the new manifold view technology, reformatting and fracture detection of medical imaging data, the problem of receptive field size and resolution trade-offs in the existing technology is solved, and accurate and continuous detection of rib fractures is achieved, which improves the detection Reliability and sensitivity.
Patent Information
- Application Number
- CN202380032298.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-07-07
- Filing Date
- 2023-04-02
- Publication Date
- 2024-11-19
AI Technical Summary
Existing medical imaging techniques face a trade-off between receptive field size and resolution when automatically detecting rib fractures, which makes detection difficult, especially the difficulty of accurately identifying fractures and fractures at adjacent rib positions, and artifacts exist in existing views. and discontinuity issues.
Using a new manifold view as input to the rib fracture detection model, the three-dimensional diagnostic images are reformatted through the mapping function and manifold generation module to generate continuous and straightened 2D manifold slices and utilize trained fracture detection The model made predictions, avoiding the limitations of Cartesian grid reconstruction, achieving consistency in rib length and visualization of subtle fractures.
Improve the accuracy and visual continuity of rib fracture detection, reduce artifacts and discontinuities, enhance the detection ability of adjacent rib fractures, reduce false positive rates, and improve the robustness and sensitivity of detection .
Smart Images

Figure CN118985004A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of medical imaging, and in particular to a method and apparatus for automatically detecting rib fractures from medical imaging scans. Background Art
[0002] Medical professionals often obtain medical imaging to identify fractures in a patient's thorax. Medical imaging, such as computed tomography (CT) scans, is a tool for visualizing a patient's anatomical structure, wherein the visualized data is used to diagnose disease or injury. Specifically, in a trauma setting, medical personnel may rely on medical imaging data in the form of CT scans for diagnostic purposes, such as detection of rib fractures. Reading imaging scans, and more specifically, trauma or emergency department scans, is a time-critical task that needs to be completed with high concentration to avoid overlooking critical findings. In some cases, where a rib fracture may not be clear, the ribs need to be examined from different perspectives (e.g., sagittal and coronal views rather than axial views), and adjacent ribs and lung areas need to be examined for potential indications, further increasing the examination time.
[0003] The diagnosis of rib fractures requires tracking of 24 individual ribs, each of which spans multiple slices of an imaging scan. Manual inspection is therefore a tedious and time-consuming process. Reliable detection and diagnosis of rib fractures is challenging, and different types of fractures may present different challenges in the detection and diagnosis of rib fractures, for example, minor fractures appear as slight irregularities in the cortical bone.
[0004] In the past few years, several methods for automatic rib fracture detection and methods for visualizing the patient's thorax in an intuitive way to simplify human examination have been introduced. However, these methods decouple automatic fracture detection and visualization.
[0005] Several methods have been proposed for simplifying the visualization and assessment of a patient's anatomy and key structures, particularly those targeting visualization of the thorax. While providing some benefits, current methods have various fundamental limitations.
[0006] A well-known visualization scheme is the "filet view" or "fishbone view". This view is based on segmenting the ribs (e.g. using a deep convolutional neural network), followed by a centerline extractor, which then marks the rib pairs in the field of view. Each rib is sampled along its trace, allowing to visualize each and every rib in a normalized and straightened manner (curved planes reformatted). A disadvantage of this type of view is that the nature of processing each rib independently leads to discontinuities between ribs, which may cause imaging artifacts from neighboring ribs appearing in the rib shape.
[0007] Another visualization scheme is the visceral cavity view. In this view, a segmentation algorithm (e.g., using a model-based approach) is applied to segment the inside of the thorax according to the deformation of a cylindrical manifold. Once the segmentation is completed, the manifold can be unfolded, and the maximum intensity projection (MIP) near the surface can be calculated. This view allows the user to inspect the thorax as a whole in terms of continuous visualization on the inspection screen. One of the disadvantages of this type of view is that the relative rib length is not maintained. The nature of the unfolded cylindrical manifold does not allow visualization of the correct rib length. For example, the first rib appears too long relative to the other ribs. In addition, the nature of MIP visualization may not allow detection of subtle fractures. For example, MIP visualization can make small rib fractures invisible and undetectable in the generated view. Another defect is that this view adds significant unrealistic distortion (wavy ribs), which limits the clinical confidence in the generated visceral cavity view.
[0008] There is a huge demand for automatic rib fracture detection from medical images such as CT images, but automatic rib fracture detection is a challenging task. Due to memory and runtime constraints, common deep learning-based methods suffer from a trade-off between input resolution and receptive field size. In order to identify subtle rib fractures, such as hairline cracks, high image resolution (e.g., 1 mm isotropic) is required. However, since rib fractures often occur at similar locations on adjacent ribs, a large receptive field size is equally beneficial for the task. In addition, it is difficult to identify the location of fractures on the thorax from high-resolution local patches. Irregular parts of the ribs, such as tubercles or sternal rib ends, may be misinterpreted as buckled or displaced fractures, especially the first and last ribs. Current methods seem to act on local patches oriented and extracted from the original image, which are then fed into a convolutional neural network. With these methods, it is difficult to address task-specific challenges.
[0009] Therefore, there is a need for an innovative solution for automatically detecting rib fractures from medical imaging scans. Summary of the invention
[0010] The present invention aims to provide a method and apparatus for automatically detecting rib fractures from medical imaging scans. The technique can be applied to scans produced by a plurality of imaging systems, including CT, CT arms, single photon emission computed tomography CT (SPECT-CT), magnetic resonance CT (MR-CT), positron emission tomography CT (PET-CT), and magnetic resonance imaging (MRI) systems.
[0011] To address the trade-off between receptive field size and resolution, the technique performs rib fracture detection on reformatted images instead of working on three-dimensional (3D) reconstructed image volumes using a Cartesian grid. A new manifold view is introduced and selected as the input of the rib fracture detection model.
[0012] The new manifold view offers advantages over existing views. For example, in contrast to the "fillet view", there are no sampling artifacts (e.g., rib duplication), and there are strong image edges (e.g., at the boundaries between adjacent ribs or background) that eliminate imaging features that could confuse fracture detectors. Compared to the visceral cavity view, the ratios of rib lengths are unchanged, and rib positions are consistent. Notably, the new manifold view is again a three-dimensional volume, showing only the thorax in a standardized, substantially smaller image.
[0013] According to a first aspect of the present invention, an image processing device is provided. The image processing device includes a mapping function module, the mapping function module includes a mapping function circuit, and the mapping function circuit is configured to: receive data representing a three-dimensional diagnostic image; segment the ribs and the spine according to the received data representing the three-dimensional diagnostic image; detect and mark the rib centerline according to the rib segmentation; detect and mark the vertebral body center mark according to the spine segmentation; map each three-dimensional position of the rib centerline and the vertebral body center mark corresponding to the received data representing the three-dimensional diagnostic image to a two-dimensional position on a defined two-dimensional manifold plane; interpolate the missing three-dimensional position coordinates on the defined two-dimensional manifold plane, deform the two-dimensional manifold plane so that it is aligned with the detected rib centerline and vertebral center mark in the three-dimensional space of the diagnostic image; and generate a coordinate mapping function from the two-dimensional manifold space to the three-dimensional image space corresponding to the mapped and interpolated position, and vice versa.
[0014] The image processing device includes a manifold generation module, which includes a manifold processing circuit, and the manifold processing circuit is configured to: receive a generated coordinate mapping function; sample image intensity from the three-dimensional diagnostic image space at each coordinate defined by a deformed two-dimensional manifold plane according to the mapping function; generate a reformatted image from the sampled image intensity as a two-dimensional manifold slice, the two-dimensional manifold slice showing a continuous and straightened visualization of the thorax and spine; shift the deformed two-dimensional manifold plane along its normal direction; and repeat the sampling to generate a stack of manifold slices covering the complete three-dimensional thorax visualization.
[0015] The image processing device includes a trained fracture detection module, which includes a fracture prediction circuit system, and the fracture prediction circuit system is configured to: receive a stack of generated manifold slices as input; perform fracture prediction via a trained fracture detection model using machine learning or deep learning technology; and generate a modified two-dimensional manifold view showing the predicted fracture in the space of the stack of generated manifold slices covering the complete three-dimensional thorax visualization based on the trained fracture detection model.
[0016] The image processing device also includes an inverse coordinate transformation module, which includes an inverse coordinate transformation circuit system, and the inverse coordinate transformation circuit system is configured to: receive the mapping function and the modified two-dimensional manifold view showing the predicted fracture as input; and map the predicted fracture from the stack of manifold slices back to the image space representing the data of the three-dimensional diagnostic image.
[0017] In a second aspect of the present invention, a method for automatically detecting rib fractures is provided. The method comprises: receiving data representing a three-dimensional diagnostic image; segmenting the ribs and spine according to the received data representing the three-dimensional diagnostic image; detecting and marking the rib centerline according to the rib segmentation; detecting and marking the vertebral body center mark according to the spine segmentation; mapping each three-dimensional position of the rib centerline and the vertebral body center mark corresponding to the received data representing the three-dimensional diagnostic image to a two-dimensional position on a defined two-dimensional manifold plane; interpolating the missing three-dimensional position coordinates on the defined two-dimensional manifold plane to deform the two-dimensional manifold plane so that it is aligned with the detected rib centerline and vertebral center mark in the three-dimensional space of the diagnostic image; generating a coordinate mapping function corresponding to the mapped and interpolated position from the two-dimensional manifold space to the three-dimensional image space and vice versa; receiving the generated coordinate mapping function; and mapping the coordinates from the three-dimensional diagnostic image space at each coordinate defined by the deformed two-dimensional manifold plane according to the mapping function. The invention relates to a method for detecting a fracture of a thoracic cavity and a spine, wherein the method comprises: sampling image intensities at intervals; generating a reformatted image as a two-dimensional manifold slice based on the sampled image intensities, the two-dimensional manifold slice showing a continuous and straightened visualization of the thorax and spine; shifting the deformed two-dimensional manifold plane along its normal direction and repeating the sampling to generate a stack of manifold slices covering a complete three-dimensional thorax visualization; receiving the generated stack of manifold slices as input to a trained fracture detection model; performing fracture prediction via the trained fracture detection model based on machine learning or deep learning techniques; generating a modified two-dimensional manifold view showing a predicted fracture in the space of the generated stack of manifold slices covering a complete three-dimensional thorax visualization based on the trained fracture detection model; receiving the mapping function and the modified two-dimensional manifold view showing the predicted fracture; and mapping the predicted fracture from the stack of manifold slices back to the image space of the data representing the three-dimensional diagnostic image.
[0018] In a third aspect of the present invention, a non-transitory computer-readable medium having stored thereon instructions for causing a processing circuit system to perform processing is provided. The processing includes: receiving data representing a three-dimensional diagnostic image; segmenting the ribs and spine according to the received data representing the three-dimensional diagnostic image; detecting and marking the rib centerline from the rib segmentation; detecting and marking the vertebral body center mark from the spine segmentation; mapping each three-dimensional position of the rib centerline and the vertebral body center mark corresponding to the received data representing the three-dimensional diagnostic image to a two-dimensional position on a defined two-dimensional manifold plane; interpolating the missing three-dimensional position coordinates on the defined two-dimensional manifold plane, which deforms the two-dimensional manifold plane so that it is aligned with the detected rib centerline and vertebral center mark in the three-dimensional space of the diagnostic image; generating a coordinate mapping function from the two-dimensional manifold space to the three-dimensional image space and corresponding to the mapped and interpolated positions, and vice versa; receiving the generated coordinate mapping function; according to the mapping function, at each coordinate defined by the deformed two-dimensional manifold plane from the three-dimensional diagnostic image spatially sampling image intensities; generating a reformatted image from the sampled image intensities as a two-dimensional manifold slice, the two-dimensional manifold slice showing a continuous and straightened visualization of the thorax and spine; shifting the deformed two-dimensional manifold plane along its normal direction and repeating the sampling to generate a stack of manifold slices covering a complete three-dimensional thorax visualization; receiving the generated stack of manifold slices as input to a trained fracture detection model; performing fracture prediction via the trained fracture detection model according to machine learning or deep learning techniques; generating a modified two-dimensional manifold view showing a predicted fracture in the space of the generated stack of manifold slices covering a complete three-dimensional thorax visualization according to the trained fracture detection model; receiving the mapping function and the modified two-dimensional manifold view showing the predicted fracture; and mapping the predicted fracture from the stack of manifold slices back to the image space of the data representing the three-dimensional diagnostic image.
[0019] In a preferred embodiment, the transformation module includes a transformation circuit system configured to: receive the generated stack of manifold slices and the mapping function as input; generate an annotated stack of manifold slices by repeatedly sampling, generating and shifting by the manifold processing circuit system using a three-dimensional annotation mask or a three-dimensional annotation mark as a sampling input; and generate an annotation mask or an annotation mark set defined in the generated stack of manifold slices by mapping annotated fractures from the three-dimensional image space to the three-dimensional annotated stack of manifold slices according to the mapping function. The trained fracture detection module receives the generated stack of manifold slices and the generated annotated stack of manifold slices as input to train the trained fracture detection model.
[0020] In one aspect of the invention, the trained fracture detection module receives as input a stack of generated manifold slices and annotated stack of generated manifold slices to train the trained fracture detection model. In a preferred embodiment, the trained fracture detection model is trained based on a slice-by-slice single shot or patch-based model. The receptive field size is tuned along the defined stack of manifold slices so that fractures at adjacent ribs are captured during training of the trained fracture detection model.
[0021] According to another aspect of the invention, the predicted fractures are shown in the space of the stack of manifold slices or in the space of the diagnostic images as a list of landmarks or voxel-by-voxel annotations with corresponding color layers. In addition, the number of predicted fractures per rib is determined and shown in the space of the stack of manifold slices or in the space of the diagnostic images.
[0022] In another aspect of the invention, the predicted fractures are subsequently analyzed by a classification model.
[0023] In a preferred embodiment, interpolating said missing coordinates on said two-dimensional manifold is performed via an interpolation technique comprising thin plate splines.In one aspect of the invention, additional inputs to said trained fracture detection module include scanned images and annotations from previous imaging scans. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] In the following figures:
[0025] Figure 1 is a schematic diagram of an exemplary imaging system;
[0026] Figure 2 Components of an imaging processing apparatus according to an embodiment are illustrated;
[0027] Figure 3 illustrates a processing workflow according to an embodiment;
[0028] Figure 4 illustrates a 3D view of ribs and a spine according to an embodiment;
[0029] Figure 5 Examples of subtle rib fractures are shown; and
[0030] Figure 6 illustrating a visceral cavity view of the thorax, a fillet view of the thorax, and a new manifold view of the thorax; and
[0031] Figure 7 is a flow chart representing a method of automatic bone fracture detection according to an embodiment. DETAILED DESCRIPTION
[0032] Figure 1 An imaging system 100, such as a computed tomography (CT) imaging system, is illustrated. The imaging system 100 includes a generally stationary gantry 102 and a rotating gantry 104. The rotating gantry 104 is rotatably supported by the stationary gantry 102 and rotates relative to the stationary gantry 102 about an examination region 106 about a longitudinal or z-axis.
[0033] A patient support 112, such as a couch, supports an object or examination subject, such as a human patient in the examination area 106. The support 112 is configured to move the object or examination subject to load, scan, and unload the object or examination subject.
[0034] A radiation source 108, such as an X-ray tube, is rotatably supported by the rotating gantry 104. The radiation source 108 rotates with the rotating gantry 104 and emits radiation that passes through the examination region 106.
[0035] A radiation sensitive detector array 110 spans the examination region 106 and subtends an angular arc opposite the radiation source 108. The detector array 110 includes one or more rows of detectors extending along the z-axis direction, detects radiation traversing the examination region 106, and generates projection data indicative thereof.
[0036] A general purpose computing system or computer is used as an operator console 114 and includes input devices 116 such as a mouse, keyboard, etc. and output devices 120 such as a display monitor, filters, etc. The console 114 allows the operator to control the operation of the system 100. This includes control of an image processing device 118. The image processing device 118 can receive data representing a three-dimensional (3D) diagnostic image generated by the imaging system 100 and generate a reformatted image that displays the predicted rib fracture in the image space of the three-dimensional (3D) diagnostic image. The reformatted image can be output to the output device 120 for reading by a radiologist or other medical personnel.
[0037] It should be understood that the processing of the image processing device 118 can be implemented by a processor and / or processing circuit system that executes computer-readable instructions, such as executable code encoded or embedded on a computer-readable storage medium (such as physical memory and other non-transient media). Additionally or alternatively, the processor and / or processing circuit system can execute computer-readable instructions carried by carrier waves, signals and other transient (or non-transient) media.
[0038] Figure 2Several components of the image processing device 118 according to some embodiments are illustrated. At the mapping module 202, a three-dimensional (3D) diagnostic image generated by an imaging system (e.g., a CT imaging system 100) is received. The ribs and spine are automatically segmented. Segmentation can be performed using machine learning or deep learning techniques. As an example, a convolutional neural network (CNN) is a deep learning technique that can be used to segment the ribs and spine. CNN architectures (such as U-Net and V-Net) can be applicable to segmentation tasks. Training based on deep learning using data sets from hundreds of scans can be used to develop algorithms that are shown to achieve fast and accurate segmentation of ribs and spine. The ribs are detected and the rib center lines are extracted and marked, and the vertebral body center marks are detected and marked. Each three-dimensional (3D) position of the rib center line and the vertebral center mark corresponding to the received data representing the 3D diagnostic image is mapped to a two-dimensional position on a defined two-dimensional (2D) manifold plane.
[0039] Since the rib and vertebra positions are known in 3D space, the correlation of all rib center lines and all vertebrae landmarks can be constructed. That is, the coordinate correspondence from 3D space to 2D space can be defined as a mapping function. Once a 3D to 2D correspondence is established or set, the correspondence or correlation of the remaining pixels from the defined 2D space may be missing. Those 3D position coordinates missing on the defined 2D manifold plane are interpolated, thereby deforming the two-dimensional manifold plane so that it is aligned with the detected rib center lines and vertebrae center landmarks in the 3D space of the diagnostic image. Finally, a coordinate mapping function corresponding to the mapped and interpolated positions is generated from the 2D manifold space to the 3D image space, and a coordinate mapping function is generated conversely.
[0040] The generated mapping function is received at the manifold generation module 204. According to the received mapping function, the image intensity from the 3D diagnostic image space is sampled at each coordinate defined by the deformed 2D manifold plane. The reformatted image is generated as a 2D manifold slice from the sampled image intensity. The 2D manifold slice shows a continuous and straightened visualization of the thorax and spine. The stack of manifold slices is generated by shifting the deformed 2D manifold plane along its normal direction toward the inside and outside of the manifold plane, and repeating the sampling for each incremental shift along the normal direction.
[0041] exist Figure 3, the process of generating a new manifold view is illustrated in . At 302, rib centerlines and vertebral body center landmarks are detected and marked according to an automatic segmentation scheme. As discussed, the automatic segmentation scheme can be based on machine learning or deep learning techniques. A 2D manifold is defined that will display the resulting new manifold view. At 304, an interpolation scheme deforms the 2D manifold so that it is aligned with the detected rib centerlines and vertebral center landmarks. Thin plate splines or any appropriate interpolation scheme can be used as an interpolation scheme. As shown in 304, the defined 2D manifold is deformed to align with the detected rib centerlines and vertebral landmarks mapped from 3D space to the 2D deformed manifold's 2D space.
[0042] The image intensity from the 3D image space is sampled at each coordinate of the deformed 2D manifold to complete or fill in the missing 3D coordinates. The result is shown at 306. The missing 3D coordinates may correspond to tissue regions between the ribs and tissue regions attached to the ribs, or any combination of tissue regions between the ribs and tissue regions attached to the ribs. This allows the deformed 2D manifold to be unfolded.
[0043] The unfolded 2D manifold is shown at 308. The new manifold view shown at 308 combines the advantages of both the fillet and visceral cavity views, but avoids the disadvantages of both the fillet and visceral cavity views. As can be seen at 308, a normalized and straightened view of the ribs is displayed while maintaining relative rib lengths. Discontinuities between ribs, imaging artifacts arising from adjacent rib positions, incorrect rib lengths and invisible fractures, and distortions due to MIP are avoided.
[0044] Figure 4 A 3D view of the ribs and spine derived from a stack of generated manifold slices covering a complete 3D thorax visualization is illustrated. A 3D visualization of the thorax and vertebral body centers can be generated by shifting or moving the deformed 2D manifold plane multiple times along its normal direction, toward the inside and outside of the manifold plane and repeatedly sampling to generate a stack of 2D manifold slices. This results in a 3D image as a stack of manifolds covering the complete thorax. Figure 4 The resulting 3D image is shown allowing all ribs to be included as a whole. The generated 3D rendering is shown at 402. The view at 404 is rotated approximately thirty degrees, and the view at 406 is rotated slightly less than ninety degrees.
[0045] return Figure 2 , receiving the generated stack of manifold slices at the trained fracture detection module 206. The fracture detection module generates a modified 2D manifold view showing the predicted fractures in the space of the generated stack of manifold slices overlaying the complete 3D thorax visualization. Fracture prediction is performed using the trained fracture detection model.
[0046] The mapping function and the modified 2D manifold showing the predicted fracture are received at the inverse coordinate transformation module 208. The inverse coordinate transformation module 208 maps the predicted fracture from the stack of modified manifold slices back to the image space of the data representing the 3D diagnostic image. Thus, the predicted fracture can be visualized on the manifold view or mapped back to the original 3D diagnostic image.
[0047] It should be understood that Figure 2 The processes of each component (mapping module, manifold generation module, trained fracture detection module, inverse coordinate transformation module, and transformation module) shown in the figure can be implemented by a general or task-specific circuit system and / or a processor that executes computer-readable instructions encoded or embedded on a computer-readable storage medium such as a physical memory and other non-transient media. Additionally or alternatively, the processor may execute computer-readable instructions carried by carrier waves, signals, and other transient (or non-transient) media.
[0048] Figure 5 An example of a subtle rib fracture is shown in axial view 502, coronal view 504, and sagittal view 506. The fracture appears as a slight irregularity in the cortical bone and is best seen in the sagittal view plane 506. Figure 5 An example of the challenges encountered in achieving automatic bone fracture detection is shown. The fixed size and normalized rib coordinates provided by the new manifold view enable the described automatic rib fracture detection.
[0049] Figure 6 A rib visualization is illustrated that is simplified for inspection by a medical professional, such as a radiologist. A visceral cavity view is shown at 602 , a fillet view is shown at 604 , and a new manifold view is shown at 606 .
[0050] To address the trade-off between receptive field size and resolution, rib fracture detection is performed on reformatted images rather than working on a 3D reconstructed image volume utilizing a Cartesian grid. More specifically, a new manifold view shown at 606 but limited to the thorax is selected as input to the rib fracture detection model. Compared to the fillet view 604, sampling artifacts (e.g., rib copies) and discontinuities between ribs that could confuse a fracture detector are eliminated. Compared to the visceral cavity view 602, the ratio of rib length to rib position is consistent. In other words, the new manifold view has no sampling artifacts and has strong image edges while maintaining rib length and position.
[0051] Applying the new manifold view, automatic fracture detection shows significant improvements compared to earlier views (fillet view and visceral cavity). The fixed size of the new manifold view results in consistent fracture detection duration. Additional benefits of the proposed solution include: fast inference with potential single-shot deployment (i.e., no patch sampling during training and testing), increased sensitivity due to a large receptive field that captures potential fractures in adjacent ribs and accompanying symptoms (e.g., pneumothorax or hemothorax), a reduced number of false positives due to normalized coordinates (e.g., the ends of the ribs, the positions of the first / last ribs are predefined), and easier time comparison and inspection of subsequent scans due to normalized coordinates. Fracture predictions of previous scans can be superimposed on novel CT scans to investigate healing progress. Previous CT scans and predictions can also be considered during fracture prediction, and reasoning is restricted to the thorax, resulting in higher robustness against FOV changes.
[0052] return Figure 2 , components of the image processing apparatus include a transformation module 210. The transformation module 210 generates data as input to train a trained fracture detection model.
[0053] The stack of manifold slices generated at 204 and the mapping function generated at 202 are received at a transformation module 210. The transformation module 210 generates an annotated stack of manifold slices by repeating the sampling, generation, and shifting performed by the manifold processing circuitry but using a three-dimensional annotation mask or a three-dimensional annotation marker as a sampling input. The transformation module 210 then generates an annotation mask or a set of annotation markers defined in the generated stack of manifold slices by mapping annotated fractures from the three-dimensional image space to the three-dimensional annotated stack of manifold slices according to the mapping function. The trained fracture detection module receives the generated stack of manifold slices and the generated annotated stack of manifold slices as input to train the trained fracture detection model.
[0054] A learning database needs to be constructed with a collection of reconstructed 3D images in the form of stacks of manifold slices that are generated to cover the complete three-dimensional thorax visualization and corresponding target rib fracture annotations (e.g., in the form of label masks, bounding boxes, or point annotations). In an embodiment, data is collected via manual annotations. First, the manifold mapping is determined. For each case, the mapping module 202 takes the 3D image as input. Each rib centerline is extracted and labeled, and the manifold is determined using thin plate spline interpolation / smoothing, whereby the centerline position is mapped to a predefined position at the manifold generation module 204. The manifold generation module 204 delivers a coordinate mapping function as an output, which is used by the transformation module 210 to resample the image data and transform the corresponding 3D annotation mask of the 3D label. The shape of the resampled multi-slice 2D manifold is arbitrary, but fixed in all cases.
[0055] Convolutional Neural Networks (CNNs) are deep learning techniques that can be used to develop learning models. CNN architectures such as U-Net and EfficientDet can be adaptive types of learning models. Deep learning-based training using datasets from hundreds of scans can be used to build a learning database.
[0056] Due to the inherent properties of the 2D manifold stacking (e.g., fixed size, manifold smoothness, and normalized coordinates), different types of learning models can be tuned for dedicated applications on the images generated at the conversion module 201. Two different types of learning models can be distinguished, namely: (1) single-shot models (such as EfficientDet), where the receptive field captures the entire input volume, or (2) patch-based models, such as U-Net with local receptive fields, which process the volume patch by patch in a sliding window manner. In the case of patch-based methods, the patch and receptive field sizes can be tuned along the 2D manifold x and y axes so that fractures at adjacent ribs are captured. In addition, the position information of each patch can be provided as an additional input channel (e.g., in the form of a distance map). For any model used, the z-axis of the manifold can be processed as a channel axis or as an additional spatial image axis. Limiting the pooling operation to the x and y axes can further improve model performance due to low stack width (i.e., the number of voxels along the z dimension).
[0057] The data generated at the transformation module 210 can be used to continuously fine-tune the trained fracture detection model. In an embodiment, the trained fracture detection model is tuned to a predetermined threshold so that no fine-tuning is required. In other words, the learning database can be fully constructed so that no additional data is required. Therefore, in an embodiment, automatic fracture detection can be performed without the transformation module 210.
[0058] The predicted fracture regions may be displayed to the user as a list of markers or color layers on the 2D manifold view, and the number of predicted fractures per rib may be determined and displayed to the user. This may provide the user with an indication of the stability of the ribs. The inverse coordinate transformation module 208 receives as input the mapping function provided by the mapping module 202 and the model predictions on the manifold stack generated at 206. It maps the model predictions back to the original 3D image space for further visualization.
[0059] Figure 7704 is a flow chart representing a method for automatically detecting rib fractures in a 3D diagnostic image. As an example, a 3D image can be generated from a plurality of imaging systems including CT, CT arms, SPECT-CT, MR-CT, PET-CT, and (MRI) systems. At 702, data representing a 3D diagnostic image is received. At 704, ribs and spine are segmented based on the received data representing the 3D diagnostic image, and at 706, rib centerlines and vertebral body center landmarks are detected and marked. This is Figure 3 At 708, each three-dimensional position of the rib centerline and vertebral body center landmarks corresponding to the received data representing the 3D diagnostic image is mapped to a 2D position on the defined 2D manifold plane. At 710, those 3D position coordinates missing on the defined 2D manifold plane are interpolated, and the 2D manifold plane is deformed so that it is aligned with the detected rib centerline and vertebral center landmarks in the 3D space of the diagnostic image. This can be done in Figure 3 304. In a preferred embodiment, thin plate splines are selected as the interpolation and smoothing technique.
[0060] At 712, a coordinate mapping function is generated. The coordinate mapping function corresponds to the mapped and interpolated positions from the 2D manifold space to the 3D image space (and vice versa). At 714, according to Figure 3 The mapping function shown at 306 samples the image intensity from the three-dimensional diagnostic image space at each coordinate defined by the deformed 2D manifold plane.
[0061] At 716, a reformatted image is generated as a 2D manifold slice based on the sampled image intensities, the 2D manifold slice showing a continuous and straightened visualization of the thorax and spine. The generated 2D manifold slice is Figure 6 At 718, the deformed 2D manifold plane is shifted along its normal direction and the sampling is repeated as the 2D manifold plane is incrementally shifted along its normal direction to generate a stack of manifold slices covering the complete 3D thorax visualization, as shown in FIG. Figure 4 shown.
[0062] At 720, fracture detection is performed via a trained fracture detection model using machine learning or deep learning techniques. A stack of generated manifold slices is received as input, and at 722, a modified 2D manifold view is generated in the space of the stack of generated manifold slices according to the trained fracture detection model. Thus, the modified 2D manifold view overlays a complete 3D thorax visualization, wherein the predicted fractures are mapped to the modified 2D manifold view. The predicted fractures may be displayed to the user as a list of landmarks / bounding boxes or color layers on the modified 2D manifold view, and the number of predicted fractures per rib may be displayed to the user.
[0063] A modified 2D manifold view showing the predicted fracture and the mapping function are received at 724. The predicted fracture from the stack of modified manifold slices is mapped back to the image space of the 3D diagnostic image at 726. Thus, the fracture prediction can be mapped back to the original image space for further review by a radiologist.
[0064] While the invention has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive; the invention is not limited to the disclosed embodiments.
[0065] Other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims.
[0066] In the claims, the word "comprising" does not exclude other elements or steps, and the word "a" or "an" does not exclude a plurality.
[0067] A single processor, device or other unit may fulfill the functions of several items recited in the claims.The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.
[0068] Operations like acquiring, determining, obtaining, outputting, providing, storing or storing, calculating, simulating, receiving, warning and stopping may be implemented as program code elements of a computer program and / or dedicated hardware.
[0069] The computer program product may be stored / distributed on an appropriate medium, such as an optical storage medium or a solid-state medium, provided together with or as a part of other hardware, but may also be published in other forms, such as via the Internet or other wired or wireless telecommunication systems.
Claims
1. An image processing device, comprising: A mapping function module, which includes: A mapping function circuit system is configured to: receiving data representing a three-dimensional diagnostic image; segmenting the ribs and spine based on the received data representing the three-dimensional diagnostic image; Detect and mark rib centerlines based on rib segmentation; Detect and mark vertebral body center landmarks based on spine segmentation; mapping each three-dimensional position of the rib centerline and the vertebral body center landmark corresponding to the received data representing the three-dimensional diagnostic image to a two-dimensional position on a defined two-dimensional manifold plane; interpolating missing three-dimensional position coordinates on the defined two-dimensional manifold plane, deforming the two-dimensional manifold plane so that it aligns with the detected rib centerlines and vertebral center landmarks in the three-dimensional space of the diagnostic image; and generating a coordinate mapping function corresponding to the mapped and interpolated positions from the two-dimensional manifold space to the three-dimensional image space and vice versa; A manifold generation module, the manifold generation module comprising: A manifold processing circuit system, the manifold processing circuit system being configured to: receiving the generated coordinate mapping function; sampling image intensities from the three-dimensional diagnostic image space at each coordinate defined by the deformed two-dimensional manifold plane according to the mapping function; generating a reformatted image as a two-dimensional manifold slice from the sampled image intensities, the two-dimensional manifold slice showing a continuous and straightened visualization of the thorax and spine; and shifting the deformed two-dimensional manifold plane along its normal direction and repeating the sampling to generate a stack of manifold slices covering a complete three-dimensional chest visualization; A trained fracture detection module, which includes: A fracture prediction circuit system is configured to: receiving as input a stack of generated manifold slices; performing fracture prediction via a trained fracture detection model using machine learning or deep learning techniques; and generating a modified two-dimensional manifold view showing predicted fractures in the space of the generated stack of manifold slices covering the complete three-dimensional chest visualization according to the trained fracture detection model; and The inverse coordinate transformation module includes: An inverse coordinate transformation circuit system is configured to: receiving as input the mapping function and the modified two-dimensional manifold view showing the predicted fracture; and The predicted fracture is mapped from the stack of the manifold slices back into the image space of the data representing the three-dimensional diagnostic image.
2. The image processing device according to claim 1, further comprising a transformation module, wherein the transformation module comprises: A conversion circuit system, the conversion circuit system is configured to: receiving as input the stack of generated manifold slices and the mapping function; generating an annotated stack of manifold slices by repeating sampling, generation, and shifting by the manifold processing circuitry using a three-dimensional annotation mask or a three-dimensional annotation marker as a sampling input; as well as An annotation mask or set of annotation landmarks defined in the generated stack of manifold slices is generated by mapping the annotated fractures from the three-dimensional image space to the three-dimensional annotated stack of manifold slices according to the mapping function.
3. The image processing device according to claim 2, wherein: The trained fracture detection module receives as input the stack of generated manifold slices and the annotated stack of generated manifold slices to train the trained fracture detection model.
4. The image processing device according to claim 3, wherein: The trained fracture detection model is trained based on a slice-by-slice single-shot or patch-based model.
5. The image processing device according to claim 1, wherein: The predicted fractures are shown in the stack of manifold slices or in the space of the diagnostic image as a list of markers or voxel-by-voxel annotations with corresponding color layers, and wherein the number of predicted fractures per rib is determined and shown in the stack of manifold slices or in the space of the diagnostic image.
6. The image processing device according to claim 4, wherein: The receptive field sizes are tuned along the stack of defined manifold slices such that fractures at adjacent ribs are captured during training of the trained fracture detection model.
7. The image processing device according to claim 1, wherein: The predicted fractures were then analyzed by a classification model.
8. The image processing device according to claim 3, wherein: Additional inputs to the trained fracture detection module include scanned images and annotations from previous imaging scans.
9. The image processing apparatus according to claim 1, wherein: Interpolating missing coordinates on the manifold is performed via an interpolation technique including thin plate splines.
10. A method for automatically detecting rib fractures in a three-dimensional diagnostic image, the method comprising: receiving data representing a three-dimensional diagnostic image; segmenting the ribs and spine based on the received data representing the three-dimensional diagnostic image; The rib centerline is not detected and marked according to the rib segmentation; Detect and mark vertebral body center landmarks based on spine segmentation; mapping each three-dimensional position of the rib centerline and the vertebral body center landmark corresponding to the received data representing the three-dimensional diagnostic image to a two-dimensional position on a defined two-dimensional manifold plane; interpolating missing three-dimensional position coordinates on the defined two-dimensional manifold plane, which deforms the two-dimensional manifold plane so that it is aligned with the detected rib centerlines and vertebral center landmarks in the three-dimensional space of the diagnostic image; generating a coordinate mapping function corresponding to the mapped and interpolated positions from the two-dimensional manifold space to the three-dimensional image space and vice versa; sampling image intensities from the three-dimensional diagnostic image space at each coordinate defined by the deformed two-dimensional manifold plane according to the mapping function; generating a reformatted image as a two-dimensional manifold slice from the sampled image intensities, the two-dimensional manifold slice showing a continuous and straightened visualization of the thorax and spine; shifting the deformed two-dimensional manifold plane along its normal direction and repeating the sampling to generate a stack of manifold slices covering a complete three-dimensional chest visualization; receiving as input a stack of generated manifold slices; Performing fracture prediction via a trained fracture detection model using machine learning or deep learning techniques; generating a modified two-dimensional manifold view showing predicted fractures in the space of a stack of manifold slices generated to cover a complete three-dimensional chest visualization according to the trained fracture detection model; receiving as input the mapping function and the modified two-dimensional manifold view showing the predicted fracture; and The predicted fracture is mapped from the stack of the manifold slices back into the image space of the data representing the three-dimensional diagnostic image.
11. The method for automatically detecting rib fractures in a three-dimensional diagnostic image according to claim 10, further comprising: receiving as input the stack of generated manifold slices and the mapping function; generating an annotated stack of manifold slices by repeating sampling, generation, and shifting by the manifold processing circuitry using a three-dimensional annotation mask or a three-dimensional annotation marker as a sampling input; and An annotation mask or set of annotation landmarks defined in the generated stack of manifold slices is generated by mapping annotated fractures from the three-dimensional image space to the three-dimensional annotated stack of manifold slices according to the mapping function.
12. The method for automatically detecting rib fractures in three-dimensional diagnostic images according to claim 11, further comprising receiving a stack of generated manifold slices and annotated stack of generated manifold slices as input to train the trained fracture detection model.
13. The method for automatically detecting rib fractures in three-dimensional diagnostic images of claim 12, further comprising training the trained fracture detection model based on a slice-by-slice single-shot or patch-based model.
14. The method for automatically detecting rib fractures in a three-dimensional diagnostic image according to claim 10, further comprising displaying the predicted fractures as a list of markers or voxel-by-voxel annotations with corresponding color layers in the stack of the manifold slices or the space of the diagnostic image, and displaying the number of predicted fractures for each rib.
15. The method for automatically detecting rib fractures in three-dimensional diagnostic images according to claim 13, wherein: The receptive field sizes are tuned along the stack of defined manifold slices such that fractures at adjacent ribs are captured during training of the trained fracture detection model.
16. The method for automatically detecting rib fractures in three-dimensional diagnostic images according to claim 10, wherein: The predicted fractures were then analyzed by a classification model.
17. The method for automatically detecting rib fractures in three-dimensional diagnostic images of claim 12, further comprising receiving scanned images and annotations from previous imaging scans as additional inputs to the trained fracture detection module.
18. The method for automatically detecting rib fractures in three-dimensional diagnostic images according to claim 10, wherein: Interpolating missing coordinates on the manifold is performed via an interpolation technique including thin plate splines.
19. A non-transitory computer-readable medium having stored thereon instructions for causing processing circuitry to perform a process comprising: receiving data representing a three-dimensional diagnostic image; segmenting the ribs and spine based on the received data representing the three-dimensional diagnostic image; Detect and mark rib centerlines based on rib segmentation; Detect and mark vertebral body center landmarks based on spine segmentation; mapping each three-dimensional position of the rib centerline and the vertebral body center landmark corresponding to the received data representing the three-dimensional diagnostic image to a two-dimensional position on a defined two-dimensional manifold plane; interpolating missing three-dimensional position coordinates on the defined two-dimensional manifold plane, which deforms the two-dimensional manifold plane so that it is aligned with the detected rib centerlines and vertebral center landmarks in the three-dimensional space of the diagnostic image; generating a coordinate mapping function corresponding to the mapped and interpolated positions from the two-dimensional manifold space to the three-dimensional image space and vice versa; sampling image intensities from the three-dimensional diagnostic image space at each coordinate defined by the deformed two-dimensional manifold plane according to the mapping function; generating a reformatted image from the sampled image intensities as a two-dimensional manifold slice showing a continuous and straightened visualization of the thorax and spine; shifting the deformed two-dimensional manifold plane along its normal direction and repeating the sampling to generate a stack of manifold slices covering a complete three-dimensional chest visualization; receiving as input a stack of generated manifold slices; Performing fracture prediction via a trained fracture detection model using machine learning or deep learning techniques; generating a modified two-dimensional manifold view showing predicted fractures in the space of a stack of manifold slices generated to cover a complete three-dimensional chest visualization according to the trained fracture detection model; receiving as input the mapping function and the modified two-dimensional manifold view showing the predicted fracture; and The predicted fracture is mapped from the stack of the manifold slices back into the image space of the data representing the three-dimensional diagnostic image.
