Automatic rib fracture detection from unfolded scan images

Abstract

A method and apparatus for automatically detecting fractures from imaging scans is disclosed. An innovative manifold view that combines the advantages of previous approaches while avoiding their limitations is combined with a deep learning based fracture detection model. Ribs and vertebrae from the 3D scan are segmented and rib centerlines and vertebral central landmarks are detected and labeled. 3D position coordinates corresponding to the received data are mapped to a defined 2D manifold plane, followed by generating 2D manifold slices and corresponding mapping functions by interpolation and sampling techniques. A 3D visualization of the rib cage is generated from the multiple manifold slices. A trained fracture prediction model receives the multiple manifold slices and generates a modified stack of 2D manifold slices showing the predicted fractures. The predicted fractures can be displayed in the 2D manifold slices or mapped back into the image space of the original 3D scan.

Description

[Technical field]

[0001] The present invention relates to the field of medical image processing, and more particularly to a method and apparatus for automatically detecting rib fractures from medical imaging scans. [Background technology]

[0002] Medical professionals typically obtain medical images to identify fractures in a patient's thorax. Medical imaging, such as computed tomography (CT) scans, is a tool used to visualize a patient's anatomy, and the visualization data is used to diagnose disease or injury. Particularly at the scene of trauma, medical professionals may rely on medical imaging data in the form of CT scans for diagnostic purposes (e.g., for detection of rib fractures). Interpretation of imaging scans, and more specifically trauma or emergency scans, is a time-demanding task that requires meticulous attention to avoid missing important findings. In some cases where rib fractures are not evident, examination of the ribs from different perspectives (e.g., sagittal and coronal views instead of axial views) and examination of neighboring ribs and lung areas for potential indications is required, thus further lengthening the examination time.

[0003] Diagnosis of rib fractures requires that 24 individual ribs be tracked where each rib spans multiple slices of the imaging scan. Thus, manual inspection is a laborious and time-consuming process. While reliable detection and diagnosis of rib fractures is difficult, different types of fractures may present different challenges in detecting and diagnosing rib fractures (e.g., microfractures appear as slight irregularities in the cortical bone).

[0004] In the last few years, several approaches for automatic detection of rib fractures and methods for intuitive visualization of the patient's thorax for simplified human examination have been introduced, however, these approaches separate automatic fracture detection and visualization.

[0005] Several approaches have been proposed to simplify the visualization and assessment of a patient's anatomy and critical structures, particularly those targeted at visualization of the thorax. Current approaches, while offering some advantages, have clear fundamental limitations.

[0006] One well-known visualization scheme is the "filet view" or "fishbone view". This view is based on segmenting the ribs (e.g., using a deep convolutional neural network) and then using a centerline extractor that successively labels rib pairs in the field of view. Each rib is sampled along its trajectory, allowing visualization of each and every rib in a normalized, linearized manner (curved surface reformatted). One drawback of this type of view is that the nature of treating each rib separately introduces discontinuities between ribs, and imaging artifacts from adjacent ribs may appear in the rib shapes.

[0007] Another visualization scheme is the visceral cavity view. In this view, a segmentation algorithm (e.g., using a model-based approach) is applied to segment the interior of the thorax in terms of deformation of a cylindrical manifold (manifold). Once segmentation is performed, the manifold can be unwound and a maximum intensity projection (MIP) close to the surface can be calculated. This view allows the user to inspect the thorax as a whole in terms of a continuous visualization on the inspection canvas. One drawback of this type of view is that the relative rib lengths are not maintained. The nature of unwrapping the cylindrical manifold does not allow visualization of the correct rib lengths. For example, the first rib appears excessively long compared to the other ribs. In addition, the nature of the MIP visualization may not allow detection of subtle fractures. For example, the MIP visualization may make small rib fractures invisible and undetectable in the generated view. Another drawback is that this view adds significant unrealistic distortions (wavy ribs), which limits the clinical reliability of the generated visceral cavity view.

[0008] There is a great demand for automated detection of rib fractures from medical images such as CT images, but automated detection of rib fractures is a challenging task. Common deep learning-based approaches suffer from a trade-off between input resolution and receptive field size due to memory and execution time constraints. High image resolution (e.g., 1 mm isotropic) is required to identify subtle rib fractures such as hairline cracks. However, since rib fractures frequently occur in similar parts of adjacent ribs, a large receptive field size is equally beneficial for the task. Furthermore, it is difficult to identify the location of rib fractures from high-resolution local patches. Irregular parts of the ribs (e.g., rib tubercles or sternal rib ends), especially in the first and last ribs, can be misinterpreted as bulge fractures or displaced fractures. Current approaches seem to operate on the orientation of the original image and the extracted local patches that are then fed into a convolutional neural network. It is difficult for these approaches to take into account the inherent difficulties of the task. Summary of the Invention [Problem to be solved by the invention]

[0009] Thus, a need exists for an innovative approach to automatically detect rib fractures from medical imaging scans.

[0010] It is an object of the present invention to provide a method and apparatus for automatically detecting rib fractures from medical imaging scans. The technique can be applied to scans produced by multiple imaging systems, including CT, CT Arm, Single Photon Emission Computed Tomography CT (SPECT-CT), Magnetic Resonance CT (MR-CT), Positron Emission Tomography CT (PET-CT), and Magnetic Resonance Imaging (MRI) systems. [Means for solving the problem]

[0011] To address the trade-off between receptive field size and resolution, the technique performs rib fracture detection on reformatted images instead of operating on a three-dimensional (3D) reconstructed image volume in a Cartesian coordinate system. A new manifold view is introduced as input for the rib fracture detection model.

[0012] The new manifold view offers advantages over existing views. For example, in contrast to the "Fissure View", there are no sampling artifacts (e.g., rib overlaps) and there are strong image edges (e.g., at the border between adjacent ribs or background) eliminating imaged features that may confuse the fracture detector. Compared to the visceral cavity view, rib length ratios remain unchanged and rib positions are consistent. It is worth noting that the new manifold view is again a three-dimensional volume showing only the rib cage in a standardized and significantly smaller image.

[0013] According to a first aspect of the present invention, there is provided an image processing apparatus, the image processing apparatus having a mapping function module including a mapping function circuit configured to: receive data representing a three-dimensional diagnostic image; segment ribs and vertebrae according to the received data representing the three-dimensional diagnostic image; detect and label rib centerlines from the rib segmentation; detect and label central landmarks of vertebrae from the vertebral segmentation; map each three-dimensional location of the rib centerlines and vertebral central landmarks corresponding to the received data representing the three-dimensional diagnostic image to a two-dimensional location on a defined two-dimensional manifold plane; interpolate missing three-dimensional location coordinates on the defined two-dimensional manifold plane, the two-dimensional manifold transforming the two-dimensional manifold such that the two-dimensional manifold aligns with the detected rib centerlines and vertebral central landmarks in the three-dimensional space of the diagnostic image; and generate a coordinate mapping function corresponding to the mapped and interpolated locations from the two-dimensional manifold space to the three-dimensional image space and vice versa.

[0014] The image processing device has a manifold generation module including a manifold processing circuit configured to: receive the generated coordinate mapping function; sample image intensities from the three-dimensional diagnostic image space at each coordinate defined by the deformed two-dimensional manifold plane in accordance with the mapping function; generate from the sampled image intensities a reformatted image as two-dimensional manifold slices representing a continuous, linearized visualization of the rib cage and spine; and shift the deformed two-dimensional manifold plane along its normal direction and repeat the sampling to generate a stack of manifold slices covering the complete three-dimensional rib cage visualization.

[0015] The image processing device has a trained fracture detection module including a fracture prediction circuit configured to: receive as input a stack of generated manifold slices; perform fracture prediction via a trained fracture detection model using machine learning or deep learning techniques; and generate a modified two-dimensional manifold view showing predicted fractures within the space of the stack of generated manifold slices covering the complete three-dimensional thorax visualization according to the trained fracture detection model.

[0016] The image processing device also has an inverse coordinate transformation module including an inverse coordinate transformation circuit configured to: receive as input the mapping function and the modified two-dimensional manifold view indicating the predicted fracture; and map the predicted fracture back from the stack of manifold slices into the image space of data representing the three-dimensional diagnostic image.

[0017] In a second aspect of the present invention, a method for automatically detecting rib fractures is provided.The method includes the steps of: receiving data representing a three-dimensional diagnostic image; segmenting the ribs and vertebrae according to the received data representing the three-dimensional diagnostic image; detecting and labeling rib centerlines from the rib segmentation; detecting and labeling central landmarks of vertebrae from the vertebral segmentation; mapping each three-dimensional location of the rib centerlines and vertebral central landmarks corresponding to the received data representing the three-dimensional diagnostic image to a two-dimensional location on a defined two-dimensional manifold plane; interpolating missing three-dimensional location coordinates on the defined two-dimensional manifold plane, transforming the two-dimensional manifold plane such that the plane aligns with the detected rib centerlines and vertebral central landmarks in the three-dimensional space of the diagnostic image; generating a coordinate mapping function from the two-dimensional manifold space to the three-dimensional image space and vice versa, corresponding to the mapped and interpolated locations; receiving the generated coordinate mapping function; sampling image intensities from the three-dimensional diagnostic image space at each coordinate defined by the transformed two-dimensional manifold plane according to the mapping function. the transformed two-dimensional manifold plane being transformed into a manifold slice that represents a continuous, linearized visualization of the rib cage and spine; shifting the transformed two-dimensional manifold plane along its normal direction and repeating the sampling to generate a stack of manifold slices that cover the complete three-dimensional rib cage visualization; receiving the generated stack of manifold slices as input to a trained fracture detection model; performing fracture prediction via a fracture detection model trained according to machine learning or deep learning techniques; generating a modified two-dimensional manifold view showing predicted fractures in the space of the generated stack of manifold slices that cover the complete three-dimensional rib cage visualization according to the trained fracture detection model; receiving a mapping function and the modified two-dimensional manifold view showing the predicted fractures; and mapping the predicted fractures from the stack of manifold slices back into the image space of data representing the three-dimensional diagnostic image.

[0018] In a third aspect of the present invention, a non-transitory computer readable medium having stored thereon instructions for causing a processing circuit to perform a process is provided.The process includes the steps of: receiving data representing a three-dimensional diagnostic image; segmenting the ribs and vertebrae according to the received data representing the three-dimensional diagnostic image; detecting and labeling rib centerlines from the rib segmentation; detecting and labeling central landmarks of vertebrae from the vertebral segmentation; mapping each three-dimensional location of the rib centerlines and vertebral central landmarks corresponding to the received data representing the three-dimensional diagnostic image to a two-dimensional location on a defined two-dimensional manifold plane; interpolating missing three-dimensional location coordinates on the defined two-dimensional manifold plane, transforming the two-dimensional manifold plane such that the plane aligns with the detected rib centerlines and vertebral central landmarks in the three-dimensional space of the diagnostic image; generating a coordinate mapping function from the two-dimensional manifold space to the three-dimensional image space and vice versa, corresponding to the mapped and interpolated locations; receiving the generated coordinate mapping function; sampling image intensities from the three-dimensional diagnostic image space at each coordinate defined by the transformed two-dimensional manifold plane according to the mapping function. the transformed two-dimensional manifold plane along its normal direction and repeated sampling to generate a stack of manifold slices covering the complete three-dimensional thorax visualization; receiving the generated stack of manifold slices as input to a trained fracture detection model; performing fracture prediction via the trained fracture detection model according to machine learning or deep learning techniques; generating a modified two-dimensional manifold view showing predicted fractures in the space of the generated stack of manifold slices covering the complete three-dimensional thorax visualization according to the trained fracture detection model; receiving a mapping function and the modified two-dimensional manifold view showing the predicted fractures; and mapping the predicted fractures from the stack of manifold slices back to the image space of data representing the three-dimensional diagnostic image.

[0019] In a preferred embodiment, the transformation module includes a transformation circuit configured to: receive as input the stack of generated manifold slices and a mapping function; generate a stack of annotated manifold slices by iteratively sampling, generating and shifting the manifold processing circuit using the three-dimensional annotation mask or the three-dimensional annotation landmarks as sampling inputs; and generate a set of annotation masks or annotation landmarks defined in the stack of generated manifold slices by mapping the annotated fractures from the three-dimensional image space to the stack of three-dimensional annotated manifold slices according to the mapping function. The trained fracture detection module receives the stack of generated manifold slices and the stack of generated annotated manifold slices as inputs for training a trained fracture detection model.

[0020] In one aspect of the present invention, the trained fracture detection module receives the generated stack of manifold slices and the generated stack of annotated manifold slices as inputs for training the trained fracture detection model. In a preferred embodiment, the trained fracture detection model is trained according to a slice-wise single shot or patch-based model. The size of the receptive field is adjusted in the defined stack of manifold slices such that fractures in adjacent ribs are captured during training of the trained fracture detection model.

[0021] According to another aspect of the invention, the predicted fractures are shown as a list of landmarks or voxel-wise annotations with corresponding colored overlays in the space of the stack of manifold slices or in the space of the diagnostic image. Further, multiple predicted fractures per rib are determined and shown in the space of the stack of manifold slices or in the space of the diagnostic image.

[0022] In another embodiment of the invention, the predicted fractures are subsequently analyzed by a classification model.

[0023] In a preferred embodiment, the interpolation of missing coordinates on the two-dimensional manifold is performed via an interpolation technique that includes thin-plate splines. In one aspect of the invention, additional inputs to the trained fracture detection module include scan images and annotations from previous imaging scans. [Brief description of the drawings]

[0024] [Figure 1] FIG. 1 is a schematic diagram of an exemplary imaging system. [Diagram 2] FIG. 2 illustrates components of an imaging processor according to one embodiment. [Diagram 3] FIG. 3 illustrates a processing workflow according to one embodiment. [Figure 4] FIG. 4 shows a 3D view of the ribs and spine according to one embodiment. [Diagram 5] Figure 5 shows an example of a minor rib fracture. [Figure 6] FIG. 6 shows a visceral cavity view of the thorax, a fissure review of the thorax, and a new manifold view of the thorax. [Figure 7] FIG. 7 is a flow chart illustrating a method for automatic fracture detection according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] 1 illustrates an imaging system 100, such as a computed tomography (CT) system. The imaging system 100 includes a generally stationary gantry 102 and a rotating gantry 104. The rotating gantry 104 is rotatably supported by the stationary gantry 102 and rotates relative to the stationary gantry 102 about a longitudinal or Z-axis about an examination region 106.

[0026] A patient support 112, such as a couch, supports an object or subject, such as a human patient, within the examination region 106. The support 112 is configured to move the object or subject for loading, scanning, and exiting the object or subject.

[0027] A radiation source 108, such as an x-ray tube, is rotatably supported by the rotating gantry 104. The radiation source 108 rotates with the rotating gantry 104 and emits radiation that traverses an examination region 106.

[0028] A radiation sensitive detector array 110 arcs at an angle across the examination region 106 opposite the radiation source 108. The detector array 110 includes one or more rows of detectors extending along a z-axis direction to detect radiation passing across the examination region 106 and generate projection data indicative thereof.

[0029] A general purpose computing system or computer serves as an operator console 114 and includes input devices 116, such as a mouse, keyboard, etc., and output devices 120, such as a display monitor, film, etc. The console 114 allows an operator to control the operation of the system 100, including control of an image processor 118, which receives data representative of three-dimensional (3D) diagnostic images produced by the imaging system 100 and generates reformatted images that display the predicted rib fractures within the image space of the three-dimensional (3D) diagnostic images. The reformatted images may be output to the output device 120 for interpretation by a radiologist or other medical personnel.

[0030] The processing of the image processing device 118 may be implemented through a processor and / or processing circuitry that is understood to execute computer-readable instructions, such as executable code, encoded or embedded in a computer-readable storage medium, such as physical memory or other non-transitory medium. Additionally or alternatively, the processor and / or processing circuitry may execute computer-readable instructions conveyed by carrier waves, signals, and other transitory (or non-transitory) media.

[0031] FIG. 2 illustrates some components of the image processing device 118 according to some embodiments. In the mapping module 202, a three-dimensional (3D) diagnostic image generated by an imaging system, for example the CT imaging system 100, is received. The ribs and vertebrae are automatically segmented. The segmentation can be performed using machine learning or deep learning techniques. As an example, convolutional neural networks (CNNs) are a deep learning technique that can be used to segment the ribs and vertebrae. CNN architectures such as U-Net and V-Net can be adapted for the segmentation task. Deep learning-based training using datasets from hundreds of scans can be used to develop algorithms that are shown to achieve fast and accurate segmentation of the ribs and vertebrae. Ribs are detected, rib centerlines are extracted and labeled, and vertebral center landmarks are detected and labeled. Each three-dimensional (3D) position of the rib centerlines and vertebral center landmarks corresponding to the received data representing the 3D diagnostic image is mapped to a two-dimensional (2D) position on a defined two-dimensional manifold plane.

[0032] Since the positions of the ribs and vertebrae are known in the 3D space, the correlation of all rib centerlines and all vertebral landmarks can be established. That is, the correspondence of coordinates from 3D space to 2D space can be defined as a mapping function. When the 3D to 2D correspondence is established or set, the correspondence or correlation from the remaining pixels in the defined 2D space will be missing. The missing 3D position coordinates on the defined 2D manifold plane are interpolated, thereby deforming the two-dimensional manifold plane to match the rib centerlines and vertebral center landmarks detected in the 3D space of the diagnostic image. Finally, a coordinate mapping function is generated that corresponds to the mapped and interpolated positions from the 2D manifold space to the 3D image space and vice versa.

[0033] The generated mapping function is received at the manifold generation module 204. Image intensities from the 3D diagnostic image space are sampled at each coordinate defined by a 2D manifold plane that has been transformed according to the received mapping function. Reformatted images are generated from the sampled image intensities as 2D manifold slices that display a continuous linearized visualization of the rib cage and spine. A stack of manifold slices is generated by shifting the transformed 2D manifold plane along its normal direction toward the inside and outside of the manifold plane, and repeating the sampling for each incremental shift along the normal direction.

[0034] The process of generating a new manifold view is shown in FIG. 3. At 302, rib centerlines and vertebral center landmarks are detected and labeled according to an automatic segmentation scheme. As mentioned above, the automatic segmentation scheme can be based on machine learning or deep learning techniques. A 2D manifold is defined that will represent the resulting new manifold view. At 304, an interpolation scheme deforms the 2D manifold to align with the detected rib centerlines and vertebral center landmarks. The interpolation scheme can be a thin plate spline or any suitable interpolation scheme. As shown at 304, the defined 2D manifold is deformed to align with the detected rib centerlines and vertebral center landmarks mapped from the 3D space to the 2D space of the 2D deformation manifold.

[0035] Image intensities from the 3D image space are sampled at each coordinate of the deformed 2D manifold to complete or fill in the missing 3D coordinates. The result is shown at 306. The missing 3D coordinates may correspond to regions of tissue between the ribs and regions of tissue that occlude the ribs, or any combination of regions of tissue between the ribs and regions that occlude the ribs. This allows the deformed 2D manifold to be unfolded.

[0036] The unfolded 2D manifold is shown at 308. The new manifold view shown at 308 combines the advantages of both the physeal and visceral cavity views while avoiding the disadvantages of both. At 308, it can be seen that a normalized and straightened view of the ribs is displayed while maintaining relative rib lengths. Inter-rib discontinuities, imaging artifacts appearing from adjacent rib positions, incorrect rib lengths, and unseen fractures and distortions due to MIP are avoided.

[0037] FIG. 4 shows a 3D view of the ribs and spine derived from a stack of generated manifold slices covering a complete three-dimensional rib visualization. A 3D visualization of the rib cage and vertebral centers can be generated by shifting or moving the deformed 2D manifold plane along its normal direction multiple times toward the inside and outside of the manifold plane and repeating the sampling to generate a stack of 2D manifold slices. As a result, a 3D image is generated as a stack of manifolds covering the complete rib cage. FIG. 4 shows that the resulting 3D image allows all ribs to be included in their entirety. The generated 3D rendering is shown in 402. The view in 404 is rotated about 30 degrees, and the view in 406 is rotated slightly less than 90 degrees.

[0038] Returning to Figure 2, the stack of generated manifold slices is received by a trained fracture detection module 206, which generates a modified 2D manifold view showing predicted fractures within the space of the stack of generated manifold slices covering the complete 3D rib cage visualization. The prediction of fractures is performed using a trained fracture detection model.

[0039] The mapping function and the modified 2D manifold indicative of the predicted fractures are received at an inverse coordinate transformation module 208, which maps the predicted fractures from the stack of modified manifold slices back to the image space of data representing the 3D diagnostic image. In this manner, the predicted fractures can be visualized on the manifold view or mapped back to the original 3D diagnostic image.

[0040] It should be understood that the processing of each of the components shown in Figure 2, namely the mapping module, the manifold generation module, the trained fracture detection module, the inverse coordinate transformation module, and the transformation module, can be implemented via general-purpose or task-specific circuitry and / or a processor executing computer-readable instructions encoded or embedded in computer-readable storage media, such as physical memory and other non-transitory media. Additionally or alternatively, such processors may execute computer-readable instructions conveyed by carrier waves, signals, and other transitory (or non-transitory) media.

[0041] Figure 5 shows an example of a subtle rib fracture in axial 502, coronal 504, and sagittal 506 views. The fracture appears as a slight irregularity in the cortical bone and is best seen in the sagittal view plane 506. Figure 5 shows an example of the difficulties faced when implementing automated fracture detection. The constant dimensions and normalized rib coordinates provided by the new manifold views enable the described automated rib fracture detection.

[0042] 6 shows a simplified visualization of the ribs for review by a medical professional such as a radiologist. The visceral cavity view is shown at 602, the fibrous view is shown at 604, and the new manifold view is shown at 606.

[0043] To solve the compromise problem between receptive field size and resolution, rib fracture detection is performed on the reformatted image instead of on the 3D reconstructed image volume with a Cartesian coordinate grid. More precisely, a new manifold view shown in 606, limited to the rib cage, is selected as input for the rib fracture detection model. In contrast to the physeal view 604, sampling artifacts (e.g. rib overlaps) and discontinuities between ribs that may confuse the fracture detector are removed. Compared to the visceral cavity view 602, the rib length ratio and rib position are consistent. In other words, the new manifold view has no sampling artifacts and has strong image edges while preserving the rib lengths and positions.

[0044] Applying the new manifold view, automated fracture detection shows a significant improvement compared to previous views (fissure view and visceral cavity view). The fixed size of the new manifold view results in consistent fracture detection times. Additional advantages of the proposed solution include faster inference due to possible single-shot deployment (i.e., no patch sampling during training and examination), improved sensitivity due to a wide receptive field that captures potential fractures in adjacent ribs and associated symptoms (e.g., pneumothorax or hemothorax), reduced number of false positives due to normalized coordinates (e.g., rib ends, first / last rib positions are predefined), and easier temporal comparison and examination of follow-up scans due to normalized coordinates. Fracture predictions from previous scans can be overlaid on the new CT scan to examine the healing progress. Previous CT scans and predictions can also be considered during fracture prediction, improving robustness against FOV variations as inference is limited to the rib cage.

[0045] Returning to Figure 2, the image processing apparatus components include a transformation module 210. The transformation module 210 generates data as input for training the trained fracture detection model.

[0046] The stack of manifold slices generated at 204 and the mapping function generated at 202 are received at a transformation module 210, which uses the three-dimensional annotation mask or the three-dimensional annotation landmarks as a sampling input, but generates an annotated stack of manifold slices by repeating the sampling, generating and shifting performed by the manifold processing circuitry. The transformation module 210 then generates an annotation mask or a set of annotation landmarks defined on the generated stack of manifold slices by mapping the annotated fractures from the three-dimensional image space to the three-dimensional annotated stack of manifold slices according to the mapping function. The generated stack of manifold slices and the generated annotated stack of manifold slices are received as inputs for training a trained fracture detection model by a trained fracture detection module.

[0047] A set of reconstructed 3D images in the form of a generated stack of manifold slices covering the complete three-dimensional rib cage visualization, and corresponding target rib fracture annotations (e.g., in the form of label masks, bounding boxes, or point annotations) are required to build a training database. In one embodiment, the data is collected via manual annotation. First, a manifold mapping is determined. For each case, the mapping module 202 takes the 3D images as input. The centerlines of the individual ribs are extracted and labeled, and a manifold is determined using a thin-plate spline interpolation / smoothing method, whereby the centerline locations are mapped to predefined locations in the manifold generation module 204. The manifold generation module 204 provides as output a coordinate mapping function, which is used by the transformation module 210 to resample the image data and to transform the corresponding 3D annotation mask of the 3D labels. The shape of the resampled multi-slice 2D manifold is arbitrary, but is constant for all cases.

[0048] Convolutional Neural Networks (CNN) is a deep learning technique that can be used to develop learning models. CNN architectures such as U-Net and EfficientDet are examples of adaptive learning models. Deep learning-based training using datasets from hundreds of scans can be used to build a learning database.

[0049] The inherent properties of the 2D manifold stack (e.g., fixed size, smoothness of the manifold, and normalized coordinates) allow different types of learning models to be adapted for dedicated applications on the images generated by the transformation module 210. Two different kinds of learning models can be distinguished: (1) single-shot models such as EfficientDet, whose receptive fields capture the entire input volume, or (2) patch-based models such as U-Net with local receptive fields that process the volume in a sliding window manner, patch by patch. For patch-based approaches, the size of the patches and receptive fields can be adjusted along the x and y axes of the 2D manifold so that adjacent rib fractures are captured. Furthermore, position information of each patch can be provided as an additional input channel, for example in the form of a distance map. For any model used, the z-axis of the manifold can be treated either as a channel axis or as an additional spatial image axis. Restricting the pooling operation to the x and y axes further improves the performance of the model due to the small stack width (i.e., the number of voxels along the z dimension).

[0050] The trained fracture detection model may be continually fine-tuned using the data generated in the transformation module 210. In one embodiment, the trained fracture detection model is adjusted to a pre-defined threshold such that fine-tuning is no longer required. In other words, the learning database may be built up until it is sufficient such that no additional data is required. Thus, in one embodiment, automatic fracture detection may be performed without the transformation module 210.

[0051] The predicted fracture regions can be shown to the user as a list of landmarks or a colored overlay on the 2D manifold view, and multiple predicted fractures per rib can be determined and displayed to the user, which can provide the user with an indication of rib stability. Inverse coordinate transformation module 208 receives as input the mapping function provided by mapping module 202 and the model predictions for the manifold stack generated in 206. Module 208 maps the model predictions back to the original 3D image space for further visualization.

[0052] FIG. 7 is a flow chart showing a method for automatically detecting rib fractures in a 3D diagnostic image. The 3D image can be generated from several imaging systems, including CT, CT arm, SPECT-CT, MR-CT, PET-CT and (MRI) systems as examples. Data representing a 3D diagnostic image is received at 702. At 704, the ribs and spine are segmented according to the received data representing the 3D diagnostic image, and at 706, rib centerline and vertebral center landmarks are detected and labeled. This is shown at 302 in FIG. 3. At 708, each three-dimensional position of the rib centerline and vertebral center landmark corresponding to the received data representing the 3D diagnostic image is mapped to a 2D position on a defined 2D manifold plane. The missing 3D position coordinates on the defined 2D manifold plane are interpolated at 710 to deform the 2D manifold plane so that the 2D manifold plane coincides with the detected rib centerline and vertebral center landmark in the 3D space of the diagnostic image. This is seen at 304 in FIG. 3. In the preferred embodiment, thin plate splines are chosen as the interpolation and smoothing technique.

[0053] At 712, a coordinate mapping function is generated that corresponds to the mapping and interpolated positions from the 2D manifold space to the 3D image space and vice versa. At 714, image intensities from the three-dimensional diagnostic image space are sampled according to the mapping function, as shown in 306 of FIG. 3, at each coordinate defined by the transformed 2D manifold plane.

[0054] At 716, reformatted images are generated from the sampled image intensities as 2D manifold slices that display a continuous, linearized visualization of the rib cage and spine. The generated 2D manifold slices are the new manifold views shown in 606 of Figure 6. At 718, the deformed 2D manifold plane is shifted along the normal direction, and the sampling is repeated as the 2D manifold plane is shifted incrementally along the normal direction to generate a stack of manifold slices that cover the complete three-dimensional rib cage visualization, as shown in Figure 4.

[0055] At 720, fracture detection is performed via a trained fracture detection model using machine learning or deep learning techniques. The generated stack of manifold slices is received as input, and at 722, a modified 2D manifold view is generated according to the fracture detection model trained on the space of the generated stack of manifold slices. In this way, the modified 2D manifold view covers the complete 3D rib visualization with the predicted fractures mapped to the modified 2D manifold view. The predicted fractures can be shown to the user as a list of landmarks / bounding boxes or a colored overlay on the modified 2D manifold view, and multiple predicted fractures per rib can be shown to the user.

[0056] At 724, a modified 2D manifold view indicating the predicted fracture and a mapping function are received. At 726, the predicted fracture from the stack of modified manifold slices is mapped back to the image space of the 3D diagnostic image. In this manner, the fracture prediction can be mapped back to the original image space for further review by a radiologist.

[0057] While the invention has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive, i.e., the invention is not limited to the disclosed embodiments.

[0058] Other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims.

[0059] In the claims, the word "comprise" does not exclude other elements or steps and the singular does not exclude a plurality.

[0060] A single processor, device or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0061] Operations such as obtaining, determining, obtaining, outputting, providing, memorizing or storing, calculating, simulating, receiving, alerting, stopping, etc. may be implemented as program code means of a computer program and / or as dedicated hardware.

[0062] The computer program may be stored and / or distributed on a suitable medium, such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless communication systems.

Claims

1. a mapping function module having a mapping function circuit; a manifold generation module having a manifold processing circuit; a trained fracture detection module comprising a fracture prediction circuit; an inverse coordinate transformation module having an inverse coordinate transformation circuit; An image processing device having: The mapping function circuit receiving data representing a three-dimensional diagnostic image; segmenting the ribs and vertebrae according to the received data representing the three-dimensional diagnostic image; Detecting and labeling rib centerlines from the rib segmentation; Detecting and labeling central landmarks of vertebral bodies from the spine segmentation; mapping the three-dimensional positions of the rib centerlines and the vertebral body central landmarks corresponding to the received data representing the three-dimensional diagnostic image to two-dimensional positions on a defined two-dimensional manifold plane; transforming the two-dimensional manifold plane so that the two-dimensional manifold is aligned with the detected rib centerlines and vertebral body central landmarks in the three-dimensional space of the diagnostic image; and interpolating missing three-dimensional position coordinates on the defined two-dimensional manifold plane; generating coordinate mapping functions from the two-dimensional manifold space to the three-dimensional image space and vice versa, corresponding to the mapped and interpolated positions; The manifold processing circuit includes: receiving the generated coordinate mapping function; sampling image intensities from the three-dimensional diagnostic image space at each coordinate defined by the transformed two-dimensional manifold plane according to the mapping function; generating, from the sampled image intensities, an image reformatted as two-dimensional manifold slices representing a continuous linear visualization of the rib cage and spine; and shifting the deformed two-dimensional manifold plane along its normal direction and repeating the sampling to generate a stack of manifold slices covering the complete three-dimensional thorax visualization; The fracture prediction circuit includes: receiving as input the stack of manifold slices generated; Performing fracture prediction via a trained fracture detection model using machine learning or deep learning techniques; and generating a modified two-dimensional manifold view showing predicted fractures within the space of the generated stack of manifold slices covering the complete three-dimensional thorax visualization according to the trained fracture detection model; The inverse coordinate transformation circuit receiving as input the mapping function and the modified two-dimensional manifold view showing the predicted fracture; and mapping the predicted fractures from the stack of modified manifold slices back to an image space of data representing the three-dimensional diagnostic image; Image processing device.

2. a conversion module including a conversion circuit; The conversion circuit receiving as input the stack of generated manifold slices and the mapping function; generating a stack of annotated manifold slices by repeatedly sampling, generating, and shifting said manifold processing circuitry using the 3D annotation mask or the 3D annotation landmarks as sampling inputs; and generating an annotation mask or a set of annotation landmarks defined on the generated stack of manifold slices by mapping annotated fractures from the 3D image space onto the stack of 3D annotated manifold slices according to the mapping function; The image processing device according to claim 1 .

3. The image processing apparatus of claim 2 , wherein the trained fracture detection module receives the generated stack of manifold slices and the generated stack of annotated manifold slices as inputs for training the trained fracture detection model.

4. The image processing apparatus of claim 3 , wherein the trained fracture detection model is trained according to a slice-wise single-shot or patch-based model.

5. The image processing device of claim 1, wherein the predicted fractures are shown as a list of landmarks or voxel-wise annotations with corresponding colored overlays in the space of the stack of manifold slices or in the space of the diagnostic image, and multiple predicted fractures per rib are determined and shown in the space of the stack of manifold slices or in the space of the diagnostic image.

6. The image processing device of claim 4 , wherein a size of a receptive field is adjusted in the defined stack of manifold slices such that fractures in adjacent ribs are captured during training of the trained fracture detection model.

7. The image processing device of claim 1 , wherein the predicted fractures are subsequently analyzed by a classification model.

8. The image processing apparatus of claim 3 , wherein additional inputs to the trained fracture detection module include scanned images and annotations from previous imaging scans.

9. The image processing apparatus of claim 1 , wherein interpolation of missing coordinates on the manifold is performed via an interpolation technique including thin-plate splines.

10. 1. A method for automatically detecting rib fractures in three-dimensional diagnostic images, comprising: receiving data representing a three-dimensional diagnostic image; segmenting ribs and vertebrae according to the received data representing the three-dimensional diagnostic image; Detecting and labeling rib centerlines from the rib segmentation; Detecting and labeling central landmarks of vertebral bodies from the segmentation of the spine; mapping the three-dimensional positions of the rib centerlines and the vertebral body central landmarks corresponding to the received data representing the three-dimensional diagnostic image to two-dimensional positions on a defined two-dimensional manifold plane; interpolating missing three-dimensional position coordinates on the defined two-dimensional manifold plane, which transforms the two-dimensional manifold plane so that the two-dimensional manifold is aligned with the detected rib centerlines and vertebral body central landmarks in the three-dimensional space of the diagnostic image; generating coordinate mapping functions from two-dimensional manifold space to three-dimensional image space and vice versa, corresponding to the mapped and interpolated positions; sampling image intensities from the three-dimensional diagnostic image space at each coordinate defined by the transformed two-dimensional manifold plane according to the mapping function; generating from the sampled image intensities a reformatted image as two-dimensional manifold slices representing a continuous linear visualization of the rib cage and spine; shifting the deformed two-dimensional manifold plane along its normal direction and repeating the sampling to generate a stack of manifold slices covering the complete three-dimensional thorax visualization; receiving the stack of generated manifold slices as input to a fracture detection model to be trained; performing fracture prediction via the fracture detection model trained according to machine learning or deep learning techniques; generating a modified two-dimensional manifold view showing predicted fractures within the space of the generated stack of manifold slices covering the complete three-dimensional thorax visualization according to the trained fracture detection model; receiving the modified two-dimensional manifold view showing the mapping function and the predicted fracture; mapping the predicted fracture from the stack of manifold slices back to an image space of data representing the three-dimensional diagnostic image; A method for automatically detecting rib fractures in three-dimensional diagnostic images, comprising:

11. receiving the generated stack of manifold slices and the mapping function; generating a stack of annotated manifold slices by iterative sampling, generating and shifting using the 3D annotation mask or the 3D annotation landmarks as sampling input; generating an annotation mask or a set of annotation landmarks defined in the generated stack of manifold slices by mapping annotated fractures from the 3D image space to the stack of 3D annotated manifold slices according to the mapping function; 11. The method of automatically detecting rib fractures in three-dimensional diagnostic images of claim 10, comprising:

12. 12. The method for automatically detecting rib fractures in three-dimensional diagnostic images of claim 11, further comprising receiving the generated stack of manifold slices and the generated stack of annotated manifold slices as inputs for training the trained fracture detection model.

13. 13. The method for automatically detecting rib fractures in three-dimensional diagnostic images of claim 12, further comprising training the fracture detection model according to a slice-wise single-shot or patch-based model.

14. 11. The method of automatically detecting rib fractures in three-dimensional diagnostic images of claim 10, further comprising the steps of displaying the predicted fractures as a list of landmarks or voxel-wise annotations with corresponding colored overlays in the space of the stack of manifold slices or in the space of the diagnostic image, and displaying multiple predicted fractures per rib.

15. A non-transitory computer-readable medium storing instructions for causing a processing circuit to perform a process, the process comprising: receiving data representing a three-dimensional diagnostic image; segmenting ribs and vertebrae according to the received data representing the three-dimensional diagnostic image; Detecting and labeling rib centerlines from the rib segmentation; Detecting and labeling central landmarks of vertebral bodies from the segmentation of the spine; mapping the three-dimensional positions of the rib centerlines and the vertebral body central landmarks corresponding to the received data representing the three-dimensional diagnostic image to two-dimensional positions on a defined two-dimensional manifold plane; interpolating missing three-dimensional position coordinates on the defined two-dimensional manifold plane, which transforms the two-dimensional manifold plane so that the two-dimensional manifold is aligned with the detected rib centerlines and vertebral body central landmarks in the three-dimensional space of the diagnostic image; generating coordinate mapping functions from two-dimensional manifold space to three-dimensional image space and vice versa, corresponding to the mapped and interpolated positions; sampling image intensities from the three-dimensional diagnostic image space at each coordinate defined by the transformed two-dimensional manifold plane according to the mapping function; generating from the sampled image intensities a reformatted image as two-dimensional manifold slices representing a continuous linear visualization of the rib cage and spine; shifting the deformed two-dimensional manifold plane along its normal direction and repeating the sampling to generate a stack of manifold slices covering the complete three-dimensional thorax visualization; receiving the stack of generated manifold slices as input to a fracture detection model to be trained; performing fracture prediction via the fracture detection model trained according to machine learning or deep learning techniques; generating a modified two-dimensional manifold view showing predicted fractures within the space of the generated stack of manifold slices covering the complete three-dimensional thorax visualization according to the trained fracture detection model; receiving the modified two-dimensional manifold view showing the mapping function and the predicted fracture; mapping the predicted fracture from the stack of manifold slices back to an image space of data representing the three-dimensional diagnostic image; 1. A non-transitory computer-readable medium having:

Images