Method for providing medical imaging decision support data

Abstract

The invention relates to a computer-implemented method for providing medical imaging decision support data. The method comprises receiving 2D scout data of a patient, generating 2D scout annotation data by applying a machine learning algorithm for scout analysis to the 2D scout data, generating medical imaging decision support data based on the 2D scout annotation data, and providing the medical imaging decision support data.

Description

TECHNICAL FIELD

[0001] The present invention relates in one aspect to a computer-implemented method for providing medical imaging decision support data. In further aspects, the present invention relates to a computer-implemented method for providing training pairs for training a machine learning algorithm, a computer-implemented method for training a machine learning algorithm for localization slice analysis, a data processing system, a medical imaging device, a computer program product and a computer readable storage medium. BACKGROUND

[0002] In computed tomography (CT), a 2D scout view (also called a scanogram or scoutogram) is acquired prior to 3D image acquisition. Currently, the main purpose of the scout view projection image is to plan the scan range and the reconstruction field of view (FoV) of the 3D CT scan to be performed and to estimate the CT dose modulation for automatic exposure control. However, human diagnostic image reading is mainly based on the 3D images. In daily clinical routine, scout view images are occasionally read for some findings that would be missed by reading the 3D images only.

[0003] The reconstruction field of view of the 3D reconstructed image is in particular usually smaller than the scout view scan range of the patient. For example, in abdominal CT imaging, findings such as lung nodules or cardiac enlargement can be captured only in the scout view image but are not included in the reduced 3D abdominal CT reconstruction field of view. In trauma imaging, findings such as skull fractures can be overlooked in the 3D axial images but are clearly visible in the 2D projection scout view data.

[0004] Zhang Y., Miao S., Mansi T., Liao R. (2018) Task Driven Generative Modeling for Unsupervised Domain Adaptation: Application to X-ray Image Segmentation. In: Frangi A., Schnabel J., Davatzikos C., Alberola-Lopez C., Fichtinger G. (eds) Medical Image Computing and Computer Assisted Intervention - MICCAI 2018. MICCAI 2018. Lecture Notes in Computer Science, vol 11071. Springer, Cham. discloses the application of a task driven generative modeling method in X-ray image segmentation. SUMMARY

[0005] It is an object of the present application to improve analysis of scout images in medical imaging examinations. This problem is solved according to the present application. Further advantageous embodiments and additional advantageous features are described in the description.

[0006] The present application relates in one aspect to a computer-implemented method for providing medical imaging decision support data, the method comprising:

[0007] - receiving 2D scout image data of a patient,

[0008] - generating 2D scout image annotation data by applying a machine learning algorithm for scout image analysis to the 2D scout image data,

[0009] - generating medical imaging decision support data based on the 2D scout image annotation data, and

[0010] - providing the medical imaging decision support data.

[0011] The AI-based automatic scout analysis can be implemented, for example, in the form of a companion diagnosis for 3D image reading. On the other hand, the AI-based automatic scout analysis is performed immediately after scout acquisition. Potential findings are made available by the system automatically suggesting an appropriate scan protocol deviating from the originally planned scan protocol based on the detected findings to (better) capture the findings in the actual 3D acquisition to be performed. The medical imaging decision support data can be generated by a rule-based system and / or an AI-based system.

[0012] The medical imaging decision support data can comprise, for example, a suggestion to adjust and / or enlarge the 3D scan range or the reconstruction field of view to cover a finding in the 3D image. For example, the medical imaging decision support data can comprise a suggestion to perform an additional contrast-enhanced scan, for example, in case of a dilated aorta detected on the scout image. For example, the medical imaging decision support data can comprise a suggestion to acquire an additional 3D image with a dedicated bone kernel in order to read potential bone fractures or dedicated lung kernels in case of a lung nodule detected from the scout image.

[0013] The medical imaging decision support data can indicate values and / or value changes of scan parameters of a scan protocol for performing a 3D medical imaging examination of a patient by a medical imaging device.

[0014] In particular, a 3D medical imaging examination of the patient can be performed based on the medical imaging decision support data, thereby obtaining 3D medical imaging data of the patient.

[0015] The scan parameters can be, for example, radiation source parameters of a radiation source, radiation detector parameters of a radiation detector, scan geometry parameters, or injection parameters of a contrast agent injector.

[0016] The AI-suggested (in particular) examination protocol deviation based on the scout findings can reduce the risk of missing findings, thereby improving the diagnostic quality and allowing to adjust the scan protocol immediately before the actual 3D scan starts, thereby avoiding the need to recall the patient after reading the images and the resulting additional workload and radiation exposure. Existing diagnostic applications can be simplified by resolving tasks on the scout instead of requiring additional CT scans, thereby reducing the radiation dose.

[0017] The medical imaging decision support data can indicate values and / or value changes of reconstruction parameters of a reconstruction algorithm for reconstructing a medical image based on the 3D medical imaging data of the patient. In particular, the 3D medical imaging data of the patient can be provided and / or a medical image can be reconstructed based on the medical imaging decision support data and the 3D medical imaging data of the patient.

[0018] The present invention relates in another aspect to a computer-implemented method for providing training pairs for training of a machine learning algorithm, the method comprising:

[0019] receiving 3D annotation data related to the 3D image data of the examination region, the examination region comprising an anatomical structure, the 3D annotation data being indicative of a feature of the anatomical structure,

[0020] receiving 2D projection image data, the 2D projection image data being related to the 3D image data by projection geometry,

[0021] generating 2D annotation data related to the 2D projection image data based on the projection geometry and the 3D annotation data, the 2D annotation data being indicative of the feature of the anatomical structure, and

[0022] providing a training pair comprising the 2D projection image data and the 2D annotation data.

[0023] Thereby consistency between analysis of annotated 2D projection image data and annotated 3D image data is guaranteed. Thus, artificial intelligence (AI) results based on 2D scout images can be obtained which can be more consistent with AI results obtained directly from 3D image data. It can even be possible to have a pair of 2D and 3D machine learning algorithms trained based on essentially the same annotations.

[0024] The training pair can in particular comprise the 2D projection image data as input to a machine learning algorithm and the 2D annotation data as a target output associated with the input. The feature of the anatomical structure can for example be a parameter value of the anatomical structure, a shape of the anatomical structure, a position of the anatomical structure or a combination thereof.

[0025] The AI results based on scout images can have a higher accuracy compared to algorithms trained on 2D data only, because the location and accuracy of true background annotations is more limited on 2D data than in 3D. Furthermore, information that can be directly extracted by manually annotating scout images can be significantly less informative than annotations automatically generated from 3D image data. By leveraging information from 3D image data, annotations can be created that cannot be created by domain experts first based on scout images due to their complexity. The 3D derived true background annotations for 2D AI training can in particular even enable the 2D AI algorithm to capture features that are significantly hidden (cannot be annotated) in 2D scout image data.

[0026] Another approach to collect accurately labeled training data for 2D scout image AI algorithms from 3D CT images for machine learning tasks in medical imaging is to manually annotate the relevant anatomical structures by domain experts. Although this process can be supported by semi-automated tools for the task at hand to minimize user interaction, it requires the knowledge and time of domain experts. Based on the proposed approach, additional training data can be provided for a machine learning system that has already been trained on a limited set of manually created annotations, thereby improving its performance.

[0027] Furthermore, costs can be reduced due to the reduced manual annotation by domain experts. Machine learning systems can be implemented faster due to the faster annotation speed of algorithms compared to domain experts.

[0028] 3D image data can be received. 3D annotation data related to the 3D image data can be computed by applying a 3D annotation algorithm to the 3D image data. This allows to generate annotated training data for a 2D scout view based machine learning system from 3D image data in a fully automatic way by exploiting annotation algorithms available for 3D images.

[0029] 2D projection image data can be computed based on the projection geometry and the 3D image data. Both the 3D annotation data and the 3D image can be projected into the 2D image space of the medical imaging modality, thereby automatically generating precise labeled training data for training a machine learning algorithm for scout view analysis. The computed 2D projection image can be referred to as a "synthetic scout view". However, physically obtained scout views can be used.

[0030] The 2D projection image data can be a CT scout view of an examination region. In this case, the annotation data needs to be projected by using the same projection geometry as used for acquiring the CT scout view. The 3D image data can be, for example, a full body 3D CT image of a person. The 3D image data can be obtained from, for example, a trauma scan, for which typically a full body CT reconstruction with full field of view is performed. The 2D scout view data of the patient can be 2D CT scout view data of the patient.

[0031] Furthermore, the examination region can also include a surrounding of the anatomical structure. The 3D annotation data can include a 3D representation of the anatomical structure. The 2D annotation data can include a 2D representation of the anatomical structure. The 3D representation of the anatomical structure and / or the 2D representation of the anatomical structure can delimit the anatomical structure from the surrounding of the anatomical structure.

[0032] The 3D annotation data can include quantitative information indicating a parameter value of the anatomical structure. The 2D annotation data can include quantitative information.

[0033] The method can be used for a variety of tasks, such as detecting lung diseases, bone fractures, and / or coronary artery calcification. One exemplary application is bone density estimation of a vertebra. This is an existing clinical application performed on 3D CT images, so that it can be easily used to generate training data for the same application on 2D CT scout views in an automatic way. The anatomical structure can include, for example, one vertebra or one group of vertebrae.

[0034] The anatomical structure can include one group of vertebrae. For each vertebra of the group of vertebrae, the 3D annotation data can include a 3D representation of the vertebra. For each vertebra of the group of vertebrae, the 2D annotation data can include a 2D representation of the vertebra.

[0035] For each vertebra of the set of vertebrae, the 3D annotation data can comprise quantitative bone density information indicative of a bone density value of the vertebra. For each vertebra of the set of vertebrae, the 2D annotation data can comprise quantitative bone density information.

[0036] The application relates in another aspect to a computer-implemented method for training a machine learning algorithm for scout analysis, the method comprising:

[0037] receiving a plurality of training pairs, each training pair of the plurality of training pairs being provided based on the method for providing training pairs for training a machine learning algorithm,

[0038] training a machine learning algorithm based on the plurality of training pairs.

[0039] Each training pair of the plurality of training pairs can be used as input for training of the machine learning algorithm. Each training pair of the plurality of training pairs can comprise a respective 2D projection image data as input for the machine learning algorithm and a respective 2D annotation data as target output associated with the input. An accuracy measure of the machine learning algorithm can be determined based on a deviation of a result of the machine learning algorithm when applied to a given input from a target output associated with the given input.

[0040] In another aspect, a machine learning algorithm for scout analysis has been trained according to the method for training a machine learning algorithm for scout analysis.

[0041] The medical imaging decision support data can be generated based on results of at least two AI algorithms based on scout images, e.g. for detecting metal and / or for identifying body regions and / or organs.

[0042] The application relates in another aspect to a data processing system comprising a data interface and a processor, the data processing system being configured for performing the method according to one aspect of the application. The data interface can be configured for receiving and / or providing data, in particular 3D annotation data, 2D projection image data, 3D image data, 2D scout data, training pairs and / or medical imaging decision support data. The processor can be configured for generating and / or computing data, in particular 2D annotation data, 3D annotation data, 2D projection image data, 2D scout annotation data and / or medical imaging decision support data.

[0043] The application relates in another aspect to a medical imaging device comprising a data processing system. The medical imaging device can for example be a computed tomography (CT) device or a magnetic resonance imaging (MRI) device.

[0044] The medical imaging device can be a computed tomography device. The 3D image data can be 3D computed tomography image data.

[0045] The application relates in another aspect to a computer program product or computer readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the method according to one aspect of the application.

[0046] Any algorithm mentioned herein, in particular machine learning algorithm and / or 3D annotation algorithm, can be based on one or more of the following architectures: convolutional neural network, deep belief network, random forest, deep residual learning, deep reinforcement learning, recurrent neural network, Siamese network, generative adversarial network or autoencoder. The trained machine learning algorithm can in particular be embodied as a deep learning algorithm, in particular a deep convolutional neural network. Throughout this disclosure, the term "artificial intelligence (AI) algorithm" is used synonymously with the term "machine learning algorithm".

[0047] The computer program product can for example be a computer program or comprise another element than the computer program. The other element can be hardware, for example a storage device on which the computer program is stored, a hardware key for using the computer program, and / or the other element can be software, for example a document or a software key for using the computer program. The computer readable storage medium can be embodied as a non-persistent main memory, for example a random access memory, or a persistent mass storage, for example a hard disk, a USB stick, an SD card, a solid state disk.

[0048] The data processing system can for example comprise at least one of a cloud computing system, a distributed computing system, a computer network, a computer, a tablet computer, a smartphone, etc. The data processing system can comprise hardware and / or software. The hardware can for example be a processor system, a memory system and combinations thereof. The hardware can be configured by and / or operable by software. The computations for performing the actions of the method can be performed in a processor.

[0049] The data, in particular each of the 3D annotation data, the 2D projection image data, the 3D image data and the 2D scout slice data, can for example be received by receiving a signal carrying the data and / or by reading the data from a computer memory. The data, in particular each of the training pairs and the medical imaging decision support data, can for example be provided by sending a signal carrying the data and / or by writing the data to a computer memory and / or by displaying the data on a display.

[0050] In the context of the present invention, the word "based on" can in particular be understood to mean "in particular by using". In particular, the wording "based on" a second feature to calculate (or generate, determine, etc.) a first feature does not exclude the possibility to calculate (or generate, determine, etc.) the first feature based on a third feature.

[0051] Reference is made to the fact that the method and the system are only preferred exemplary embodiments of the present invention and that a person skilled in the art can vary the present invention without departing from the scope of the present invention as specified by the claims. BRIEF DESCRIPTION OF DRAWINGS

[0052] The present invention will be illustrated below by using exemplary embodiments with reference to the accompanying drawings. The illustrations in the drawings are schematic and highly simplified and are not necessarily drawn to scale.

[0053] Figure 1 An annotated 3D image is shown.

[0054] Figure 2 An annotated 2D projection image is shown.

[0055] Figure 3 A flowchart of a method for providing training pairs is shown.

[0056] Figure 4 A flowchart of another example of a method for providing training pairs is shown.

[0057] Figure 5 A flowchart of a computer-implemented method for training a machine learning algorithm for locating a slide analysis is shown.

[0058] Figure 6 A flowchart of a method for providing medical imaging decision support data is shown.

[0059] Figure 7 A data processing system is shown. DETAILED DESCRIPTION

[0060] Figure 1 An annotated 3D image M3 in the form of a rendered chest 3D CT image is shown, which comprises a 3D representation of vertebrae V1, V2 of a set of vertebrae V and further comprises a 3D representation of ribs R1, R2 of a rib cage R.

[0061] Figure 2Annotated 2D projection image M2 is shown in the form of a synthetic scout film with 2D representation of vertebrae V1, V2. The synthetic scout film has been computed from the chest 3D CT image based on coronal projection geometry. The vertebrae V1, V2 are automatically segmented from the 3D CT image and then projected into the synthetically created scout film image. A set of similarly generated scout films can now be used to train a machine learning system to detect vertebrae in actual scout films.

[0062] Figure 3 A flowchart of a computer-implemented method for providing a training pair for training of a machine learning algorithm is shown, the method comprising:

[0063] receiving T1 3D annotation data related to 3D image data of an examination region, the examination region comprising an anatomical structure, the 3D annotation data being indicative of a feature of the anatomical structure,

[0064] receiving T2 2D projection image data, the 2D projection image data being related to the 3D image data by a projection geometry,

[0065] generating T3 2D annotation data related to the 2D projection image data based on the projection geometry and the 3D annotation data, the 2D annotation data being indicative of the feature of the anatomical structure, and

[0066] providing T4 a training pair comprising the 2D projection image data and the 2D annotation data.

[0067] Figure 4 A flowchart of another example of a method for providing a training pair is shown, the method further comprising:

[0068] receiving T10 3D image data,

[0069] computing T11 3D annotation data related to the 3D image data by applying a 3D annotation algorithm to the 3D image data, and

[0070] computing T12 2D projection image data from the projection geometry and the 3D image data.

[0071] Figure 5 A flowchart of a computer-implemented method for training a machine learning algorithm for scout film analysis is shown, the method comprising:

[0072] receiving S11 a plurality of training pairs, each training pair of the plurality of training pairs being provided based on the method for providing a training pair,

[0073] training S12 the machine learning algorithm based on the plurality of training pairs.

[0074] Figure 6A flowchart of a computer-implemented method for providing medical imaging decision support data is shown, the method comprising:

[0075] - receiving S1 2D scout data of a patient,

[0076] - generating S2 2D scout annotation data by applying a machine learning algorithm for scout analysis to the 2D scout data,

[0077] - generating S3 medical imaging decision support data based on the 2D scout annotation data, and

[0078] - providing S4 the medical imaging decision support data.

[0079] Figure 7 A data processing system 3 is shown, comprising a data interface 3A and a processor 3B, the data processing system 3 being configured for performing the method as described with respect to any one of Figures 3 to 6 the preceding figures.

Claims

1. A computer-implemented method for providing training pairs for training of a machine learning algorithm, the method comprising: receiving (T1) 3D annotation data related to 3D image data of an examination region, the examination region comprising an anatomical structure, the 3D annotation data being indicative of a feature of the anatomical structure, receiving (T2) 2D projection image data, the 2D projection image data being related to the 3D image data by projection geometry, generating (T3) 2D annotation data related to the 2D projection image data based on the projection geometry and the 3D annotation data, the 2D annotation data being indicative of the feature of the anatomical structure, and providing (T4) a training pair comprising the 2D projection image data and the 2D annotation data.

2. The method of claim 1, further comprising: receiving (T10) the 3D image data, computing (T11) the 3D annotation data related to the 3D image data by applying a 3D annotation algorithm to the 3D image data.

3. The method of claim 1 or 2, further comprising: receiving (T10) the 3D image data, computing (T12) the 2D projection image data based on the projection geometry and the 3D image data.

4. The method of claim 1 or 2, the 2D projection image data being a CT scout view of the examination region.

5. The method of claim 1 or 2, the 3D annotation data comprising a 3D representation of the anatomical structure, the 2D annotation data comprising a 2D representation of the anatomical structure.

6. The method of claim 1 or 2, the anatomical structure comprising a set of vertebrae, the 3D annotation data comprising, for each vertebra of the set of vertebrae, a 3D representation of the vertebra, the 2D annotation data comprising, for each vertebra of the set of vertebrae, a 2D representation of the vertebra.

7. The method of claim 6, the 3D annotation data comprising, for each vertebra of the set of vertebrae, quantitative bone density information indicative of a bone density value of the vertebra, the 2D annotation data comprising the quantitative bone density information for each vertebra of the set of vertebrae.

8. A computer-implemented method for training a machine learning algorithm for scout view analysis, the method comprising: receiving (S11) a plurality of training pairs, each training pair of the plurality of training pairs being provided based on the method of any one of claims 1 to 7, training (S12) the machine learning algorithm based on the plurality of training pairs.

9. A computer-implemented method for providing medical imaging decision support data, the method comprising: receiving (S1) 2D scout view data of a patient, generating (S2) 2D scout view annotation data by applying a machine learning algorithm for scout view analysis to the 2D scout view data, wherein the machine learning algorithm for scout view analysis has been trained according to the method of claim 8, generating (S3) the medical imaging decision support data based on the 2D scout view annotation data, and providing (S4) the medical imaging decision support data.

10. The method of claim 9, the medical imaging decision support data is indicative of a value and / or a value variation of a scan parameter of a scan protocol for performing a 3D medical imaging examination of the patient by a medical imaging device.

11. The method of claim 9 or 10, the medical imaging decision support data is indicative of a value and / or a value variation of a reconstruction parameter of a reconstruction algorithm for reconstructing a medical image based on 3D medical imaging data of the patient.

12. A data processing system (3) comprising a data interface (3A) and a processor (3B), the data processing system (3) being configured to perform the method according to any one of claims 1 to 11.

13. A medical imaging device comprising the data processing system according to claim 12.

14. The medical imaging device of claim 13, the medical imaging device is a computed tomography device, the 3D image data is 3D computed tomography image data.

15. A computer program product or a computer readable storage medium comprising instructions which, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 11.

Images