Multi-task deep learning methods for neural networks used in automated pathology detection

Abstract

A multi-task deep learning method for neural networks used in automated pathological detection includes the following steps: receiving (S1) first image data (I) for a first image recognition task; receiving (S2) second image data (V) for a second image recognition task; wherein the first image data (I) is of a first data type, the second image data (V) is of a second data type, and the second data type is different from the first data type; determining (S3) first labeled image data (I) by labeling the first image data (I). L The second synthesized annotated image data (I) is determined by synthesizing and annotating the second image data (V). SL Based on the received first image data (I), the received second image data (V), and the determined first labeled image data (I), L ) and the determined second labeled synthetic image data (I SL (S4) trains the neural network; wherein the first image recognition task and the second image recognition task relate to the same anatomical region, wherein the corresponding image data is taken from the same pathology to be identified in the corresponding image data and / or relates to the same anatomical pathology to be identified in the corresponding image data.

Description

Technical Field

[0001] This invention relates to a multi-task deep learning method, device, computer program, and computer-readable medium for neural networks used in automated pathological detection. Background Technology

[0002] In current clinical practice, the need for automation is becoming increasingly important, both to be cost-effective and to free radiologists from tedious daily tasks. One such task is evaluating X-ray acquisition. Detecting specific diseases on X-ray synthesis can be considered a reasonable interpretation task. Currently, machine learning algorithms, such as neural networks based on deep learning methods, are increasingly being used for such purposes in various application areas. Training neural networks requires a large amount of annotated training data. Their success largely depends on this extensive training database. Therefore, most traditional deep learning methods have successfully utilized pre-trained networks from the field of computer vision.

[0003] Moturu et al.’s “Creation of Synthetic X-Rays to Train a Neural Network to Detect Lung Cancer” and Chang et al.’s “Detecting Early Stage Lung Cancer using a Neural Network Trained with Patches from Synthetically Generated X-Rays” disclose neural networks used to train lung cancer detection. Summary of the Invention

[0004] Therefore, an improved learning method may be needed for neural networks used in automated pathology detection.

[0005] The objective of this invention is achieved through the subject matter of the independent claims, wherein further embodiments are incorporated in the dependent claims. It should be noted that the aspects of the invention described below also apply to multi-task deep learning methods, devices, computer programs, and computer-readable media.

[0006] According to one aspect, a multi-task deep learning method for a neural network used for automated pathology detection is provided. The method includes the following steps: receiving first image data for a first image recognition task, receiving second image data for a second image recognition task, wherein the first image data is of a first data type and the second image data is of a second data type, the second data type being different from the first data type; determining first labeled image data by labeling the first image data; determining second labeled synthesized image data by synthesizing and labeling the second image data; and training a neural network based on the received first image data, the received second image data, the determined first labeled image data, and the determined second labeled synthesized image data, wherein the first image recognition task and the second image recognition task relate to the same anatomical region, wherein the corresponding image data is taken from the same pathology to be identified in the corresponding image data and / or relates to the same anatomical pathology to be identified in the corresponding image data.

[0007] Preferably, both the first image data and the second image data are determined from the same area of ​​the patient, particularly from the patient's chest area.

[0008] Preferably, the neural network is trained using a multi-task deep learning algorithm.

[0009] As used herein, the term "synthesis" includes methods that combine data selection, geometric transformations (including forward projection), physical simulations and / or deep learning methods for data conversion with image processing techniques for resizing and registration. The synthesis of image data (i.e., for generating digitally reconstructed radiographs from CT images) is determined via forward projection, preferably including raycasting and snowballing algorithms. Specifically, raycasting is used to integrate the attenuation coefficients of the CT image along a path from a hypothetical X-ray source to the detector to approximate the principles of X-ray image formation. In other words, second image data of a second data type is converted into a first data type of first image data, or at least a similar data type, wherein the first and second image data can be processed mutually, particularly simultaneously. Preferably, the image data types are similar in pixel resolution and / or synthesis geometry.

[0010] As used herein, the term "multi-task deep learning approach" encompasses deep learning methods that allow the use of all different types of input data, except for the final input data, during the application of the method. Models can be improved on the original task by sharing representations among related tasks. Multi-task deep learning preferably includes hard parameter sharing or soft parameter sharing. Hard parameter sharing involves sharing intermediate layers across all tasks while retaining several task-specific output layers. Soft parameter sharing involves adjusting the distance between model parameters to converge the parameters, where, for each task, there exists a separate model with its own parameters.

[0011] Preferably, training the neural network based on the received first image data, the received second image data, the determined first labeled image data, and the determined second labeled synthetic image data includes training the neural network simultaneously on the received first image data, the received second image data, the determined first labeled image data, and the determined second labeled synthetic image data.

[0012] Therefore, by synthesizing (especially forward projection) second image data, and synthesizing image data with a data type similar to the first image data, the multi-task deep learning method can process the first image data and the second image data simultaneously.

[0013] The terms “annotated image data” and “annotation” can also be referred to as “annotated data” and “annotation”.

[0014] The labeling information used to train the neural network can be obtained from radiologists based on identified diseases and clinical measurements, such as using radiology reports and / or segmentation, particularly semi-automatic segmentation. Therefore, the labeling information is manually determined by the radiologist based on the provided first and / or second image data. Alternatively, the labeling information is automatically derived from the radiology report, for example, using natural language processing algorithms or image segmentation algorithms. In an alternative embodiment, the second image data is automatically labeled. Based on those automatically determined labels, the labeling information for the labeled synthetic image data is derived. In other words, when synthesizing the second image data, the automatically determined labeling information of the second image data is synthesized into the labeling information for the labeled synthetic image data.

[0015] As used herein, the term “data type” refers to imaging modalities such as computed tomography and / or magnetic resonance imaging, and / or different acquisition protocols.

[0016] The second image recognition task can also be described as an auxiliary image recognition task, wherein the first image recognition task used to generate training data is the same as the image recognition task applied later, and the second image recognition task used to generate training data is only similar to the image recognition task applied later.

[0017] Preferably, the first image recognition task and the second image recognition task can be almost identical, such as the detection of chest lesions; however, the auxiliary image recognition task can also be modality-specific, such as the precise localization of landmarks that are not directly visible in a synthetic image.

[0018] For example, if emphysema should be detected in the lungs, such emphysema is almost undetectable on X-ray images. However, it is relatively easy to detect in computed tomography (CT) images. Therefore, emphysema can be easily labeled in CT images. Thus, CT images, which are image types projected forward onto X-ray images, can provide machine learning algorithms with high-quality, realistic depictions of emphysema in image data.

[0019] Therefore, in addition to the type of data that must be solved in medical problems, model training also takes into account the tasks of different data.

[0020] Therefore, it can be determined that higher quality labeled image data is used during the training process.

[0021] Therefore, not only can different data sources be used, but neural networks can also be optimized for different but related image recognition tasks.

[0022] Therefore, the training data for neural networks can be improved, thereby improving the learning methods for neural networks used in automated pathology detection.

[0023] In a preferred embodiment, the method includes the following steps: determining synthesized second image data by synthesizing second image data into a first data type and / or a first dimension, and determining second annotated synthesized image data by annotating the synthesized second image data.

[0024] Preferably, the synthesis, particularly forward projection, includes selecting a subset of data, physical simulations for data transformation, and / or the application of deep learning methods, such as recurrent GANs combined with image processing techniques for resizing and registration.

[0025] Determining the synthetic image data preferably involves selecting individual slices from 3D image data, particularly 3D computed tomography images, and resizing them to the correct size for 2D images (especially 2D X-ray images) by interpolation and / or by artificial forward synthesis.

[0026] For classification tasks, labeled image data can be used directly, while for detection tasks, including segmentation, labeled image data must be synthesized.

[0027] Therefore, different modalities can be used to determine the training data for neural networks, thereby improving the learning methods of neural networks used for automated pathology detection.

[0028] Preferably, the 3D image data is forward-synthesized into a 2D composite geometry configuration that is closely related to the geometric configuration of the 2D image data, especially taking into account pixel resolution and / or pixel geometry.

[0029] Additionally, if the dimensions of the first image data and the second image data are not the same, size synthesis must be performed before training, specifically by synthesizing the second image data to the same data size.

[0030] Furthermore, if the properties of the first image data are not comparable to those of the second image data in terms of field of view, size, or resolution, additional post-processing methods, such as resampling, must be employed.

[0031] Therefore, different modalities can be used to determine the training data for neural networks, thereby improving the learning methods of neural networks used for automated pathology detection.

[0032] In a preferred embodiment, the method includes the following steps: determining second labeled image data by labeling second image data and determining second labeled composite image data by synthesizing the second labeled image data into a first image data first data type and / or dimension.

[0033] Preferably, the synthesis includes selecting a subset of data, physical simulations for data transformation, and / or the application of deep learning methods, such as a recurrent GAN combined with image processing techniques for resizing and registration.

[0034] Synthesizing the second image data involves selecting individual slices from the 3D image data and resizing them to 2D images with the correct dimensions via interpolation and / or manual forward synthesis.

[0035] For classification tasks, labeled image data can be used directly, while for detection tasks, including segmentation, labeled image data must be synthesized.

[0036] Therefore, different modalities can be used to determine the training data for neural networks, thereby improving the learning methods of neural networks used for automated pathology detection.

[0037] Preferably, the 3D image data is forward-synthesized into a 2D composite geometry configuration that is closely related to the geometric configuration of the 2D image data, especially taking into account pixel resolution and / or pixel geometry.

[0038] Additionally, if the dimensions of the first image data and the second image data are not the same, size synthesis must be performed before training, specifically by synthesizing the second image data to the same data size.

[0039] Furthermore, if the properties of the first image data are not comparable to those of the second image data in terms of field of view, size, or resolution, additional post-processing methods, such as resampling, must be employed.

[0040] Therefore, different modalities can be used to determine the training data for neural networks, thereby improving the learning methods of neural networks used for automated pathology detection.

[0041] In a preferred embodiment, the first image recognition task and / or the second image recognition task include classification, localization, and / or segmentation.

[0042] Preferably, the first image recognition task and / or the second image recognition task include classification, localization, object detection, semantic segmentation, and / or instance segmentation. The term "classification" involves identifying what an object is given an image containing objects. In other words, it involves categorizing an object into a single category from a set of predefined categories. The term "localization" involves locating the object and drawing a bounding box around it. The term "object detection" involves classifying and detecting all objects in an image, assigning a category to each object, and drawing a bounding box around it. The term "semantic segmentation" involves classifying each pixel in an image into a class based on its context, so as to assign each pixel to a single object. The term "instance segmentation" involves classifying each pixel in an image into a class, so as to assign each pixel to a different instance of an object.

[0043] In a preferred embodiment, classification, localization, object detection, and / or segmentation involve pathology, foreign objects, and / or anatomical regions.

[0044] In a preferred embodiment, the first data type includes 2D image data and the second data type includes 3D image data.

[0045] Preferably, the 2D image data is determined by X-ray methods and the 3D image data is generated by computed tomography.

[0046] Alternatively, the first data type includes 2D X-ray imaging data, while the second data type includes 2D X-ray dark-field imaging data. Still alternatively, the first data type includes 4D dynamic contrast-enhanced magnetic resonance imaging (MRI) data, and the second data type includes 3D MRI data.

[0047] Therefore, different modalities can be used to determine the training data for neural networks, thereby improving the learning methods of neural networks used for automated pathology detection.

[0048] In a preferred embodiment, the first image data has a first dimension and the second image data has a second dimension different from the first dimension.

[0049] As used in this article, the term "dimension" refers to the number of degrees of freedom of an image element, such as one-dimensional, 1D, two-dimensional, 2D, three-dimensional, 3D, four-dimensional, 4D, etc.

[0050] In a preferred embodiment, the first labeled image data and the second labeled synthetic image data include the identification and / or location of pathological, foreign body and / or anatomical regions.

[0051] In a preferred embodiment, the first image data includes X-ray image data and the second image data includes computed tomography image data.

[0052] Preferably, the task should be performed on 2D images, particularly X-ray synthesis, whereby the neural network is trained on 2D images, particularly X-ray synthesis, and on 3D data, particularly computed tomography image data.

[0053] Different imaging modalities are typically used throughout the care cycle. For example, X-ray imaging is often used in screening or emergencies because it is fast, effective, and results in a small dose exposure for the patient. In contrast, imaging techniques such as computed tomography (CT) provide quantitative 3D information, allowing for more detailed and accurate diagnoses, but at the cost of higher doses and costs for the patient. They also generally allow for simpler realignment, particularly in cases of normal chest tissue, which can be easily identified using Henle unit values.

[0054] Preferably, the X-ray image undergoes a substantial transformation, particularly downsampling to, for example, 224x224 pixels and / or a heuristic intensity transformation.

[0055] Therefore, when using combined training with data from different sources (such as X-rays and computed tomography), one can benefit from the different data characteristics of different image data types, especially large-volume X-ray images and / or 3D quantitative computed tomography data with a clear diagnosis, and allow for the development of better diagnostic solutions.

[0056] Therefore, computed tomography image data is synthesized into X-ray-like image data.

[0057] In a preferred embodiment, computed tomography image data is synthesized into 2D image data in the form of digitally reconstructed radiographs.

[0058] In a preferred embodiment, the pathology includes one of the following: cardiac hypertrophy, emphysema, edema, hernia, pneumothorax, effusion, mass, fibrosis, atelectasis, consolidation, pleural thickening, nodules, and pneumonia.

[0059] According to another aspect, an apparatus is provided that is configured to perform the methods as described herein.

[0060] According to another aspect, a computer program is provided that includes instructions for causing a computer to perform the methods described herein.

[0061] According to another aspect, a computer-readable medium having a computer program as described herein stored thereon is provided. Attached Figure Description

[0062] Exemplary embodiments of the invention will now be described with reference to the following figures, which are not to scale, wherein:

[0063] Figure 1 A schematic block diagram of a device running a multi-task deep learning method is shown;

[0064] Figure 2 A schematic block diagram illustrating the application of a trained deep learning model is shown; and

[0065] Figure 3 A flowchart of a multi-task deep learning method for neural networks used in automated pathology detection is shown.

[0066] List of reference numerals in the attached diagram:

[0067] 10 Equipment

[0068] 20 Synthesis Units

[0069] 30 annotation units

[0070] 40 model training units

[0071] 50 Model Application Units

[0072] V 3D volume data

[0073] 2D image data

[0074] I S Synthetic 2D image data

[0075] I L Annotated 2D image data (first annotated image data)

[0076] I SL Annotated and synthesized 2D image data (second annotated and synthesized image data)

[0077] M-model

[0078] S1 receives the first image data

[0079] S2 receives the second image data.

[0080] S3 Determine the first labeled image data

[0081] S4 Training Neural Network Detailed Implementation

[0082] Figure 1 The device 10 described includes a synthesis unit 20, an annotation unit 30, and a model training unit 40. First image data in the form of 2D image data I and second image data in the form of 3D volume data V are provided to the device 10. The 3D volume data V is the result of a first image recognition task, in this case, derived from the detection of pneumothorax in X-ray imaging. The 2D image data I is the result of a second image recognition task, in this case, derived from the detection of pneumothorax in computed tomography imaging. The first and second image recognition tasks are related to each other, particularly considering their association with the same anatomical region, in which the corresponding image data is taken from and / or related to the same pathology to be identified in the corresponding image data.

[0083] In this case, the 3D volume data V and the 2D image data I are not the same image data type. For mutual processing, the image data types of 3D volume data V and 2D image data I must be the same, or at least similar. Preferably, the image data types are similar in pixel resolution and / or synthesis geometry. Therefore, the 3D volume data V must be synthesized into an image data type similar to the image data type of the 2D image data I. Thus, the 3D volume data V is provided to the synthesis unit 20. The synthesis unit 20 synthesizes the 3D volume data V into multiple 2D image data, taking into account pixel resolution and synthesis geometry, into an image data type close to the image data type of the 2D image data I. In other words, the 3D volume of the 3D volume data V is sliced ​​into a stack of 2D image data. In this case, synthesizing the 3D volume data V from a computed tomography image is called a digital reconstructed radiograph, DDR. Therefore, the 3D volume data V is synthesized into the synthesized 2D image data I. S Synthesized 2D image data I PS It is then provided to annotation unit 30.

[0084] In image recognition, essentially two image recognition tasks are performed: segmentation and / or classification. Segmentation involves annotation of pixel-level labels, while classification involves annotation of the overall image. Therefore, the provided image data must be labeled in all cases. Thus, the annotation unit 30 is not only provided with the synthesized 2D image data I S Furthermore, it is provided as 2D image data I. Then, the annotation unit 30 annotates the provided image data, specifically by performing a first image recognition task on the 2D image data I and by annotating the synthesized image data I. S Perform the second image recognition task to determine the labeled 2D image data IL and the labeled synthetic 2D image data I. SL Therefore, different architectures are possible. The first architecture includes multi-task learning for X-ray classification and computed tomography (CT) classification. The second architecture includes multi-task learning for X-ray segmentation and CT segmentation. The third architecture includes multi-task learning for X-ray segmentation and CT classification. The fourth architecture includes multi-task learning for X-ray classification and CT segmentation.

[0085] Then the determined labeled two-dimensional image data I is provided to the model training unit 40. L Annotated and synthesized two-dimensional image data I SL Two-dimensional image data I and synthesized two-dimensional image data I S The model training unit 40 then trains the model based on the provided, precisely labeled two-dimensional image data. IL Annotated and synthesized two-dimensional image data I SL Two-dimensional image data I and synthesized two-dimensional image data I S Specifically, a multi-task deep learning approach is used to train the model M using a neural network. Therefore, X-ray images and synthetic computed tomography images are used to train a network architecture with shared layers. In this case, this method significantly increases the amount of training data available to the neural network. Since the training data, based on the provided 3D volume data V and the provided 2D image data I, is generated in a relevant image recognition task, the amount of training data is increased, taking into account the anatomical regions and the detection of the same pathology, which also improves the accuracy of image data annotation.

[0086] Generally, to leverage the advantages of multi-task learning through joint training with computed tomography (CT) and X-ray data, a hard parameter-sharing approach can be employed, where the first layer of the neural network is reused for multiple tasks. For such applications, the CT data must be tuned. This can be achieved either by selecting individual slices from the CT volume and resizing them through interpolation or by manually forward synthesizing them into 2D images with the correct dimensions. Alternatively, a soft parameter-sharing approach can be used, where each network has its own parameters. This also allows for different data types while employing regularization methods to encourage joint learning.

[0087] like Figure 2 As shown, the trained model M can then be used for further applications. Therefore, the model application unit 50 has the trained model M. For example, the image recognition task that the model application unit 50 should perform includes detecting pneumothorax in 2D image data I generated from X-ray imaging. Therefore, 2D image data I is provided to the model application unit 50. The trained model M is trained on a relatively large amount of training data because the model is trained not only on pathological detection training data in 2D X-ray imaging but also on pathological detection training data of similar regions of the body in 3D computed tomography imaging, which also improves accuracy. Therefore, the model application unit 50 is able to determine the annotated 2D image data I by annotating the provided 2D image data I in an improved manner. L .

[0088] Figure 3 A flowchart of a multi-task deep learning method for a neural network used in automated pathological detection is shown. In a first step S1, first image data I is received for a first image recognition task. In a second step S2, second image data V is received for a second image recognition task, wherein the first image data I is of a first data type, and the second image data V is of a second data type, different from the first data type. The first image recognition task and the second image recognition task relate to the same anatomical region, wherein the corresponding image data are taken from the same pathology to be identified in the corresponding image data and / or related to the same anatomical pathology to be identified in the corresponding image data. In a third step S3, first annotated image data I is determined by labeling the first image data I. L Furthermore, the second annotated and synthesized image data I is determined by synthesizing and annotating the second image data V. SL In the fourth step S4, based on the received first image data I, the received second image data V, and the determined first labeled image data I... L and the determined second labeled synthetic image data I SL To train the neural network.

Claims

1. A multi-task deep learning method for neural networks used in automated pathological detection, comprising the following steps: Receive (S1) first image data (I) for the first image recognition task; Receive (S2) second image data (V) for the second image recognition task; Wherein, the first image data (I) is a first data type, and the second image data (V) is a second data type, the second data type being different from the first data type; The first labeled image data (I) is determined by labeling the first image data (I) (S3). L The second labeled synthesized image data (I) is determined by synthesizing and annotating the second image data (V). SL ); Based on the received first image data (I), the received second image data (V), and the determined first labeled image data (I) L ) and the determined second labeled synthetic image data (I SL (S4) to train the neural network; The first image recognition task and the second image recognition task are related to the same anatomical region, in which the corresponding image data is taken from and / or related to the same pathology to be identified in the corresponding image data.

2. The method according to claim 1, wherein, The method includes the following steps: The synthesized second image data (I) is determined by synthesizing the second image data (V) into the first data type and / or the first dimension. S );and By processing the synthesized second image data (I) S The annotation is used to determine the second annotated synthetic image data (I). SL ).

3. The method according to claim 1, wherein, The method includes the following steps: The second labeled image data is determined by labeling the second image data (V); and The second labeled composite image data (I) is determined by synthesizing the second labeled image data into the first image data (I) using the first data type and / or dimension. SL ).

4. The method according to any one of claims 1-3, wherein, The first image recognition task and / or the second image recognition task include classification, localization, object detection and / or segmentation.

5. The method according to claim 4, wherein, Classification, localization, object detection, and / or segmentation involve pathological, foreign body, and / or anatomical regions.

6. The method according to any one of claims 1-3, wherein, The first data type includes 2D image data; and The second data type includes 3D image data.

7. The method according to any one of claims 1-3, wherein, The first image data (I) has a first dimension, and the second image data (V) has a second dimension, which is different from the first dimension.

8. The method according to any one of claims 1-3, wherein, The first labeled image data (I) L ) and the second labeled synthetic image data (I SL This includes the identification and / or location of pathological, foreign body, and / or anatomical areas.

9. The method according to any one of claims 1-3, wherein, The first image data (I) includes X-ray image data; and The second image data (V) includes computed tomography image data.

10. The method according to claim 9, wherein, The computed tomography image data is synthesized into 2D image data in the form of digitally reconstructed radiographs.

11. The method according to any one of claims 1-3, wherein, The pathology includes one of the following: cardiomegaly, emphysema, edema, hernia, pneumothorax, effusion, mass, fibrosis, atelectasis, consolidation, pleural thickening, nodules, and pneumonia.

12. An apparatus (10) configured to perform the method according to any one of claims 1 to 11.

13. A computer program product comprising instructions that cause a computer to perform the method according to any one of claims 1 to 11.

14. A computer-readable medium having stored thereon instructions that cause a computer to perform the method according to any one of claims 1 to 11.

Images