Evaluation of abnormal patterns associated with COVID-19 from X-ray images

Abstract

The present invention relates generally to assessing abnormal patterns associated with a disease from x-ray images, and in particular to assessing abnormal patterns associated with COVID-19 (coronavirus disease 2019) from x-ray images using a machine learning network trained on DRRs (digital reconstructed radiographs) and ground truth derived from CT (computed tomography). Systems and methods of assessing a disease are provided. An input medical image of a first modality is received. A lung is segmented from the input medical image using a trained lung segmentation network, and an abnormal pattern associated with the disease is segmented from the input medical image using a trained abnormal pattern segmentation network. The trained lung segmentation network and the trained abnormal pattern segmentation network are trained based on 1) synthetic images of the first modality generated from training images of a second modality and 2) target segmentation masks for the synthetic images generated from training segmentation masks for the training images. An assessment of the disease is determined based on the segmented lung and the segmented abnormal pattern.

Description

TECHNICAL FIELD

[0001] The present invention relates generally to assessing abnormal patterns associated with a disease from x-ray images, and in particular to assessing abnormal patterns associated with COVID-19 (coronavirus disease 2019) from x-ray images using a machine learning network trained based on DRRs (digital reconstructed radiographs) and ground truth derived from CTs (computed tomography). BACKGROUND

[0002] COVID-19 is an infectious disease caused by severe acute respiratory syndrome coronavirus 2 (SARS-Cov2). Common symptoms of COVID-19 include fever, cough, and difficulty breathing. In most cases, patients infected with COVID-19 experience mild to moderate symptoms that do not require hospitalization. However, in severe cases, COVID-19 can lead to pneumonia, severe acute respiratory syndrome, multi-organ failure, and death.

[0003] In current clinical workflows, diagnostic imaging is typically performed on patients suspected or confirmed to have COVID-19 to visualize the extent and severity of COVID-19 in the lungs and other organs. One form of diagnostic imaging is CT (computed tomography) imaging, which provides high-resolution 3D images that can be used to assess COVID-19. However, many clinical sites do not have the resources to acquire such CT imaging or to sanitize the CT imaging scanner after acquisition. Another form of diagnostic imaging is x-ray imaging, which is relatively lower in cost and more readily available compared to CT imaging. However, x-ray imaging is inherently ambiguous due to its 2D nature, resulting in inconsistent readings by radiologists.

[0004] Existing machine learning-based approaches have been proposed to classify abnormal patterns in x-ray imaging. Typically, a base machine learning model is trained based on x-ray training images annotated by radiologists. However, given the ambiguity of x-ray images, the performance of machine learning models trained on manually annotated x-ray training images is limited due to uncertainty in the accuracy and consistency of annotations. SUMMARY

[0005] According to one or more embodiments, systems and methods for assessing a disease are provided. An input medical image in a first modality is received. A lung is segmented from the input medical image using a trained lung segmentation network, and an abnormality pattern associated with the disease is segmented from the input medical image using a trained abnormality pattern segmentation network. The trained lung segmentation network and the trained abnormality pattern segmentation network are trained based on 1) a synthetic image in the first modality generated from a training image in a second modality and 2) a target segmentation mask for the synthetic image generated from a training segmentation mask for the training image. An assessment of the disease is determined based on the segmented lung and the segmented abnormality pattern.

[0006] In one embodiment, the disease is COVID-19 (coronavirus disease 2019), and the abnormality pattern includes at least one of a GGO (ground-glass opacity), a consolidation, and a crazy-paving pattern.

[0007] In one embodiment, at least one of the lung segmentation network and the abnormality pattern segmentation network is trained by receiving a training image of an anatomical object of interest in a second modality and a training segmentation mask for the training image, generating a synthetic image in the first modality based on the training image, generating a target segmentation mask for the synthetic image based on the training segmentation mask, and training at least one of the lung segmentation network and the abnormality pattern segmentation network based on the synthetic image and the target segmentation mask.

[0008] In one embodiment, the target segmentation mask is generated by projecting a region of the training image within its corresponding training segmentation mask to a two-dimensional mask, and assigning a value to each pixel in the two-dimensional mask corresponding to a thickness of the anatomical object of interest in the training image that the projection penetrates. The value of each pixel in the two-dimensional mask is compared to a threshold value, and a final value is assigned to each pixel in the two-dimensional mask based on the comparison.

[0009] In one embodiment, the target segmentation mask is generated by projecting a region of the training image within its corresponding training segmentation mask to a two-dimensional mask, and assigning a value to each pixel in the two-dimensional mask corresponding to a sum of intensity values of voxels in the training image that are within the anatomical object of interest and that are projected to penetrate. The value of each pixel in the two-dimensional mask is compared to a threshold value, and a final value is assigned to each pixel in the two-dimensional mask based on the comparison.

[0010] In one embodiment, the assessment of the disease is determined by calculating a percentage of an affected lung area metric based on an area of the lung determined from the segmented lung and an area of the abnormality pattern determined from the segmented abnormality pattern.

[0011] In one embodiment, the disease-based assessment detects a disease in the input medical image.

[0012] In one embodiment, the disease-based assessment and the assessment of the disease determined for different time points are used to predict an evolution of the disease.

[0013] In one embodiment, the first modality is x-ray and the second modality is CT (Computed Tomography).

[0014] These and other advantages of the application will become apparent to those of ordinary skill in the art upon reading the following detailed description and upon having reference to the BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 A workflow for assessing a patient's disease according to one or more embodiments is shown;

[0016] Figure 2 A method for assessing a patient's disease according to one or more embodiments is shown;

[0017] Figure 3 A workflow for training a segmentation network for segmenting an anatomical object of interest based on CT training images according to one or more embodiments is shown;

[0018] Figure 4 A method for training a segmentation network for segmenting an anatomical object of interest from medical images according to one or more embodiments is shown;

[0019] Figure 5 A diagram illustrating a calculation of pixel values for generating a thickness mask or an intensity mask according to one or more embodiments is shown;

[0020] Figure 6 A chart showing a correlation between a percent of affected lung area (POa) and a percent of affected lung volume (POv) according to one or more embodiments is shown;

[0021] Figure 7 An exemplary image of a lung segmentation segmented by a segmentation network trained according to one or more embodiments is shown;

[0022] Figure 8 Various images depicting an assessment of COVID-19 (Coronavirus Disease 2019) according to one or more embodiments are shown;

[0023] Figure 9 A chart comparing POa calculated according to one or more embodiments with POv of an experimental validation dataset is shown;

[0024] Figure 10 An exemplary artificial neural network that can be used to implement one or more embodiments described herein is shown;

[0025] Figure 11 A convolutional neural network that can be used to implement one or more embodiments described herein is shown; and

[0026] Figure 12 A high-level block diagram of a computer that can be used to implement one or more embodiments described herein is shown. DETAILED DESCRIPTION

[0027] The present invention relates generally to methods and systems for assessing abnormal patterns associated with COVID-19 (Coronavirus Disease 2019) from x-ray images. Embodiments of the present invention are described herein to give a visual understanding of such methods and systems. Digital images are often composed of digital representations of one or more objects (or shapes). Descriptions of the digital representations of objects are often described herein in terms of identifying and manipulating objects. Such manipulations are often virtual manipulations accomplished in the memory or other circuitry / hardware of a computer system. Accordingly, it should be understood that embodiments of the present invention can be performed within a computer system using data stored within the computer system.

[0028] COVID-19 is an infectious disease that typically exhibits a wide range of symptoms from asymptomatic to mild, moderate, severe, and critical. Chest diagnostic images of patients with COVID-19 can show radiographic abnormal patterns associated with COVID-19. One form of diagnostic imaging is x-ray images or radiographs, which are relatively low cost to acquire and readily available. However, such x-ray images are inherently ambiguous due to their 2D nature, resulting in inconsistent and unreliable annotations of the lungs, abnormal patterns, and other anatomical objects of interest in the x-ray images by radiologists. Therefore, machine learning-based systems trained using such manually annotated x-ray images have limited accuracy and performance.

[0029] The embodiments described herein provide for segmenting lungs and abnormal patterns from x-ray images using a machine learning based segmentation network, thereby evaluating abnormal patterns associated with COVID-19 from x-ray images. Such a segmentation network is trained using synthetic x-ray images generated from 3D CT training images and ground truth target segmentation masks generated from training segmentation masks annotated by a radiologist (or any other user) from 3D CT training images. Due to the 3D nature of 3D CT images, it is easier for a radiologist to annotate lungs, abnormal patterns, or any other anatomical object of interest from 3D CT images more accurately and consistently compared to x-ray images. Advantageously, according to the embodiments described herein, a segmentation network trained using synthetic x-ray images and target segmentation masks for the synthetic x-ray images segments lungs and abnormal patterns from x-ray images with improved accuracy and performance and is not limited by the inherent blurriness in x-ray images.

[0030] Figure 1 is a workflow 100 for evaluating a patient for a disease according to one or more embodiments. Figure 2 is a method 200 for evaluating a patient for a disease according to one or more embodiments. Together, the method 200 will be described Figure 1 and Figure 2 . The steps of the method 200 can be performed by one or more suitable computing devices, such as Figure 11 the computer 1102 of the system 1000.

[0031] At step 202, an input medical image of a first modality is received. The input medical image can be a medical image of a chest of a patient suspected or confirmed to have a disease. In one embodiment, the disease is a member of the coronavirus family. For example, the disease can be COVID-19. As used herein, COVID-19 includes mutations of the COVID-19 virus (which can be referred to by different terms). However, the disease can include any disease having identifiable abnormal patterns in the lungs, such as, for example, consolidation, interstitial disease, atelectasis, nodules, masses, decreased density or lucency, etc. For example, the disease can be other types of viral pneumonia (e.g., influenza, adenovirus, respiratory syncytial virus, SARS (severe acute respiratory syndrome), MERS (Middle East respiratory syndrome), etc.), bacterial pneumonia, fungal pneumonia, mycoplasma pneumonia, or other types of pneumonia or other types of diseases.

[0032] The input medical image can depict the lungs of a patient and can depict radiographic abnormality patterns associated with a disease in the patient’s lungs. For example, in the case that the disease is COVID-19, the abnormality patterns can include opacities such as, for example, GGOs (ground-glass opacities), consolidations, crazy paving, atelectasis, interlobular septal thickening, pleural effusion, bronchiectasis, halo sign, etc.

[0033] In one embodiment, the first modality is 2D x-ray imaging or 2D radiography. For example, the input medical image can be an x-ray image 102 of Figure 1 However, the first modality can be any suitable modality such as, for example, CT (computed tomography), MRI (magnetic resonance imaging), US (ultrasound), or any other modality or combination of modalities. The input medical image can comprise a 2D image or a 3D volume and can comprise a single image or multiple images (e.g., a sequence of images acquired over time). The input medical image can be received directly from an image acquisition device such as, for example, from an x-ray scanner when the input medical image is acquired, or can be received by loading a previously acquired input medical image from a storage device or memory of the computer system or receiving the input medical image from a remote computer system.

[0034] At step 204, the lungs are segmented from the input medical image using the trained lung segmentation network. The lung segmentation network predicts a probability map representing the segmented lungs. The probability map defines a pixel-by-pixel probability that each pixel depicts the lungs. By comparing the probability of each pixel to a threshold (e.g., 0.5), the probability map can be represented as a binary mask. The binary mask assigns a value to each pixel, for example, the value is 0 if the pixel does not depict the lungs and 1 if the pixel depicts the lungs. In one example, the lung segmentation network is the lung segmentation network 110, which generates a predicted 2D probability map 112 represented as a binary mask 114 in Figure 1 Figure 6 An exemplary lung segmentation is shown in

[0035] ​In one embodiment, the lung segmentation network is an image-to-image CNN (convolutional neural network), however the lung segmentation network can be any suitable machine learning based network. During a previous offline or training phase, the lung segmentation network is trained based on 1) synthetic images in the first modality generated from training images in the second modality and 2) lung ground truth segmentation masks for the synthetic images generated from lung training segmentation masks for the training images. In one embodiment, the second modality is 3D CT imaging. However, the second modality can be any suitable modality such as, for example, x-ray, MRI, US, or any other modality or combination of modalities, and can include 2D images or 3D volumes. The training of the lung segmentation network is further described below with reference to Figure 3 and Figure 4 The lung segmentation network is applied during an online or inference phase at step 204 once trained.

[0036] At step 206, the trained abnormality pattern segmentation network is used to segment the abnormality pattern associated with the disease from the input medical image. The abnormality pattern segmentation network predicts a probability map representing the segmented abnormality pattern. The probability map defines a pixel-wise probability that each pixel depicts the abnormality pattern. The probability map can be represented as a binary mask by comparing the probability of each pixel to a threshold (e.g., 0.5). The binary mask assigns a value to each pixel, for example, the value is 0 if the pixel does not depict the abnormality pattern and 1 if the pixel depicts the abnormality pattern. In one example, the abnormality pattern segmentation network is the lesion segmentation network 104, which generates a predicted 2D probability map 106 represented as a binary mask 108 in Figure 1

[0037] In one embodiment, the abnormality pattern segmentation network is an image-to-image CNN, however the abnormality pattern segmentation network can be any suitable machine learning based network. During a previous offline or training phase, the abnormality pattern segmentation network is trained based on 1) synthetic images in the first modality generated from training images in the second modality and 2) abnormality pattern ground truth segmentation masks for the synthetic images generated from abnormality pattern training segmentation masks for the training images. The training of the abnormality pattern segmentation network is further described below with reference to Figure 3 and Figure 4 The training of the abnormality pattern segmentation network is further described below. The abnormality pattern segmentation network is applied during an online or inference phase at step 206 once trained.

[0038] At step 208, an assessment of the disease is determined based on the segmented lung and the segmented abnormality pattern. In one embodiment, the assessment of the disease is determined by computing a quantitative metric that quantifies the disease. For example, the quantitative metric can be a quantification 116 of the disease determined from the binary mask 108 of the lesion and the binary mask 114 of the lung in Figure 1 Figure 1 At step 208, an assessment of the disease is determined based on the segmented lung and the segmented abnormality pattern. In one embodiment, the assessment of the disease is determined by computing a quantitative metric that quantifies the disease. For example, the quantitative metric can be a quantification 116 of the disease determined from the binary mask 108 of the lesion and the binary mask 114 of the lung in

[0039] In one embodiment, the quantitative measure is the percentage of affected lung area (POa), which is computed as the percentage of total lung area affected by the disease, as defined in equation (1):

[0040]

[0041] where the area of abnormal patterns in the lung is determined as the area of segmented abnormal patterns, and the lung area is determined as the area of segmented lung. The quantitative measure can be any other measure suitable to quantify the disease, such as, for example, LSS (Lung Severity Score), which is computed for each lobe of the lung as the percentage of total lobe area affected by the disease.

[0042] At step 210, the assessment of the disease is output. For example, the assessment of the disease can be output by displaying the assessment of the disease on a display device of the computer system, storing the assessment of the disease on a storage device or memory of the computer system, or by transmitting the assessment of the disease to a remote computer system.

[0043] The embodiments described herein provide for assessing a disease using a segmentation network trained using synthetic x-ray images and ground truth target segmentation masks generated from annotated CT training images, which are relatively easier to annotate by radiologists due to the 3D nature of the CT training images and the inherent loss of anatomical details in 2D projection images due to tissue and anatomical overlap, as compared to 2D x-ray images. According to the embodiments described herein, the segmentation network is not trained based solely on annotated x-ray images, thereby avoiding the inherent uncertainty of x-ray images. Advantageously, the embodiments described herein provide for segmentation of the lung and abnormal patterns of the disease with improved accuracy and performance, thereby providing for assessment of the disease with improved accuracy.

[0044] In one embodiment, an ensemble (i.e., multiple) segmentation networks can be applied to model uncertainty. Given that there is limited visibility of volumetric structures when projecting 3D CT images to DRR images (Digital Reconstructed Radiographs), there is inherent uncertainty in both the mapping between DRR images (synthetic x-ray images) and target lung segmentation masks and the mapping between DRR images and target abnormal pattern segmentation masks. Using standard training procedures that ignore such uncertainty, one can expect per-sample bias in the estimation of lesion regions. In one embodiment, during the training phase or the testing phase, an ensemble segmentation network is utilized to model the uncertainty. For example, multiple segmentation networks can be trained on the same training data, and predictions from each segmentation network can be averaged, thereby accounting for the uncertainty.

[0045] In one embodiment, a machine learning based detection network can additionally be applied to detect diseases (e.g., COVID-19) in the input medical images. In one embodiment, the detection can be formulated as a mapping from the feature space of the segmentation network, as well as lung and abnormal pattern segmentation, to a disease score or probability measure of disease using an image-wise disease classifier or detector (e.g., bounding box). In another embodiment, the detection can be performed by using extracted quantitative biomarkers (e.g., percentage of opacity). Additional clinical data can also be input into the detection network. The additional clinical data can include patient data (e.g., demographic features), clinical data, genetic data, laboratory data, etc.

[0046] In one embodiment, temporal changes in x-ray images can be detected and quantified with high quality annotations of CT images. Detecting and quantifying temporal changes can include analyzing evolution of existing lesions (e.g., capturing increase in size of lung nodules), detecting and quantifying new lesions, and tracking geometric changes of organs.

[0047] In one embodiment, evolution or progression of disease can be predicted. Based on the assessment of disease determined at multiple time points, as well as possible detection of disease (using the detection network), a wide range of measurements can be extracted, such as, for example, POa, location and extent of lesions, symmetry, etc. By integrating the measurements over time with subsequent radiological images, such measurements can be used to predict evolution of disease, allowing early detection of trends in patient condition. For example, one can predict severity of disease, assess need for future intubation and projected period of intubation, etc. Such prognostic information enables more efficient management of disease and patients. Moreover, in low resource locations, the information can enable better management of valuable or scarce resources, such as, for example, hospital beds, ventilators, etc. In one embodiment, evolution of disease is predicted by generating quantitative biomarkers (e.g., percentage of opacity) from x-ray images at each of the multiple time points. The biomarkers can be recorded in a database and tracked over different longitudinal studies to understand effectiveness of different treatments. In another embodiment, spatial locations of detection and segmentation determined according to the embodiments described herein can also be used to track expansion or contraction of lesions in the same region or new or disappeared lesions using image registration techniques. Since the embodiments described herein train the network using CT images with higher reading confidence, disease tracking accuracy is superior to systems trained on x-ray annotations alone.

[0048] In one embodiment, the disease is not limited to lung disease. The embodiments described herein can be extended for use in the evaluation of any disease affecting any organ for which radiological imaging is used, which can be correlated with more advanced imaging modalities (e.g., CT or MR). For example, the embodiments described herein can be extended to detect and monitor pathological abnormalities of abdominal cancers (e.g., colon cancer, pancreatic cancer, hepatocellular carcinoma, renal cell carcinoma, etc.), gallbladder, liver, or pancreas, etc.

[0049] Figure 3 A workflow 300 for training a segmentation network for segmenting an anatomical object of interest based on CT training images is shown in accordance with one or more embodiments. Figure 4 A method 400 for training a segmentation network for segmenting an anatomical object of interest from a medical image is shown in accordance with one or more embodiments. Figure 3 and Figure 4 will be described together. The steps of the method 400 can be performed by one or more suitable computing devices, such as Figure 11 the computer 1102 of the system 1000.

[0050] The steps of the workflow 300 and the method 400 are performed during an offline or training phase to train a segmentation network for segmenting an anatomical object of interest from a medical image. The steps of the workflow 300 and the method 400 can be performed any number of iterations to separately train segmentation networks for segmenting different anatomical objects of interest from a medical image. Once trained, the segmentation networks are applied during an online or testing phase. In one embodiment, the steps of the workflow 300 and the method 400 can be repeatedly performed to train a lung segmentation network applied in step 204 of the method 400 during a first iteration, and to train an abnormality pattern segmentation network applied in step 206 of the method 400 during a second iteration. Figure 2 the lung segmentation network of the system 1000. Figure 2 the abnormality pattern segmentation network of the system 1000.

[0051] At step 402, a training image in a second modality and a training segmentation mask for the training image are received. The training image depicts the anatomical object of interest that the segmentation network is being trained to segment. In one embodiment, the anatomical object of interest includes the lungs of a patient, and can include abnormality patterns associated with a disease in the lungs of the patient. As discussed above with respect to the workflow 300, in one embodiment, the second modality is 3D CT imaging. However, the second modality can be any suitable modality or combination of modalities. In one example, the training image and the segmentation mask can be annotated CT images 302 in the dataset 304 of the system 1000. Figure 2 Figure 3

[0052] ​​The training segmentation mask for the training image identifies the anatomical object of interest in the training image that the segmentation network is being trained to segment. The training segmentation mask is manually annotated by a radiologist in the training image. In one embodiment, for example, where the training image is a 3D CT training image, the training segmentation mask is a 3D training segmentation mask. Due to the 3D nature of 3D CT images, it is easier to perform manual annotation of a 3D CT training image by a radiologist to generate a 3D training segmentation mask than a 2D x-ray image.

[0053] The training image and the training segmentation mask can be received by loading the training image and the training segmentation mask from a storage device or memory of the computer system or receiving the training image and the training segmentation mask from a remote computer system.

[0054] At step 404, a synthetic image in a first modality is generated based on the training image. As discussed above with respect to Figure 2 the first modality is 2D x-ray imaging. However, the first modality can be any suitable modality or combination of modalities. The synthetic image can be generated by converting the training image from the second modality to the first modality using any suitable technique.

[0055] In one embodiment, as shown in Figure 3 the synthetic 2D x-ray image is generated from the 3D CT training image 302 by first performing integral projection to project the 3D CT training image 302 to a 2D DRR (digital reconstructed radiograph) image 304 using Beer's law to approximate the physical x-rays traveling through the tissue. Typically, the x-ray image has a high resolution. However, the projected DRR image 304 has a relatively lower resolution. Therefore, the DRR image 304 is upsampled using a 3D super-resolution network to generate an upsampled DRR image 306. The upsampled DRR image 306 then undergoes frequency domain normalization to normalize the intensities of the upsampled DRR image 306, thereby generating a normalized upsampled DRR image 308. The frequency domain normalization addresses the limited contrast and high noise in the upsampled DRR image 306 resulting from the projection of the 3D CT training image 302 to the 2D DRR image 304. The normalized upsampled DRR image 308 represents the synthetic x-ray image.

[0056] It should be appreciated that any other suitable technique can be used to generate the synthetic image. For example, a machine learning based network such as, for example, a generative adversarial network can be used to generate the synthetic image from the training image.

[0057] At step 406, a target segmentation mask for the synthetic image is generated based on the training segmentation mask. The target segmentation mask represents the ground truth identification of the target anatomical object of interest in the synthetic image.

[0058] In one embodiment, for example, in the case where the training segmentation mask is a 3D training segmentation mask of the anatomical object of interest of a 3D CT training image, the regions of the 3D CT training image within its corresponding 3D training segmentation mask are projected to a 2D mask (e.g., using Beer's law). Each pixel in the 2D mask is assigned a value corresponding to the thickness or depth (e.g., in millimeters) of the anatomical object of interest in the projected x-ray penetrating 3D CT training image for that pixel to generate the 2D target segmentation mask. Thus, the 2D target segmentation mask generated according to this embodiment can be considered a "thickness mask." In the 3D CT training image, the thickness mask has a stronger correlation with the geometric information. Referring temporarily to Figure 5 FIG. 500 illustrates the calculation of pixel values of a 2D target segmentation mask according to one or more embodiments. As shown in FIG. 500, an anatomical object 502 (e.g., a lung or an abnormal pattern) in a 3D CT training image is projected to a 2D projection plane 506 according to a projection ray 504. The projection ray 504 is within the anatomical object 502 at a region 508. According to one embodiment, the value assigned to a pixel 510 of the 2D projection plane 506 is determined as the distance of the projection ray 504 within the region 508 to form a thickness mask.

[0059] Returning to step 406 of Figure 4 In another embodiment, for example, in the case where the training segmentation mask is a 3D training segmentation mask of the anatomical object of interest of a 3D CT training image, the regions of the 3D CT training image within its corresponding 3D training segmentation mask are projected to a 2D mask (e.g., using Beer's law). Each pixel in the 2D target segmentation mask is assigned a value corresponding to the sum of intensity values of voxels in the 3D CT training image within the anatomical object of interest and penetrated by the projection of the x-ray for that pixel to generate the 2D target segmentation mask. Thus, the 2D target segmentation mask generated according to this embodiment can be considered an "intensity mask." In the 3D CT training image, the intensity mask has a stronger correlation with the visibility. Referring temporarily to Figure 5 According to one embodiment, the value assigned to a pixel 510 of the 2D projection plane 506 is determined as the sum of intensity values along the projection ray 504 within the region 508 to form an intensity mask.

[0060] Other techniques for generating a target segmentation mask are also contemplated.

[0061] In one embodiment, the thickness mask and / or the intensity mask are thresholded to minimize the numerical difference between quantification metrics that would be derived from the 2D synthetic x-ray images and the corresponding 3D CT images. For example, in the case where the quantification metric is POa, a threshold can be determined to minimize the absolute error between the POa of the 2D synthetic x-ray images and the percentage of affected lung volume (POv) of the 3D CT images. The thickness mask and / or the intensity mask are thresholded by comparing each pixel of the 2D thickness mask and / or the intensity mask to the threshold and assigning a final value of 0 or 1 to the pixel based on the comparison (e.g., if the pixel value in the thickness mask and / or the intensity mask does not satisfy the threshold, the pixel is assigned 0, and if the pixel value satisfies the threshold, the pixel is assigned 1). Figure 6 A graph 600 is shown that illustrates the correlation between POa and POv using a threshold of 25,000 on the intensity mask, in accordance with one or more embodiments. The threshold of 25,000 achieves a mean absolute error (MAE) of 3.28%.

[0062] At step 408, a segmentation network is trained based on the synthetic images and the target segmentation masks. The segmentation network is trained to learn the mapping between the synthetic images and the target segmentation masks. In one embodiment, the segmentation network is an image-to-image CNN. However, the segmentation network can be any suitable machine learning based network.

[0063] In one embodiment, the segmentation network is initially pre-trained using a large-scale natural image dataset for parameter initialization. Alternatively, a large number of annotated x-ray images can be used for pre-training. The segmentation network is then trained using a combination of positive synthetic x-ray images, negative synthetic x-ray images, and true negative x-ray images that are confirmed by a user. True negative x-ray images are real x-ray images that do not depict abnormal patterns related to the disease. True positive x-ray images are not used to train the segmentation network as it is difficult to accurately and consistently annotate abnormal patterns in x-ray images given the inherent ambiguity of x-ray images. The positive synthetic x-ray images, negative synthetic x-ray images, and true negative x-ray images can be selected randomly or according to a predefined weighting scheme to balance the images. The segmentation network can be optimized using a pixel-wise loss (e.g., cross-entropy loss) or an intersection-based loss (e.g., soft dice loss). The final parameters and segmentation network model are selected based on a validation dataset (e.g., 10% of the training data that is reserved).

[0064] In one example, as Figure 3The segmentation network 314 is trained based on a combination of the positive and negative normalized upsampled DRR images 308 and the true negative x-ray images 310 (and their associated target segmentation masks), as shown. The images are the positive and negative normalized upsampled DRR images 308, and the true negative x-ray images 310 are selected by the data balancer 312 to balance the images. The segmentation network 314 is trained according to the training loss for backpropagation to optimize (minimize) the difference between the predicted 2D probability maps 316 and the target segmentation masks 318.

[0065] At step 410, the trained segmentation network is output. For example, the trained segmentation network can be output by storing the trained segmentation network on a memory or storage device of the computer system, or by transmitting the trained segmentation network to a remote computer system, for use during an online or testing phase (e.g., in step 206 of FIG. 2). Figure 2

[0066] Figure 7 Exemplary images 700 of lung segmentations by a segmentation network trained according to one or more embodiments described herein are shown. Image 702 shows a lung segmentation according to a thickness mask. Image 704 shows a thresholded lung segmentation mask according to a thickness mask. Image 706 shows a lung segmentation according to an intensity mask. Image 708 shows a thresholded lung segmentation according to an intensity mask.

[0067] Figure 8 Various images 800 depicting an evaluation of COVID-19 according to one or more embodiments are shown. Image 802 is a DRR image representing a synthetic x-ray image that is input into a lung and abnormality pattern segmentation network. Image 804 depicts segmentation of an abnormality pattern in the DRR image 802 (DRR of lesion only) using an intensity mask. Image 806 shows a target lung segmentation mask 814 (shown as a contoured region) and a target abnormality pattern segmentation mask 816 (shown as a shaded region) overlaid on the DRR image 802. Image 808 shows an output of the evaluation of COVID-19 (DRR model output) that depicts a predicted segmentation of a lung 818 (shown as a contoured region) and an abnormality pattern 820 (shown as a shaded region) overlaid on the DRR image 802, which results in a POa of 43.55%. Image 810 shows a true x-ray image, and image 812 shows an output of the evaluation of COVID-19 (x-ray model output) that depicts a predicted segmentation of a lung 822 (shown as a contoured region) and an abnormality pattern 824 (shown as a shaded region) from the x-ray image 810, which results in a POa of 38.17%.

[0068] ​The embodiments described herein were experimentally validated using a dataset comprising real x-ray images and CT image pairs collected from the same patient within 48 hours of each other. The POa determined from the predicted segmentation of the 2D x-ray images was compared to the POv determined from the manual annotation of the 3D CT images. Figure 9 A graph 900 comparing POa and POv is shown. Each point in the graph 900 represents a comparison of POa and POv for a pair of x-ray images and CT images. As shown in the graph 900, most points lie near the diagonal, representing an exact correlation between POa and POv. However, some points deviate from the diagonal. This can be attributed to a change in patient condition (e.g., improvement or decline), or can be attributed to limited visibility in the x-ray images.

[0069] The embodiments described herein are described in relation to the claimed system as well as the claimed method. Features, advantages or alternative embodiments herein can be assigned to other claimed objects and vice versa. In other words, claims of the system can be improved with features described or claimed in the context of the method. In this case, the functional features of the method are embodied by the target unit providing the system.

[0070] Further still, the embodiments described herein are described in relation to methods and systems for assessing abnormal patterns associated with COVID-19 from x-ray images using a machine learning based segmentation network, as well as methods and systems for training a machine learning based segmentation network for assessing abnormal patterns associated with COVID-19. Features, advantages or alternative embodiments herein can be assigned to other claimed objects and vice versa. In other words, claims of the methods and systems for training a machine learning based network can be improved with features described or claimed in the context of the methods and systems for utilizing the trained machine learning based network, and vice versa.

[0071] In particular, the trained machine learning based segmentation network for assessing abnormal patterns associated with COVID-19 from x-ray images can be adapted by the methods and systems for training a machine learning based segmentation network for assessing abnormal patterns associated with COVID-19 from x-ray images. Further still, the input data of the trained machine learning based segmentation network can comprise advantageous features and embodiments of the training input data, and vice versa. Further still, the output data of the trained machine learning based segmentation network can comprise advantageous features and embodiments of the output training data, and vice versa.

[0072] Generally speaking, trained machine learning-based networks mimic the cognitive functions that humans associate with other human minds. In particular, by being trained based on training data, trained machine learning-based networks are able to adapt to new environments and detect and infer patterns.

[0073] Generally speaking, the parameters of a machine learning-based network can be adapted through training. In particular, supervised training, semi-supervised training, unsupervised training, reinforcement learning, and / or active learning can be used. Furthermore, representation learning (an alternative term is "feature learning") can be used. In particular, the parameters of a trained machine learning-based network can be iteratively adapted through several training steps.

[0074] In particular, the trained machine learning-based network may include a neural network, a support vector machine, a decision tree, and / or a Bayesian network, and / or the trained machine learning-based network may be based on k-means clustering, Q-learning, a genetic algorithm, and / or association rules. In particular, the neural network may be a deep neural network, a convolutional neural network, or a convolutional deep neural network. Furthermore, the neural network may be an adversarial network, a deep adversarial network, and / or a generative adversarial network.

[0075] Figure 10 An embodiment of an artificial neural network 1000 according to one or more embodiments is shown. Alternative terms for "artificial neural network" are "neural network", "artificial neural network" or "neural network". The machine learning networks described herein, such as Figure 1 The lesion segmentation network 104 and the lung segmentation network 110, Figure 2 Step 204 applies the lung segmentation network and Figure 2 Step 206 applies the abnormal pattern segmentation pattern network, Figure 3 The segmentation network 314 and Figure 4 The segmentation network trained in step 408 can be implemented using the artificial neural network 1000.

[0076] The artificial neural network 1000 includes nodes 1002-1022 and edges 1032, 1034, ..., 1036, wherein each edge 1032, 1034, ..., 1036 is a directed connection from a first node 1002-1022 to a second node 1002-1022. Typically, the first node 1002-1022 and the second node 1002-1022 are different nodes 1002-1022, or the first node 1002-1022 and the second node 1002-1022 may be the same. For example, Figure 10In this embodiment, edge 1032 is a directed connection from node 1002 to node 1006, and edge 1034 is a directed connection from node 1004 to node 1006. Edges 1032, 1034,..., 1036 from the first nodes 1002-1022 to the second nodes 1002-1022 are also denoted as "in-edges" of the second nodes 1002-1022 and "out-edges" of the first nodes 1002-1022.

[0077] In this embodiment, nodes 1002-1022 of artificial neural network 1000 can be arranged in layers 1024-1030, wherein the layers can comprise an inherent order introduced by edges 1032, 1034,..., 1036 between nodes 1002-1022. In particular, edges 1032, 1034,..., 1036 can only exist between adjacent layers of nodes. In this embodiment, the layers 1024-1030 are arranged in a feed-forward manner, i.e. the output of a layer is used as input for the next layer. Figure 10 In the illustrated embodiment, there are an input layer 1024 comprising only nodes 1002 and 1004 without in-edges, an output layer 1030 comprising only node 1022 without out-edges, and hidden layers 1026, 1028 between input layer 1024 and output layer 1030. In general, the number of hidden layers 1026, 1028 can be chosen arbitrarily. The number of nodes 1002 and 1004 within input layer 1024 is typically related to the number of input values of neural network 1000, and the number of nodes 1022 within output layer 1030 is typically related to the number of output values of neural network 1000.

[0078] In particular, a number (a real number) can be assigned as a value to each node 1002-1022 of neural network 1000. Herein, denotes the value of the i-th node 1002-1022 of the n-th layer 1024-1030. The values of nodes 1002-1022 of input layer 1024 are identical to the input values of neural network 1000, and the value of node 1022 of output layer 1030 is identical to the output value of neural network 1000. Further, each edge 1032, 1034,..., 1036 can comprise a weight being a real number, in particular, the weight is a real number within the interval [-1, 1] or within the interval [0, 1]. Herein, denotes the weight of the edge between the i-th node 1002-1022 of the m-th layer 1024-1030 and the j-th node 1002-1022 of the n-th layer 1024-1030. Further, for the weight The following abbreviations are defined .

[0079] In particular, for calculating the output values of the neural network 1000, the input values are propagated through the neural network. In particular, the values of the nodes 1002-1022 of the (n+1)-th layer 1024-1030 can be calculated based on the values of the nodes 1002-1022 of the n-th layer 1024-1030 by

[0080] .

[0081] Herein, the function f is a transfer function (another term is "activation function"). Known transfer functions are a step function, a sigmoid function (e.g. logistic function, generalized logistic function, hyperbolic tangent, arctangent function, error function, smooth step function) or a rectifier function. The transfer function is mainly used for normalization purposes.

[0082] In particular, the values are propagated through the neural network layer by layer, wherein the values of the input layer 1024 are given by the input of the neural network 1000, wherein the values of the first hidden layer 1026 can be calculated based on the values of the input layer 1024 of the neural network, wherein the values of the second hidden layer 1028 can be calculated based on the values of the first hidden layer 1026, etc.

[0083] For setting the values for the edges , the neural network 1000 has to be trained using training data. In particular, the training data comprises training input data and training output data (denoted as t i ). For a training step, the neural network 1000 is applied to the training input data to generate calculated output data. In particular, the training data and the calculated output data comprise a number of values, which number is equal to the number of nodes of the output layer.

[0084] In particular, a comparison between the calculated output data and the training data is used to recursively adapt the weights within the neural network 1000 (backpropagation algorithm). In particular, the weights are changed according to

[0085]

[0086] wherein is a learning rate and wherein the number can be recursively calculated as

[0087]

[0088] and wherein, if the (n+1)-th layer is the output layer 1030, the number

[0089]

[0090] ​wherein is the first derivative of the activation function, and is the comparison training value of the jth node of the output layer 1030.

[0091] Figure 11 A convolutional neural network 1100 according to one or more embodiments is shown. The machine learning networks described herein, such as for example the lesion segmentation network 104 and the lung segmentation network 110 of Figure 1 the step 204 of the method 200 of Figure 2 the step 206 of the method 200 of Figure 2 the segmentation network 314 of the method 300 of Figure 3 the step 408 of the method 400 of Figure 4 may be implemented using the convolutional neural network 1100.

[0092] In the embodiment shown in Figure 11 the convolutional neural network 1100 comprises an input layer 1102, convolutional layers 1104, pooling layers 1106, fully connected layers 1108, and an output layer 1110. Alternatively, the convolutional neural network 1100 can comprise several convolutional layers 1104, several pooling layers 1106, and several fully connected layers 1108, as well as other types of layers. The order of the layers can be chosen arbitrarily, typically the fully connected layers 1108 are used as the last layers before the output layer 1110.

[0093] In particular, within the convolutional neural network 1100, the nodes 1112-1120 of one layer 1102-1110 can be considered to be arranged as a d-dimensional matrix or d-dimensional image. In particular, in the two-dimensional case, the values of the nodes 1112-1120 indexed by i and j in the n-th layer 1102-1110 can be denoted as x (n) [i,j] However, the arrangement of the nodes 1112-1120 of one layer 1102-1110 has no influence on the calculations performed within the convolutional neural network 1100 as such, as these are given by the structure of the edges and the weights only.

[0094] In particular, the convolutional layers 1104 are characterized by a structure of the edges and the weights of the input edges forming a convolution operation based on a certain number of kernels. In particular, the structure of the edges and the weights of the input edges is chosen such that the value of a node 1114 of the convolutional layer 1104 is computed based on the values of the nodes 1112 of the preceding layer 1102 as a convolution wherein the convolution is defined in the two-dimensional case as

[0095] .

[0096] Here, the kth core is a d-dimensional matrix (a two-dimensional matrix in this embodiment), which is typically smaller than the number of nodes 1112-1118 (e.g., a 3×3 matrix or a 5×5 matrix). In particular, this means that the weights of the incoming edges are not independent, but are chosen so that they produce the convolution equation. In particular, for a kernel that is a 3×3 matrix, there are only 9 independent weights (one for each entry of the kernel matrix), regardless of the number of nodes 1112-1120 in the corresponding layer 1102-1110. In particular, for a convolutional layer 1104, the number of nodes 1114 in the convolutional layer is equal to the number of nodes 1112 in the previous layer 1102 multiplied by the number of kernels.

[0097] If the nodes 1112 of the previous layer 1102 are arranged as a d-dimensional matrix, then using multiple kernels can be interpreted as adding another dimension (denoted as the "depth" dimension), so that the nodes 1114 of the convolution layer 1114 are arranged as a (d+1)-dimensional matrix. If the nodes 1112 of the previous layer 1102 are already arranged as a (d+1)-dimensional matrix including the depth dimension, then using multiple kernels can be interpreted as expanding along the depth dimension, so that the nodes 1114 of the convolution layer 1104 are also arranged as a (d+1)-dimensional matrix, where the size of the (d+1)-dimensional matrix relative to the depth dimension is larger than in the previous layer 1102, which is a multiple of the number of kernels in the previous layer 1102.

[0098] An advantage of using convolutional layers 1104 is that spatially local correlations in the input data can be exploited by enforcing local connectivity patterns between nodes in adjacent layers, in particular by having each node connected only to a small region of nodes in the previous layer.

[0099] exist Figure 11 In the illustrated embodiment, the input layer 1102 includes 36 nodes 1112 arranged as a two-dimensional 6×6 matrix. The convolution layer 1104 includes 72 nodes 1114 arranged as two two-dimensional 6×6 matrices, each of which is the result of convolving the values ​​of the input layer with the kernel. Equivalently, the nodes 1114 of the convolution layer 1104 can be interpreted as being arranged as a three-dimensional 6×6×2 matrix, where the last dimension is the depth dimension.

[0100] The pooling layer 1106 may be characterized by the structure and weights of the input edges and the activation function of its nodes 1116 forming a pooling operation based on a nonlinear pooling function f. For example, in a two-dimensional case, the value of the node 1116 of the pooling layer 1106 is Can be based on the value of node 1114 of the previous layer 1104 The calculation is as follows

[0101] .

[0102] In other words, by using the pooling layer 1106, the number of nodes 1114, 1116 can be reduced by replacing the nodes in the previous layer 1104 with a single node 1116. The single node 1116 is implemented by a number of adjacent nodes 1114, and the single node 1116 is calculated as a function of the values ​​of the number of adjacent nodes in the pooling layer. In particular, the pooling function f can be a maximum function, an average value, or an L2 norm. In particular, for the pooling layer 1106, the weights of the input edges are fixed and are not modified by training.

[0103] The advantage of using the pooling layer 1106 is that the number of nodes 1114, 1116 and the number of parameters are reduced. This results in a reduction in the amount of computation in the network and controls overfitting.

[0104] exist Figure 11 In the embodiment shown, pooling layer 1106 is max pooling, which replaces four adjacent nodes with only one node whose value is the maximum of the four adjacent nodes. Max pooling is applied to each d-dimensional matrix of the previous layer; in this embodiment, max pooling is applied to each of the two 2D matrices, thereby reducing the number of nodes from 72 to 18.

[0105] The fully connected layer 1108 is characterized by the fact that there are most, in particular all, edges between the nodes 1116 of the previous layer 1106 and the nodes 1118 of the fully connected layer 1108, and wherein the weight of each of the edges can be adjusted individually.

[0106] In this embodiment, nodes 1116 of the previous layer 1106 of the fully connected layer 1108 are displayed as a two-dimensional matrix, and are additionally displayed as non-relevant nodes (indicated as a row of nodes, where the number of nodes is reduced for better presentation). In this embodiment, the number of nodes 1118 in the fully connected layer 1108 is equal to the number of nodes 1116 in the previous layer 1106. Alternatively, the number of nodes 1116, 1118 may be different.

[0107] Furthermore, in this embodiment, the values ​​of the nodes 1120 of the output layer 1110 are determined by applying the Softmax function to the values ​​of the nodes 1118 of the previous layer 1108. By applying the Softmax function, the sum of the values ​​of all the nodes 1120 of the output layer 1110 is 1, and all the values ​​of all the nodes 1120 of the output layer are real numbers between 0 and 1.

[0108] The convolutional neural network 1100 can also include ReLU (rectified linear unit) layers or activation layers with a non-linear transfer function. In particular, the number and structure of nodes included in a ReLU layer are identical to the number and structure of nodes included in the previous layer. In particular, the value of each node in the ReLU layer is computed by applying a rectification function to the value of the corresponding node of the previous layer.

[0109] The inputs and outputs of different convolutional neural network blocks can be connected using summation (residual / dense neural networks), element-wise multiplication (attention), or other differentiable operators. Thus, the convolutional neural network architecture can be nested, rather than sequential, if the entire pipeline is differentiable.

[0110] In particular, the convolutional neural network 1100 can be trained based on a backpropagation algorithm. To prevent overfitting, regularization methods can be used, for example, dropout of nodes 1112-1120, random pooling, use of artificial data, weight decay based on L1 or L2 norm, or maximum norm constraint. Different loss functions can be combined for training the same neural network, reflecting a joint training objective. A subset of the neural network parameters can be excluded from optimization, to preserve weights pre-trained on other datasets.

[0111] The systems, apparatuses, and methods described herein can be implemented using digital circuitry or using one or more computers using well-known computer processors, memory units, storage devices, computer software, and other components. A computer includes a processor(s) for executing instructions and one or more memory / storage devices for storing instructions and data. A computer can also include, or be coupled to, one or more mass storage devices, such as disks, internal hard disks and removable disks, magnetic tapes, magnetic drum, optical disk etc.

[0112] The systems, apparatuses, and methods described herein can be implemented using computers operating in a client-server relationship. Typically, in such a system, the client computer is remote to the server computer and interacts via a network. The client-server relationship can be defined and controlled by computer programs running on the respective client and server computers.

[0113] The systems, apparatuses, and methods described herein can be implemented in a network-based cloud computing system. In such a network-based cloud computing system, a server or another processor connected to a network communicates with one or more client computers via the network. For example, a client computer can communicate with the server via a web browser application resident and operating on the client computer. The client computer can store data on the server and access the data via the network. The client computer can transmit to the server via the network a request for data or a request for an online service. The server can perform the requested service and provide the data to the client computer. The server can also transmit data adapted to cause the client computer to perform a specified function, e.g., to perform a calculation, to display specified data on a screen, etc. For example, the server can transmit a request adapted to cause the client computer to perform one or more steps or functions of the methods and workflows described herein, including Figure 2 and Figure 4 One or more steps or functions of the methods and workflows described herein (including one or more steps of Figure 2 and Figure 4 ) can be performed by a server or another processor in a network-based cloud computing system. Certain steps or functions of the methods and workflows described herein (including one or more steps of Figure 2 and Figure 4 ) can be performed by a client computer in a network-based cloud computing system. Steps or functions of the methods and workflows described herein (including one or more steps of Figure 2 and Figure 4 ) can be performed by a server and / or a client computer in a network-based cloud computing system in any combination.

[0114] The systems, apparatuses, and methods described herein can be implemented using a computer program product tangibly embodied in an information carrier, e.g., in a non-transitory machine- readable storage device, for execution by a programmable processor; and the steps of the methods and workflows described herein (including one or more steps or functions of Figure 2 and Figure 4 ) can be implemented using one or more computer programs that are executable by such a processor. A computer program is a set of instructions that can be used, directly or indirectly, in a computer to perform an activity or bring about a result. A computer program can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0115] Figure 12A high-level block diagram of an example computer 1202 that can be used to implement the systems, apparatus, and methods described herein is depicted in FIG. 12. The computer 1202 includes a processor 1204 that is operatively coupled to a data storage device 1212 and a memory 1210. The processor 1204 controls the overall operation of the computer 1202 by executing computer program instructions. The computer program instructions can be stored in the data storage device 1212 or other computer readable medium, and loaded into the memory 1210 when execution of the computer program instructions is desired. Thus, Figure 2 and Figure 4 The methods and workflow steps or functions of Figure 2 and Figure 4 may be defined by computer program instructions stored in the memory 1210 and / or data storage device 1212 and controlled by the processor 1204 executing the computer program instructions. For example, the computer program instructions can be implemented as computer executable code programmed by one skilled in the art to perform the methods and workflow steps or functions of Figure 2 and Figure 4 Thus, by executing the computer program instructions, the processor 1204 performs the methods and workflow steps or functions of and

[0116] The computer 1202 can also include one or more network interfaces 1206 for communicating with other devices via a network. The computer 1202 can also include one or more input / output devices 1208 (e.g., display, keyboard, mouse, speakers, buttons, etc.) that enable a user to interact with the computer 1202.

[0117] The data storage device 1212 and the memory 1210 each include a tangible, non-transitory computer-readable storage medium. The data storage device 1212 and the memory 1210 may each include a high-speed random access memory, such as a dynamic random access memory (DRAM), a static random access memory (SRAM), a double data rate synchronous dynamic random access memory (DDR RAM), or other random access solid-state memory devices, and may include non-volatile memory, such as one or more magnetic disk storage devices (such as an internal hard disk and a removable disk), a magneto-optical disk storage device, an optical disk storage device, a flash memory device, a semiconductor memory device, such as an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a compact disk read-only memory (CD-ROM), a digital versatile disk read-only memory (DVD-ROM) disk, or other non-volatile solid-state storage devices.

[0118] Input / output devices 1208 may include peripheral devices such as printers, scanners, display screens, etc. For example, input / output devices 1208 may include a display device such as a cathode ray tube (CRT) or liquid crystal display (LCD) monitor for displaying information to a user, a keyboard, and a pointing device such as a mouse or trackball through which a user can provide input to the computer 1202.

[0119] Image acquisition device 1214 can be connected to computer 1202 to input image data (e.g., medical images) into computer 1202. Image acquisition device 1214 and computer 1202 can be implemented as a single device. Image acquisition device 1214 and computer 1202 can also communicate wirelessly over a network. In a possible embodiment, computer 1202 can be remotely located relative to image acquisition device 1214.

[0120] Any or all of the systems and devices discussed herein, including Figure 1 The lesion segmentation network 104 and the lung segmentation network 110, Figure 2 Step 204 applies the lung segmentation network and Figure 2 Step 206 applies the abnormal pattern segmentation pattern network, Figure 3 The segmentation network 314 and Figure 4 Step 408 of training the segmentation network may be implemented using one or more computers such as computer 1202 .

[0121] Those skilled in the art will recognize that actual computer or computer system implementations may have other structures and may include other components, and for illustrative purposes, Figure 12 is a high-level representation of some of the components of such a computer.

[0122] The foregoing detailed description is to be understood as being illustrative only and not limiting of the scope of the invention as defined by the claims, and the breadth of the disclosure of the invention is not limited to the exact details set forth in the description. It is to be understood that the embodiments shown and described herein are merely for illustrative purposes and that changes and modifications can be made in the arrangement, operation and details of the methods and apparatus described herein without departing from the scope and spirit of the application. Various other combinations of features can be implemented by those skilled in the art without departing from the scope and spirit of the application.

Claims

1. A computer-implemented method for processing medical images, comprising: receiving an input medical image in a first modality; segmenting lungs from the input medical image using the trained lung segmentation network; segmenting abnormal patterns associated with the disease from the input medical image using the trained abnormal pattern segmentation network; and determining an assessment of disease based on the segmented lung and the segmented abnormality pattern, wherein the trained lung segmentation network and the trained abnormal pattern segmentation network are trained based on 1) a synthetic image in the first modality generated from the training images in the second modality and 2) a target segmentation mask for the synthetic image generated from the training segmentation masks for the training images, The target segmentation mask is generated as follows: Projecting the regions of the training image that are within their corresponding training segmentation masks onto a two-dimensional mask; and Each pixel in the two-dimensional mask is assigned a value corresponding to the sum of the intensity values ​​of voxels in the training image that are within the anatomical object of interest and that are penetrated by the projection for that pixel.

2. The computer-implemented method of claim 1 , wherein: The disease is COVID-19, and the abnormal pattern includes at least one of ground-glass opacity, consolidation, and a gravel road sign.

3. The computer-implemented method of claim 1 , further comprising training at least one of the lung segmentation network and the abnormal pattern segmentation network by: receiving a training image of the anatomical object of interest in the second modality and a training segmentation mask for the training image; generating a synthetic image in a first modality based on the training image; generating a target segmentation mask for a synthetic image based on the training segmentation mask; and At least one of the lung segmentation network and the abnormal pattern segmentation network is trained based on the synthesized image and the target segmentation mask.

4. The computationally implemented method of claim 1 , wherein generating a target segmentation mask for a synthesized image based on the training segmentation mask further comprises: Compare the value of each pixel in the two-dimensional mask to a threshold; and Each pixel in the two-dimensional mask is assigned a final value based on the comparison.

5. The computer-implemented method of claim 1 , wherein generating a target segmentation mask for the synthesized image based on the training segmentation mask further comprises: Compare the value of each pixel in the two-dimensional mask to a threshold; and Each pixel in the two-dimensional mask is assigned a final value based on the comparison.

6. The computer-implemented method of claim 1 , wherein: Assessments to determine disease based on the segmented lung and the segmented abnormality patterns include: A percentage of the affected lung area measure is calculated based on an area of ​​the lung determined from the segmented lung and an area of ​​the abnormal pattern determined from the segmented abnormal pattern.

7. The computer-implemented method of claim 1 , further comprising: Diseases are detected in input medical images based on disease assessment.

8. The computer-implemented method of claim 1 , further comprising: The evolution of the disease is predicted based on the assessment of the disease and the assessment of the disease determined for different points in time.

9. The computer-implemented method of claim 1, wherein the first modality is x-ray and the second modality is computed tomography.

10. A device for processing medical images, comprising: means for receiving an input medical image in a first modality; for segmenting a component of a lung from the input medical image using the trained lung segmentation network; means for segmenting abnormal patterns associated with a disease from the input medical image using the trained abnormal pattern segmentation network; and means for determining an assessment of disease based on the segmented lung and the segmented abnormality pattern, wherein the trained lung segmentation network and the trained abnormal pattern segmentation network are trained based on 1) a synthetic image in the first modality generated from the training images in the second modality and 2) a target segmentation mask for the synthetic image generated from the training segmentation masks for the training images, The target segmentation mask is generated as follows: Projecting the regions of the training image that are within their corresponding training segmentation masks onto a two-dimensional mask; and Each pixel in the two-dimensional mask is assigned a value corresponding to the sum of the intensity values ​​of voxels in the training image that are within the anatomical object of interest and that are penetrated by the projection for that pixel.

11. The device according to claim 10, wherein The disease is COVID-19, and the abnormal pattern includes at least one of ground-glass opacity, consolidation, and a gravel road sign.

12. The apparatus of claim 10, further comprising means for training at least one of a lung segmentation network and an abnormal pattern segmentation network by: means for receiving a training image of the anatomical object of interest in a second modality and a training segmentation mask for the training image; means for generating a synthetic image in a first modality based on the training image; means for generating a target segmentation mask for a synthesized image based on the training segmentation mask; and Means for training at least one of a lung segmentation network and an abnormal pattern segmentation network based on the synthesized image and the target segmentation mask.

13. The device according to claim 12, wherein The component for generating a target segmentation mask for a synthesized image based on the training segmentation mask further comprises: means for comparing the value of each pixel in the two-dimensional mask to a threshold value; and means for assigning a final value to each pixel in said two-dimensional mask based on said comparison.

14. The apparatus according to claim 10, further comprising: A component for detecting a disease in an input medical image based on an assessment of the disease.

15. A non-transitory computer readable medium storing computer program instructions which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 9.

Images