Systems and methods for generating canonical imaging data for medical image processing using deep learning

Abstract

The present disclosure provides a method and system for generating a normative medical image from an abnormal medical image. In one example, the method includes: receiving an abnormal medical image, wherein the abnormal medical image includes abnormal data; mapping the abnormal medical image to a normative medical image using a trained generative network of a generative adversarial network (GAN), wherein the abnormal data of the abnormal medical image is mapped to normative data in the normative medical image. In some examples, the method may further include displaying the normative medical image via a display device, and / or utilizing the normative medical image for further image analysis tasks to generate robust results from the abnormal medical image.

Description

Technical Field

[0001] Embodiments of the subject matter disclosed herein relate to generating normative imaging data for abnormal medical images using deep learning. Background Art

[0002] In the presence of lesions such as non-invasive tumors (e.g., gliomas, glioblastomas, etc.), the surrounding healthy tissue is compressed by the growing tumor. When considering treatment for these patients, as well as studying the progression of tumor growth from a baseline state of the tissue, it may be helpful to understand the brain structure before the tumor appears. Baseline non-lesion images of patients can also help mark healthy surrounding tissue so that critical functional areas can be avoided during radiation therapy or surgery. In addition, in some deep learning applications, the neural network used to perform a specific task (e.g., scan plane segmentation) has been trained with "normal" data. Applying this model to "abnormal" data (e.g., lesions) may not produce ideal results. Current methods for generating such baseline images include the use of refined biomechanical models or registration-based inversion methods. However, these methods rely on prior information about the location of the lesion and must utilize delineation of the region of interest, which will utilize additional computing resources and time. Summary of the invention

[0003] The present disclosure at least partially solves the above problems by estimating normative data of a given abnormal data set using a generative adversarial network (GAN). In one embodiment, a method for generating a normative medical image from an abnormal medical image includes: receiving an abnormal medical image, wherein the abnormal medical image includes abnormal data; mapping the abnormal medical image to a normative medical image using a trained generative network of a generative adversarial network (GAN), wherein the abnormal data of the abnormal medical image is mapped to normative data in the normative medical image; and displaying the normative medical image via a display device.

[0004] Since GANs can implicitly model the parametric form of the data distribution, GANs may be useful for learning data style transfer. In this way, the above method relaxes the constraints on the input and output data so that they do not require a one-to-one exact match, which solves the difficulty of obtaining exactly matching data in the presence of lesions (e.g., since many lesions cannot be predicted, scans can only be completed after the lesions exist, and there may be no scans available when the individual is healthy). In addition, the method described in this article can utilize fewer computational resources, rely on less prior information (e.g., lesion location), and generate normative data faster than the above-mentioned existing methods.

[0005] The above advantages and other advantages and features of the present specification will be apparent from the following detailed description when taken alone or in conjunction with the accompanying drawings. It should be understood that the above overview is provided to introduce some concepts that are further described in the detailed description in a simplified form. This is not meant to identify key or essential features of the claimed subject matter, the scope of which is uniquely defined by the claims that follow the detailed description. Furthermore, the claimed subject matter is not limited to embodiments that address any disadvantages noted above or in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The present disclosure will be better understood by reading the following description of non-limiting embodiments with reference to the accompanying drawings, in which:

[0007] Figure 1 An exemplary embodiment of an image processing system for processing medical images is shown.

[0008] Figure 2 An example block diagram of a canonical data generation system including a generation network is shown.

[0009] Figure 3 A flow chart of an example method for generating normative data from a medical image using a generative network and optionally further processing the normative data is shown.

[0010] Figure 4 A flow chart of an example method for training a generative network to generate normative data from abnormal medical images is shown.

[0011] Figure 5 An example workflow for synthesizing normative data from abnormal medical images of the knee is shown.

[0012] Figure 6 An example workflow for synthesizing normative data from abnormal medical images of the brain and performing image registration on the normative data is shown.

[0013] Figure 7 Example reference data selection using a data curation module for an abnormal scan of an input brain is shown.

[0014] Figure 8 An example canonical image generation using a generative network for a scan of an input brain including an anomaly is shown.

[0015] Fig. 9 A further example canonical image generation using the generative network for a scan of a corresponding input brain including anomalies is shown.

[0016] Fig.10 Example normative data generation using a generative network for an abnormal scan of an input knee is shown. DETAILED DESCRIPTION

[0017] In the presence of lesions such as non-invasive tumors (e.g., gliomas, glioblastomas, etc.), healthy tissue surrounding the tumor is compressed by the growing tumor. When considering treatment for patients with one or more such tumors, and in order to study the progression of tumor growth from a baseline, it is useful to understand the brain structure of the patient before the appearance of one or more tumors. The patient's baseline non-lesion medical images (e.g., magnetic resonance (MR) images, computed tomography (CT) images, positron emission tomography (PET) images, x-ray images, ultrasound images, etc.) also help mark healthy surrounding tissue to avoid critical functional areas during radiation therapy or surgery. In addition, many image analysis applications for analyzing medical images utilize models trained using non-abnormal data (e.g., non-lesion medical images). Therefore, applying such models to data affected by abnormalities (e.g., lesions, implants, image artifacts, etc.) may result in a significant reduction in the accuracy or performance of image analysis relative to applying models on normal data. Therefore, baseline images of anatomical regions of patients affected by abnormalities can help improve the accuracy and performance of such image analysis. However, once a lesion exists, such baseline images cannot be directly obtained via medical image scans of the patient. It is also not possible to scan a given patient with a lesion or other abnormality before the abnormality appears, as many scans are performed only when a problem is suspected.

[0018] The present disclosure at least partially solves the above-mentioned problems. Figure 1 The image processing system 200 shown receives a medical image generated by a medical device (such as a magnetic resonance (MR) imaging device, a computed tomography (CT) imaging device, a positron emission tomography (PET) imaging device, an x-ray imaging device, an ultrasound imaging device, etc.), and executes the method disclosed herein to generate specification data corresponding to the received medical image. The image processing system 200 may be incorporated into or communicatively coupled to the medical device, or may be configured to receive the medical image from a storage device. The image processing system 200 may execute Figure 3 The method 400 shown is used to map an abnormal medical image to a standard medical image using a generative network of a GAN; performing Figure 4 The method 500 shown is used to train a GAN using selected reference data; and / or respectively perform Figure 5 and Figure 6 Workflows 600 and 700 show examples of how to integrate the specification data generation method of the present disclosure into an image processing workflow. Figures 7 to 10 Results of example specification data generation and / or downstream processing based on the generated specification data are shown.

[0019] In this manner, non-canonical (e.g., abnormal) medical images can be used to generate canonical imaging data for modalities and protocols of interest. Since many applications are trained using canonical data, the generated canonical imaging data can be used to replace the original non-canonical medical images to apply such applications with greater accuracy, reliability, and overall performance, while still maintaining relevance to the subject of the original non-canonical medical images.

[0020] refer to Figure 1 , which shows an image processing system 200 according to an exemplary embodiment. In some embodiments, the image processing system 200 is incorporated into a medical device, such as MR, CT, x-ray, PET, ultrasound and other equipment. In some embodiments, the image processing system 200 is arranged on a device (e.g., an edge device, a server, etc.), which is communicatively coupled to the medical device via a wired and / or wireless connection. In some embodiments, the image processing system 200 is arranged on a separate device (e.g., a workstation), which can receive images from a medical device or from a storage device storing images generated by the medical device. The image processing system 200 may include an image processing system 31, a user input device 32, and a display device 33.

[0021] The image processing system 31 includes a processor 204 configured to execute machine-readable instructions stored in a non-transitory memory 206. The processor 204 can be single-core or multi-core, and the program executed thereon can be configured to perform parallel or distributed processing. In some embodiments, the processor 204 can optionally include separate components distributed in two or more devices, which can be remotely located and / or configured for coordinated processing. In some embodiments, one or more aspects of the processor 204 can be virtualized and executed by a remotely accessible networked computing device configured with a cloud computing configuration.

[0022] The non-volatile memory 206 may include: a generative adversarial network (GAN) module 208, a data curation module 210, an anomaly marking module 212, and medical image data 214. The GAN module 208 includes instructions for converting non-normative (abnormal) medical image data (e.g., abnormal medical image data including lesions, image artifacts, implants, and / or other damaged or distorted medical image data) into normative image data that is similar to medical image data captured from imaging healthy individuals. The GAN module 208 may operate using deep learning and / or other machine learning / artificial intelligence mechanisms. The data curation module 210 includes instructions for intelligently selecting reference data that is similar to a given one or more abnormal medical images and does not have abnormal / abnormal data, which will be referenced below. Figure 2 Describe in more detail.

[0023] Anomaly marking module 212 is optionally included in image processing system 200 and includes instructions for identifying that a medical image includes an anomaly that renders the image non-standard and, in response thereto, generating a label for the image. For example, anomaly marking module 212 may be configured to detect anomalies such as lesions (e.g., tumors), implants (e.g., foreign objects present in an imaged portion of an individual, and / or image artifacts that obscure at least a portion of an image. In some examples, detection performed by anomaly marking module 212 may be automatic (e.g., based on an analysis of the image anomaly). In other examples, detection performed by anomaly marking module 212 may be based on user input indicating that an image includes an anomaly. A label generated by anomaly marking module 212 may include a bit or string associated with a given image that is set to a value corresponding to the anomaly when an anomaly is identified in the given image. The value of the image. When no abnormality is identified, the bit or string can be set to a different value corresponding to a non-abnormal image. In some examples, the abnormality marking module 212 may optionally include further instructions for identifying and providing information about the abnormality (such as type, size, location, etc.). When the further instructions are included, the abnormality marking module can reduce the computing time of the image by sending only the abnormal image for further processing (e.g., avoiding attempts to generate a standard image for an image that does not include an abnormality). In other examples, the abnormality marking module may not be included in the image processing system, such as examples where it should be understood that the data is collected from a specific group of sick subjects.

[0024] The medical image data 214 includes a portion of a non-transitory memory that stores and / or is allocated to store medical image data, such as MR, CT, x-ray, PET, and / or ultrasound images. For example, the medical image data 214 may store non-canonical and canonical medical images captured from a medical device and canonical medical images generated by the GAN module 208 using the non-canonical medical images and reference data selected by the data curation module 210.

[0025] In some embodiments, non-transitory memory 206 may include components disposed on two or more devices that may be remotely located and / or configured for coordinated processing. In some embodiments, one or more aspects of non-transitory memory 206 may include a remotely accessible networked storage device configured in a cloud computing configuration.

[0026] The image processing system 200 may further include a user input device 32. The user input device 32 may include one or more of a touch screen, a keyboard, a mouse, a trackpad, a motion sensing camera, or other devices configured to enable a user to interact with and manipulate data within the image processing system 31. In one example, the user input device 32 may enable a user to select an intensity threshold or intensity range, such as by interacting with an intensity threshold control interface that may be displayed via the display device 33.

[0027] The display device 33 may include one or more display devices utilizing almost any type of technology. In some embodiments, the display device 33 may include a computer monitor and may display both unprocessed and processed medical images. The display device 33 may be combined with the processor 204, the non-transitory memory 206, and / or the user input device 32 in the same package, or may be a peripheral display device and may include a monitor, a touch screen, a projector, or other display devices known in the art that may enable a user to view medical images generated by the medical device and / or interact with various data stored in the non-transitory memory 206.

[0028] It should be understood that Figure 1 The image processing system 200 shown in FIG. 2 is for illustration and not limitation. Another suitable image processing system may include more, fewer, or different components. For example, in some embodiments, selecting reference data and / or identifying anomalies may be done manually, and therefore, the data management module 210 and / or the anomaly marking module 212 may be omitted.

[0029] Steering Figure 2, which shows a canonical data generation system 300 for mapping abnormal medical images to canonical medical images. Although MR images are used as examples for illustration here, it should be understood that the described system can be applied to other modalities, such as CT, x-ray, PET, ultrasound, etc. In one embodiment, the canonical data generation system 300 is implemented by an image processing system, such as image processing system 31. The canonical data generation system 300 is configured to generate a canonical medical image (such as a canonical MR image 310) from an input abnormal medical image (such as an abnormal MR image 302) by mapping abnormal data to canonical anatomical data using a neural network model trained via an adversarial method. GAN 318 includes a first generator / discriminator pair, which includes a generator network 304 and a generator network discriminator 306, which are configured to map abnormal medical images to canonical medical images. GAN 318 further includes a second generator / discriminator pair, which includes a reconstruction network 312 and a reconstruction network discriminator 314, which are configured to use the output canonical medical image to reconstruct the input abnormal medical image, thereby enabling the determination of a cycle consistency loss. In some embodiments, the GAN 318 may be stored as machine executable instructions in the GAN module 208 of the image processing system 31. In an exemplary embodiment, the normative data generation system 300 is configured to map an abnormal medical image to a normative medical image by performing one or more operations of one or more methods disclosed herein.

[0030] The normative data generation system 300 can be configured to receive an abnormal medical image, such as an abnormal MR image 302. The MR image 302 can be any suitable contrast, such as T1-weighted, T2-weighted scan, T1-FLAIR, T2-FLAIR, proton density weighted, etc. The abnormal MR image 302 may include one or more abnormal regions corresponding to non-anatomical regions or abnormal anatomical regions. The non-anatomical regions or abnormal regions may include lesions (such as tumors, lesions, cysts, or deformed tissue / anatomical structures, etc.), or non-pathological abnormalities (such as implants, invasions, or image artifacts). Figure 2 In the illustrated embodiment of the normative data generation system 300, the abnormal MR image 302 includes an abnormal region corresponding to a brain tumor.

[0031] In response to receiving the abnormal MR image 302, the normative data generation system 300 may transmit the abnormal MR image 302 to the data management module 210. The data management module 210 is configured to extract one or more attributes of the abnormal MR image 302 and match the extracted attributes with a plurality of non-abnormal MR images stored in the medical image data 214. In some embodiments, the extracted attributes may include: anatomical features (e.g., the type of anatomical features and / or the shape and / or gradient of anatomical features and / or images), medical image contrast, slice depth, or other attributes that do not include attributes associated with abnormal data. In some embodiments, the data management module 210 may include a neural network that is trained to recognize one or more attributes of the received image. In some embodiments, the data management module 210 may read in metadata associated with the abnormal MR image 302, and may select an MR image from the medical image data 214 based on the read in metadata. In response to extracting one or more attributes from the received medical image, the data management module 210 may compare the one or more extracted attributes with one or more attributes in a plurality of non-abnormal medical images stored in the medical image data 214. The data curation module 210 may select the non-abnormal MR images from the medical image data 214 that share one or more attributes with the abnormal MR image 302. In one embodiment, the data curation module 210 may select the non-abnormal MR images from the medical image data 214 in response to the non-abnormal MR images including one or more anatomical features / structures that are also present in the abnormal MR image 302. In some embodiments, the data curation module 210 may select the MR images from the medical image data 214 that share more than a threshold number of attributes with the abnormal MR image 302.

[0032] The reference data 308 includes MR image data of healthy / non-abnormal anatomical features that are substantially similar in appearance to the abnormal MR image 302. The non-abnormal MR images selected by the data curation module 210 from the medical image data 214 based on the attributes extracted from the abnormal MR image 302 constitute the reference data 308. In other words, the reference data 308 is a subset of MR images from the medical image data 214 that are selected based on filters defined by the data curation module 210 in response to one or more extracted attributes of the abnormal MR image 302. Although Figure 2 As shown, reference data 308 includes only a single MR image, but it should be understood that reference data 308 can include any suitable number of MR images. Reference data 308 is used in conjunction with the output of generative network 304, such as normative MR image 310, to train generative network discriminator 306. In some embodiments, reference data 308 can be manually selected.

[0033] The generative network discriminator 306 is configured to distinguish between the canonical MR image 310 and the reference data 308. The generative discriminator 306 may include a neural network classifier that is trained to classify a received medical image as "real" (i.e., a medical image without synthetic data and / or derived from the reference data 308) or "fake" (i.e., a medical image that includes synthetic data and / or includes the output of the generative network 304, such as the canonical MR image 310). In some embodiments, the generative discriminator 306 may include 70×70PatchGANs that are configured to classify 70×70 pixel patches of an input image as "real" or "fake" in a convolutional manner. Block-level discrimination enables the discriminator to classify images of arbitrary size in a computationally efficient manner without relying on a large number of features. The parameters (weights and biases) of the generative network discriminator 306 are learned / optimized according to a gradient descent algorithm, which in one embodiment includes back propagation, using the reference data 308 and the output of the generative network 304, such as the canonical MR image 310. The parameters of the generative network discriminator 306 are adjusted to maximize the likelihood that the input medical image is correctly classified as “real” if it originates from the reference data 308, or that the input medical image is correctly classified as “fake” if it originates from the generative network 304. Thus, the function of the generative network discriminator 306 is to discriminate or distinguish between medical images from the reference data 308 (which images do not have synthetic data) and canonical medical images (such as canonical MR images 310) produced by the generative network 304.

[0034] The training data for the generative network discriminator 306 includes image / classification pairs, which include medical images with corresponding classifications as "true" from the reference data 308, or medical images with corresponding classifications as "false" generated by the generative network 304, such as the canonical MR image 310. For a given medical image input into the generative network discriminator 306, a correct result includes producing a classification that matches the corresponding classification of the medical image, and an incorrect result includes producing a classification that does not match the corresponding classification of the medical image. In other words, the cost function used to train the generative network discriminator 306 is based on the classifications produced by the generative network discriminator 306, which match the correct classifications for each medical image of the training data. The output of the generative network discriminator 306 is used to train the generative network 304, and the output from the generative network 304 is used to train the generative network discriminator 306, so that the performance of the generative network 304 and the generative network discriminator 306 can be continuously improved in an iterative manner (see below for Figure 4 for a more detailed discussion).

[0035] The generating network 304 includes a convolutional neural network configured to receive medical image data at a plurality of input nodes of an input layer and map the medical image data to output data by propagating the medical image data from the input layer through one or more convolutional layers to the output layer. The propagation of data from one layer to the next is controlled by the learning parameters of the generating network 304, wherein the learning parameters include weights and biases. In some embodiments, the generating network 304 includes convolutions with a stride of 2 and convolutions with fractional strides (a stride of 1 / 2).

[0036] The parameters of the generative network 304 and the associated generative network discriminator 306 may be learned via an adversarial machine learning scheme, wherein the weights and biases of the generative network 304 are adjusted based on the output of the generative network discriminator 306, and wherein the weights and biases of the generative network discriminator 306 are adjusted based on the output of the generative network 304 and further based on the reference data 308. In some embodiments, the parameters of the generative network 304 may be learned using a gradient descent algorithm (such as back propagation) based on a cost function, wherein the cost function is determined based on the output of the generative network discriminator 306. When the number of canonical medical images generated by the generative network 304 and misclassified by the generative network discriminator 306 is maximized (for fixed parameters of the generative discriminator), the parameters of the generative network 304 may be considered to be optimized under the current conditions.

[0037] The GAN 318 is also configured to map the canonical medical image (such as the canonical MR image 310) generated by the generation network 304 to the abnormal medical image using the reconstruction network 312, thereby enabling cyclic mapping of medical images (i.e., mapping a first abnormal medical image to a canonical medical image, which is then mapped to a second abnormal medical image, and vice versa). Figure 2 In the example, the normative data generation system 300 maps the abnormal MR image 302 to the normative MR image 310 via the generation network 304, and the normative MR image 310 is mapped to the reconstructed abnormal MR image 316 via the reconstruction network 312. The cyclic mapping can determine the cycle consistency loss, which is a measure of the deviation in the medical image caused by the cyclic mapping of the medical image. Training the GAN 318 by adjusting the hyperparameters of one or more of the networks to reduce the cycle consistency loss can increase the probability that the abnormal medical image mapped to the normative medical image maintains a threshold amount of anatomical consistency, and reduce the possibility that the normative medical image generated by the generation network 304 includes deviations from anatomical features outside of one or more abnormal regions within the original abnormal medical image. In other words, adjusting the GAN 318 by adjusting one or more hyperparameters to reduce the cycle consistency loss can enable the abnormal region of the input medical image to be selectively mapped to the anatomical feature without distorting the non-abnormal region of the input medical image.

[0038] The reconstruction network 312 and the reconstruction network discriminator 314 are similar to the generation network 304 and the generation network discriminator 306, respectively, and therefore may include similar or identical overall architectures. In short, the reconstruction network 312 includes a convolutional neural network configured to receive normative medical image data at multiple input nodes of an input layer, and map the normative medical image data to abnormal medical image data by propagating the normative medical image data from the input layer through one or more convolutional layers to the output layer. The propagation of data from one layer to the next is controlled by the learning parameters of the reconstruction network 312, wherein the learning parameters include weights and biases. In some embodiments, the reconstruction network 312 includes convolutions with a step size of 2 and convolutions with fractional steps (a step size of 1 / 2).

[0039] The reconstruction network discriminator 314 and the reconstruction network 312 represent a generator / discriminator pair that can perform adversarial machine learning, as discussed above with reference to the generator network 304 and the generator network discriminator 306. In short, the reconstruction network discriminator 314 can be trained to distinguish between abnormal medical images (such as the abnormal MR image 302) and reconstructed abnormal medical images (such as the reconstructed abnormal MR image 316) via gradient descent. Subsequently, the reconstruction network 312 can be trained to maximize the number of reconstructed abnormal medical images that are misidentified as abnormal medical images by the reconstruction network discriminator 314 by producing reconstructed abnormal medical images that include features similar to those of the original abnormal medical images.

[0040] therefore, Figure 2 The canonical data generation system 300 can enable a medical image including one or more abnormalities to be mapped to a canonical medical image, wherein one or more regions of the canonical medical image corresponding to the one or more abnormalities in the input image include synthesized anatomical feature data. In addition, including the reconstruction network 312 and the reconstruction network discriminator 314 enables cycle consistency loss to be reduced, thereby reducing the probability that one or more anatomical features present in the abnormal medical image will be distorted via the mapping performed by the generation network 304.

[0041] Figure 3 A flow chart of an example method 400 for generating normative data associated with a medical image is shown. For example, the method 400 may use the method described above with reference to Figure 2 The method may be performed by components of the specification data generation system 300 described above. At 402, the method includes receiving a medical image. In some examples, the medical image may be received from an MRI (magnetic resonance imaging) system and may include MR images, MR parameter maps, and / or other image data of patient anatomical features. In other examples, the medical image may be received from an imaging system using a different imaging modality (such as a CT scanning system, an x-ray system, and / or an ultrasound system).

[0042] At 404, the method includes determining whether the medical image includes an anomaly / abnormal data. Examples of anomalies may include lesions (e.g., tumors or other pathological growths and / or abnormal anatomical features), implants (e.g., non-biological objects in the body), image artifacts (e.g., inappropriate contrast, corrupted image data, etc.), deformations (e.g., compressed or otherwise topologically transformed anatomical features, such as compressed tissue surrounding a non-invasive tumor), and / or other features that cause an obstruction or alteration of the anatomical structure captured by the medical image. In one embodiment, the medical image received at 402 may include an abnormality marker indicating that the medical image includes abnormal image data, and thus, determining at 404 whether the received medical image includes an anomaly may include determining whether the received medical image includes an abnormality marker. In another embodiment, determining whether the medical image includes an anomaly includes inputting the received image into a trained classifier, wherein the classifier includes a neural network trained to classify the input image as abnormal or non-abnormal, and also includes concluding that the received medical image includes an anomaly based on an output of the classifier indicating that the input image includes abnormal data. In some embodiments, the classifier may include a convolutional neural network trained using an input medical image of abnormal or non-abnormal data, the input medical image having a corresponding classification. If the medical image does not include an abnormality (e.g., "No" at 404), the method proceeds to a step as shown at 418, i.e., displaying the received medical image as a canonical medical image via a display device (e.g., without converting the received medical image).

[0043] In one embodiment, if the medical image is determined to include an abnormality (e.g., "yes" at 404), the method includes the step of selecting a trained generative network (e.g., Figure 2 For example, step 414 may include selecting a pre-trained generative network based on one or more attributes of the medical image that match one or more attributes of the reference data used to train the generative network discriminator and the generative network. In one example, there may be different generative networks trained for different anatomical structures (e.g., brain, knee, etc.), and based on the medical image including brain pathology, a pre-trained generative network trained using a brain reference image may be selected. In another example, if the medical image includes an image of a knee, a pre-trained generative network trained using a brain reference image may be selected.

[0044] In another embodiment, if the medical image is determined to include an abnormality (e.g., "yes" at 404), the method includes training a generative network discriminator using reference data selected based on the medical image, and training an associated generative network using the generative network discriminator. In one example, reference data may be selected for training the generative network discriminator that share one or more attributes with the medical image, where the attributes may include similar depth, anatomical features, contrast, etc.

[0045] Turn to Figure 4 , which shows an example method 500 for training a selected generative network discriminator and a generative network. At 502, the method includes selecting reference data based on a received medical image. For example, as described above with respect to Figure 1 and Figure 2 As described in the data processing module 210, reference data that shares features or attributes with the received medical image can be intelligently selected (e.g., Figure 3 The medical image received at 402 and / or with Figure 3 In other words, from the pool of medical images, medical images that share one or more or, in some embodiments, a threshold number of attributes with the received medical image can be selected as reference medical images to train the generative network discriminator. In one embodiment, the reference data can be selected by a data manager, such as data manager 210, which can apply one or more filters to a plurality of stored medical images to obtain a subset of the pool of medical images that meets one or more criteria. In one example, the one or more criteria can include reference data that shares one or more attributes with the medical image received at 502. Although described herein as occurring in response to determining that the received medical image includes an abnormality, it should be understood that the operations of selecting reference data based on the received medical image and training the generative network discriminator and the generative network using the selected reference medical image can occur at other times. For example, selecting reference data and training the generative network discriminator and the generative network can occur before receiving the medical image, thereby enabling the received medical image to be processed more quickly using pre-trained GANs. Once the reference data is selected at 502, method 500 can proceed to 504.

[0046] At 504, the method includes learning the parameters of the generative network discriminator based on the reference data and the generative network output, while keeping the parameters of the generative network fixed. In one example, back propagation of the error associated with the cost function can be used to adjust the parameters (weights and biases) of the generative network discriminator so as to maximize the classification of the canonical medical image produced by the associated generative network as "fake". By keeping the parameters of the generative network fixed, the parameters of the generative network discriminator can be stably converged.

[0047] At 506, the method includes learning the parameters of the generative network based on the generative network discriminator output while keeping the parameters of the generative network discriminator fixed. In one example, back propagation of the error associated with the cost function can be used to adjust the parameters (weights and biases) of the generative network so as to minimize the classification of canonical medical images generated by the associated generative network as "fake". By keeping the parameters of the generative network discriminator fixed, the parameters of the generative network can be stably converged. Therefore, steps 504 and 506 include alternating learning phases, wherein at 504, the parameters of the generative network discriminator are adjusted to maximize the number of correctly classified medical images in a training data set (classified by the generative discriminator), wherein the training data set includes medical images with synthetic data (generated by the generative network), and medical images without synthetic data (not generated by the generative network). Conversely, at 506, the parameters of the generative network are adjusted to minimize the number of correctly classified medical images in the training data set (classified by the generative discriminator).

[0048] At 508, the method includes determining whether the generating network and generating network discriminator parameters have converged. If the parameters have not yet converged (e.g., "No" at 508), the method includes returning to 504 to perform another iteration of parameter learning for the generating network discriminator and the generating network. Once the parameters have converged (e.g., "Yes" at 508), the method includes storing the generative adversarial network (GAN) parameters, as shown at 510. In one embodiment, parameter convergence can be inferred in response to a rate of change at which one or more or all GAN ​​parameters fall below a threshold, where the rate of change of the parameters is measured on a per iteration basis (that is, where the difference between the first parameter in the nth iteration of step 504 of method 500 and the first parameter in the (n+1)th iteration is referred to as the rate of change of the first parameter). For example, the stored parameters can define a trained GAN that can be used in a subsequent iteration of method 400 to generate canonical image data for similar images (e.g., medical images similar to the images used to select reference data at 502). It should be understood that Figure 4 The method 500 shown in FIG. 5 is for illustration and not limitation. Another suitable method may include more, fewer, or different operations.

[0049] Back to Figure 3 The method includes using a trained generative network selected at 414 (e.g., based on Figure 4 Method 500 of mapping a medical image to a canonical medical image. For example, a medical image can be input into a trained generative network (e.g., after training is completed) Figure 2), which may output a corresponding canonical medical image. In one embodiment, the output, i.e., the canonical medical image, may include one or more regions of inferred / synthesized data that replace one or more corresponding regions of abnormal data in the received medical image. After mapping the medical image to the canonical medical image, the method includes displaying the mapped canonical medical image via a display device, as shown at 418.

[0050] Once the canonical medical image has been generated (or if the received medical image has been determined to be free of abnormalities and therefore canonical), the method may include performing additional processing on the medical image, including processing adapted to be applied to canonical data. For example, at 420, the method optionally includes processing the canonical medical image with application specific steps such as additional deep learning networks or image processing and analysis steps to accomplish the task of interest. Figure 5 Examples of such processing are described in more detail. At 422, the method additionally or alternatively includes registering the canonical medical image using the labeled atlas. Figure 6 An example of performing image registration on a canonical medical image is described. It should be understood that Figure 3 The method 400 shown in FIG. 4 is for illustration and not limitation. Another suitable method may include more, fewer, or different operations.

[0051] Steering Figure 5 , an example workflow 600 for synthesizing normative data from abnormal MR images of a knee is shown. Workflow 600 enables more robust downstream image processing, particularly in image processing workflows using machine learning models that are trained primarily using non-abnormal data. In a training data set, abnormal data may occur less frequently than non-abnormal data, a problem known as data imbalance. Therefore, image processing models can perform better on normative data than on abnormal data because abnormal data account for a relatively small portion and non-abnormal data account for a relatively large portion of the data used for training these models. Workflow 600 can solve the problem of unbalanced training data by ensuring that abnormal data is converted into normal data before being input into a model trained primarily using non-abnormal data. MR images associated with one or more components or steps of workflow 600 can be displayed above the associated components / steps. Although MR images are used as examples in this article, it should be understood that workflow 600 can be applied to other modalities, such as CT, x-ray, PET, ultrasound, etc.

[0052] Workflow 600 begins at 602, where a new MR image is captured. At 602, the captured MR image can be evaluated to determine whether the captured MR image includes an abnormality. In some embodiments, abnormality identification can be performed at 602 by a neural network classifier that is trained with abnormal MR image data. In response to determining that the captured MR image includes an abnormality, operation 602 can further include marking the captured MR image with an abnormality tag. In some embodiments, the abnormality tag may include one or more details about one or more abnormalities included in the captured MR image. In the example depicted by workflow 600, the knee portion captured at operation 602 includes a knee slice affected by the presence of metal in the knee (e.g., primarily due to the use of surgical screws, fixation devices, and / or other implants). Workflow 600 can then proceed to 604.

[0053] At 604 of workflow 600, it is determined whether the captured image includes an abnormality marker. In some embodiments, the abnormality marker can be included in the metadata of the captured MR image. In response to indicating at 604 that the captured image does not include an abnormality marker, workflow 610 can proceed to operation 610, in which downstream image processing is performed. However, if it is determined at 604 that the captured image includes an abnormality marker, workflow 600 proceeds to operation 606.

[0054] At 606, workflow 600 includes applying a generated network based on the captured image. In short, Figure 5 Operation 606 in the illustrated embodiment applies a generative network trained by a process in which the data curation module 210 selects a subset of MR image data stored in the medical image data 214 based on one or more attributes of the captured MR images. Figure 5 In the example shown, the data curation module 210 may select MR images that include similar anatomical features and / or by matching image metadata included in the medical image data 214 with metadata of the captured MR image. The MR images selected by the data curation module 210 from the medical image data 214 are then used as reference data in the training of the GAN 318, as shown in FIG. Figure 4 Once the GAN 318 is trained, operation 606 includes applying the generative network of the GAN 318 to the captured image. The workflow 600 may then proceed to 608.

[0055] At 608 , the workflow 600 includes outputting a normative MR image corresponding to the captured MR image from the generative network. The normative MR image of operation 608 includes synthetic anatomical structures / features that replace abnormalities in the original captured image. The workflow 600 may then proceed to 610 .

[0056] At 610, workflow 600 includes inputting a canonical MR image into an image processing model, and in some embodiments, the image processing model is trained using non-abnormal training data. Non-abnormal training data refers to MR images without lesions, implants, image artifacts, and other non-anatomical features. In some embodiments, the image processing model of operation 610 may include a neural network-based segmentation algorithm that can distinguish between the foreground and background of the MR image in order to reduce background noise. In other embodiments, the image processing model of operation 610 may include additional processing for determining different imaging planes (e.g., for knee joint images, the planes may include meniscus planes, anterior cruciate ligament planes, etc.) to obtain continuity of landmarks / structures, thereby obtaining consistent image quality.

[0057] In this way, workflow 600 can use downstream image processing by using machine learning models trained via non-abnormal data that may otherwise be incompatible with MR images including abnormalities, thereby greatly increasing the utility of such downstream image processing tools.

[0058] Figure 6 An example workflow 700 for an image registration application applied to a canonical MR image is shown. Although MR images are used as an example herein, it should be understood that the workflow 700 can be applied to other modalities, such as CT, x-ray, PET, ultrasound, etc. The image registration described in the workflow 700 can be performed to provide registration between data affected by a lesion (also referred to as a moving image) and an atlas (e.g., a map of relevant anatomical regions or features, also referred to as a reference image). The registration process includes determining a displacement vector at each pixel in the moving image and minimizing the sum of all displacement vectors of the moving image by adjusting the position of the moving image relative to the reference image to achieve a close fit between the moving image and the reference image. The displacement vector at each pixel on the image volume is called a deformation field. Since the atlas is typically based on normal data (e.g., data of non-pathological anatomy), the presence of any lesions or abnormalities in the input image (moving image) will result in poor registration results and will affect the registration, even in image areas not affected by lesions or abnormalities. Therefore, the use of synthetic canonical data improves the initial registration of normal tissue structures with the atlas, and the initial deformation field can be used to further refine the results of the lesion image. The normative data can also be used to determine an anomaly map (eg, lesion areas in an image) by correlating or subtracting the normative image and the lesion image in a block-by-block manner. The anomaly map can be used for further constrained registration in the area affected by the lesion.

[0059] The workflow 700 thus includes an initial stage of generating normative data, followed by applying image registration using the normative data. The normative data generation described with reference to the workflow 700 may be similar to that described above with reference to Figures 2 to 5 At 702, the workflow includes imaging data in which a lesion is present. An example image resulting from imaging such a lesion is shown at 704 (e.g., including a tumor indicated by arrow 705). As shown at 706, a normative data generation model is applied to the lesion image data. For example, a generative model (such as a generative adversarial network (GAN)) is formed. Figure 2 The GAN 318 of canonical image data can be applied to the lesion image data to effectively remove or reduce the appearance of lesions in the image data. Therefore, at 708, canonical data is synthesized for the lesion image data. An example of canonical data is shown at 710, wherein the canonical data looks similar to the input lesion data in an area where the lesion is not present, and wherein based on the application of the canonical data generation model, the canonical data replaces the lesion with the estimated canonical data. Because the synthesized normal data reduces the appearance of lesions in the input image while still maintaining correspondence with the original image in non-lesion areas of the image, the synthesized normal data can be used to perform image registration, as shown at 712, without causing the difficulties described above when attempting to perform image registration with non-canonical data.

[0060] Figure 6 Also shown in is an example of a workflow for performing image registration. At 714, the workflow includes retrieving data for image registration, the data including a labeled atlas (an example of which is shown at 716) and normative data synthesized at 708 (an example of which is shown at 718). At 720, the workflow includes applying an image registration tool. At 722, the workflow includes applying a transformation and identifying lesions in an input image. Recognition of lesions in an input image (e.g., an abnormality map) can be driven by using normative data by (a) evaluating correlation between normative data and lesion data, and determining low correlation areas (e.g., correlations below a threshold, such as a threshold based on estimated bias due to imaging tolerance) as lesions, and / or (b) subtracting the entire normative volume and corresponding lesion data, and thresholding the resulting difference map to determine the abnormality map. An example identification of lesions in a transformed image is shown at 724. For example, applying a transformation can include applying a transformation model to perform a linear and / or elastic / non-rigid transformation of the synthesized normative data to align the synthesized normative data with the atlas.

[0061] At 726, the workflow includes determining whether the conversion is successful. If the synthesized normative data is determined to reach a threshold amount of alignment with the atlas, the conversion can be considered to be successfully applied. Success can be further evaluated by maximizing or minimizing a cost function, such as maximizing cross-correlation, maximizing mutual information, and / or minimizing the mean square error of the conversion. If the conversion is successful (e.g., "yes" at 726), the workflow is stopped and the registration between the input image and the atlas can be established. If the conversion is unsuccessful (e.g., "no" at 726), the workflow includes running the registration in a constrained manner for the lesion area of ​​the image and the non-lesion area of ​​the image, as shown at 728, until the transformation is successful. Constraints for running the registration in a constrained manner include controlling registration parameters, such as gradient smoothing, search radius for metric calculations, differential homeomorphism registration velocity field regularizers, registration scales, B-spline curve registration order, etc.

[0062] Figure 7 802 is shown as an example output result of the data management module 210 given a set of input data 806 corresponding to a human brain MR scan. As shown, the matching numbers represent corresponding input / output pairs of the data management module 210, however, the result 802 does not represent a one-to-one identical appearance match with the input data 806. Instead, the result 802 is generated by the data management module 210 (using the above description of Figure 1 and 2 A process described in more detail) intelligently selects from a set of normal data (not associated with the input data 806) based on overall similarity in one or more attributes, where the attributes may include shape, size, slice depth, tissue structure, anatomical features, etc., to use as a reference image for generating normalized data corresponding to the input data 806. For example, the brain scan of the input data 806 labeled "1" shows a different brain slice than the brain scan of the input data 806 labeled "3" (e.g., brain scan "1" is closer to the top of the head and therefore the patient's eyes are not visible, while the eyes are visible in brain scan "3"). Therefore, in order to provide normalized data that replaces the anomaly 808 in brain scan "1" while still maintaining an overall similar appearance to brain scan "1", the normal output result labeled "1" may be selected for use as a reference image (e.g., applied to the reference image as described above). Figure 2 in the generative adversarial network described above).

[0063] Figure 8 An exemplary canonical image data generation operation for a human brain scan is shown. The generation network 902 (in some examples, it may be Figure 2906. The example of a generative network 304 of FIG. 902 may receive as input an abnormal MR image 904 of a brain slice that includes a tumor indicated by arrow 906. The generative network 902 may output a resulting normative image 908 that includes synthesized data of the brain captured by the MR image 904, wherein the tumor is effectively removed, as shown in the region indicated by arrow 910. Thus, while the output normative image 908 may be formed from synthesized data, the synthesized data includes first image data that closely corresponds to the original input data in the region outside of the tumor, as well as first image data that closely corresponds to normal brain tissue (e.g., present in one or more reference images, as described above with reference to FIG. 906). Figures 2 to 4 said) corresponding second image data so as to replace the tumor with an approximation of normal tissue of the affected area of ​​the brain.

[0064] Fig. 9 A series of exemplary canonical image data generation operations for different human brain scans are shown. In each example, a different abnormal MR image 1002 (e.g., 1002a, 1002b, 1002c, 1002d) is provided as input to a generation network 1004, which may be Figure 2 The generative network 1004 may output, for each input abnormal image, a corresponding resulting normative image 1006 (e.g., 1006a, 1006b, 1006c, 1006d) comprising synthetic data of the relevant brain captured by the corresponding MR image 1002, with the corresponding abnormality removed. Thus, as described above with reference to Figure 8 As described, in each example, the output normative image 1006 is formed from synthesized data including data that closely corresponds to a non-abnormal region of the corresponding input MR image and data corresponding to normal brain tissue provided in place of an abnormality of the corresponding input MR image.

[0065] For example, the input abnormal MR image 1002a includes an abnormality (e.g., a tumor) indicated by arrow 1008, and a healthy tissue region indicated by arrow 1010. In the corresponding output normative image 1006a, the brain region including the abnormality is filled with synthetic healthy tissue indicated by arrow 1012. Arrows 1014 indicate regions in the output image corresponding to healthy tissue regions of the patient, which appear in the output image as closely corresponding to the relevant regions of the input image (e.g., the regions indicated by arrow 1010). In other words, the generative network normalizes the input MR image 1002 so that the regions indicated by arrow 1012 of the output image 1006a appear different from the corresponding regions of the input image (e.g., the abnormal data is replaced with synthetic normal data), while the regions indicated by arrow 1014 of the output image 1006a appear similar to the corresponding regions of the input image. In this way, the normative image 1006a normalizes the image data while maintaining the specificity of the relevant patient, so that the overall output of the generative network appears similar to the input image.

[0066] Similar abnormality normalization is shown in corresponding input / output image pairs 1002b / 1006b, 1002c / 1006c, and 1002d / 1006d. For example, abnormalities are indicated by arrows 1016, 1018, and 1020 in input images 1002b, 1002c, and 1002d, respectively. As shown by arrows 1022, 1024, and 1026 in output images 1006b, 1006c, and 1006d, respectively, the regions corresponding to the abnormalities are filled with synthetic normal brain tissue to effectively remove or reduce the appearance of the abnormalities in the output images.

[0067] Fig.10 106 is shown as example reference data 1106 selected by the data management module 210 based on a set of input data 1102 corresponding to an MR scan of a human knee. As shown, the matching numbers indicate corresponding input / output pairs of the data management module 210, however, the results 1106 do not represent a one-to-one identical appearance match with the input data 1102. Instead, the results 1106 are generated by the data management module 210 (using the reference data above). Figure 1 and Figure 21 , a process described in more detail) intelligently selects from a set of normal data (i.e., non-anomalous data) based on overall similarity in one or more attributes, where the attributes may include shape, size, slice depth, tissue structure, anatomical features, etc., for use as reference images / data for generating normative data corresponding to the input data 1102. For example, the knee scan of the input data 1102 labeled "1" shows a different knee orientation than the knee scan of the input data 1102 labeled "3" (e.g., knee scan "1" shows a horizontal cross-section of the knee, while knee scan "3" shows a vertical cross-section). Therefore, in order to provide normalized data so that the generative network can replace the anomaly 1108 in the knee scan "1" while still maintaining an overall similar appearance to the knee scan "1", the reference data labeled "1" in the output 1106 can be selected by the data curation module 210 for use as a reference image (e.g., for use in training a generative adversarial network, as described above in reference to FIG. 1 ). Figure 2 described above).

[0068] As used herein, an element or step listed in the singular and beginning with the word "one" or "a kind of" should be understood as not excluding a plurality of said elements or steps, unless such exclusion is explicitly stated. In addition, reference to "an embodiment" of the present disclosure is not intended to be interpreted as excluding the existence of additional embodiments that also include the described features. In addition, unless explicitly stated to the contrary, an embodiment that "includes," "includes," or "has" an element or multiple elements with a particular characteristic may include additional such elements that do not have the characteristic. The terms "including" and "in which" are used as the concise language equivalents of the corresponding terms "comprising" and "wherein." In addition, the terms "first," "second," and "third," etc. are used only as marks, and are not intended to impose numerical requirements or specific positional order on their objects.

[0069] This written description uses examples to disclose the invention, including the best mode, and also to enable a person skilled in the relevant art to practice the invention, including making and using any devices or systems and performing any included methods. The patentable scope of the invention is defined by the claims, and may include other examples that occur to a person skilled in the art. Such other examples are intended to fall within the scope of the claims if they have structural elements that do not differ from the literal language of the claims, or if they include equivalent structural elements with insignificant differences from the literal language of the claims.

Claims

1. A method for generating a standard medical image from an abnormal medical image, the method comprising: receiving an abnormal medical image, wherein the abnormal medical image includes abnormal data; as well as mapping the abnormal medical image to a normative medical image using a trained generative network of a generative adversarial network, wherein the abnormal data of the abnormal medical image is mapped to normative data in the normative medical image; The method of mapping the abnormal medical image to a standard medical image using a trained generative network of a generative adversarial network comprises: learning a plurality of weights for a discriminator network of the generative adversarial network using reference data comprising a subset of a plurality of non-abnormal medical images, wherein the subset of the plurality of non-abnormal medical images shares one or more properties of the abnormal medical images; using outputs from the discriminator network to learn a plurality of weights of the generator network; and In response to the plurality of weights of the discriminator network and the plurality of weights of the generator network converging: storing the plurality of weights of the discriminator network and the plurality of weights of the generator network in a storage device; and The abnormal medical image is mapped to the normative medical image using the plurality of weights of the generative network.

2. The method of claim 1, wherein the specification data comprises magnetic resonance data, computed tomography data, x-ray data, positron emission tomography data, or ultrasound data corresponding to one or more anatomical features.

3. The method of claim 1, wherein the abnormal data comprises medical image data corresponding to one or more of image artifacts, distortions, lesions, and implants. 4 . The method of claim 1 , wherein mapping the abnormal medical image to the normative medical image using the generative network comprises training the generative network based on the abnormal medical image.

5. The method according to claim 1, wherein: Also includes: receiving medical images; determining whether the medical image includes the abnormal data, wherein the medical image including the abnormal data is the abnormal medical image; as well as The standard medical image is displayed via a display device.

6. The method according to claim 5, wherein: A region of the normative medical image corresponding to the abnormal data includes synthesized normative data.

7. The method of claim 1, wherein mapping the abnormal medical image to the normative medical image using the generative network of the generative adversarial network comprises: training a discriminator of the generative adversarial network based on the abnormal medical image; training the generative network of the generative adversarial network using the trained discriminator; Inputting the abnormal medical image into an input layer of the trained generative network of the generative adversarial network; as well as The standard medical image is output from an output layer of the generative network.

8. The method of claim 1, wherein the one or more attributes include one or more of an anatomical feature, a shape of an anatomical feature, a gradient of an anatomical feature, and image contrast.

9. The method according to claim 1, further comprising: The canonical medical image is input into an image processing neural network trained with canonical data.

10. The method according to claim 1, further comprising: The canonical medical image is registered using the labeled atlas.

11. The method of claim 7, wherein determining whether the medical image includes the abnormality data comprises determining whether the medical image includes an abnormality marker.

12. A medical image processing system, the system comprising: processor; and a display device coupled to the processor; The processor is configured to: receiving a first medical image; determining whether the medical image includes an abnormality; Responding to the first medical image containing the abnormality by: mapping the first medical image to a canonical medical image using a generative network of a generative adversarial network, wherein a region of the canonical medical image corresponding to the abnormality comprises synthesized canonical data; as well as displaying the canonical medical image map via the display device; The mapping of the first medical image to the standard medical image using the generative network comprises: learning a plurality of weights for a discriminator network of the generative adversarial network using reference data comprising a subset of a plurality of second medical images, wherein the subset of the plurality of second medical images shares one or more properties of the first medical image; using outputs from the discriminator network to learn a plurality of weights of the generator network; and In response to the plurality of weights of the discriminator network and the plurality of weights of the generator network converging: storing the plurality of weights of the discriminator network and the plurality of weights of the generator network in a storage device; and The first medical image is mapped to the canonical medical image using the plurality of weights of the generative network.

13. The medical image processing system according to claim 12, wherein the storage device is used to further store the generator network and the discriminator network, and the storage device is also used to further store a plurality of second medical images.

14. The medical image processing system according to claim 12, wherein the processor is configured to determine whether the first medical image includes the abnormality by inputting the first medical image into a classifier, wherein the classifier is configured to classify the input first medical image as abnormal or non-abnormal. 15 . The medical image processing system of claim 12 , wherein the one or more attributes of the first medical image include anatomical features, contrast, and slice depth.

16. The medical image processing system of claim 12, wherein the abnormality comprises one or more of a lesion, an image artifact, and an implant.

Images