Medical image processing method, medical image processing apparatus, and program

Abstract

To improve the accuracy of image processing, including feature extraction. [Solution] The medical image processing method of the embodiment includes using a trained neural network to infer output image data having better image quality than the input image data from input image data acquired by a medical imaging system, wherein the training of the neural network includes the process of extracting features from an image dataset using a feature extractor specific to a particular domain, and the feature extractor is a discriminator included in a generative adversarial neural network that has been trained by repeatedly and alternately training a generator and a discriminator.

Description

Technical Field

[0001] The embodiments disclosed in this specification and the drawings relate to a medical image processing method, a medical image processing apparatus, and a program.

Background Art

[0002] Deep learning-based approaches for image data processing have proven to outperform conventional methods in various medical imaging modalities. However, in these approaches, blurring of the final output image often occurs. To reduce image blurring, advanced techniques such as perceptual loss and contrastive learning loss have been proposed for training neural networks. Generally speaking, these techniques use an encoder to extract features of an image. During the training of the network, the difference between the features extracted from the output of the network and the features extracted from the target image is minimized, aiming to retain the features extracted by the encoder.

[0003] Usually, the encoder is implemented by another neural network such as VGG19 or VGG16. However, these neural networks are generally trained with millions of natural images for the purpose of classification and are not specialized for the medical imaging domain. As a result, the extracted features may not be relevant to medical imaging processing tasks.

[0004] To improve image quality, there is a need for an improved approach that provides more domain-specific and task-related training for neural networks used in medical imaging systems.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

[0006] One of the problems that the embodiments disclosed herein and in the drawings aim to solve is to improve the accuracy of image processing, including feature extraction. However, the problems that the embodiments disclosed herein and in the drawings aim to solve are not limited to the above problem. Problems corresponding to the effects of each configuration shown in the embodiments described later can also be positioned as other problems. [Means for solving the problem]

[0007] The medical image processing method of the embodiment includes using a trained neural network to infer output image data having better image quality than the input image data from input image data acquired by a medical imaging system, wherein the training of the neural network includes the process of extracting features from an image dataset using a feature extractor specific to a particular domain, and the feature extractor is a discriminator included in a generative adversarial neural network that has been trained by repeatedly and alternately training a generator and a discriminator. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 shows examples of different medical imaging domains according to embodiments of the present disclosure. [Figure 2] Figure 2 shows an exemplary scenario for training a denoising neural network based on a controlled learning loss function. [Figure 3]Figure 3 shows an exemplary scenario for training a denoising neural network based on a perceptual loss function. [Figure 4] Figure 4 is a block diagram of an imaging data processing device 400 according to an embodiment of the present disclosure. [Figure 5] Figure 5 shows an exemplary scenario according to an embodiment of the present disclosure, in which a pre-trained domain-specific feature extractor is used to train a denoising neural network based on a contrast learning loss function. [Figure 6] Figure 6 shows an exemplary scenario according to an embodiment of the present disclosure, in which a pre-trained domain-specific feature extractor is used to train a denoising neural network based on a perceptual loss function. [Figure 7] Figure 7 shows an exemplary scenario in which a generative adversarial network (GAN) is trained to obtain a domain-specific feature extractor (implemented by the discriminator portion of the GAN) according to an embodiment of the present disclosure. [Figure 8] Figure 8 is a flowchart showing an exemplary procedure 800 for performing image data processing according to an embodiment of the present disclosure. [Figure 9] Figure 9 is a schematic block diagram of an exemplary X-ray diagnostic system that can incorporate the technologies disclosed herein. [Figure 10] Figure 10 is a schematic diagram of an exemplary computed tomography (CT) scanner implementation. [Figure 11] Figure 11 is a block diagram showing an exemplary computer system for performing a machine learning training and inference method according to an exemplary embodiment of the present disclosure. [Modes for carrying out the invention]

[0009] The embodiments relate to medical imaging technologies, including but not limited to 2D projected X-ray imaging, computed tomography (CT) imaging, magnetic resonance imaging (MRI), positron emission tomography (PET) imaging, and ultrasound (US) imaging.

[0010] The following disclosure provides embodiments or examples for carrying out different features of the subject matter provided. Specific examples of components and arrangements are described below for the sake of simplicity in this disclosure. Of course, these are merely examples and are not intended to be limiting.

[0011] For example, the order in which the different steps described herein are presented is for clarity. In general, these steps can be performed in any suitable order. Furthermore, although each of the different features, techniques, configurations, etc., of this specification may be discussed in different places within this disclosure, each of the concepts is intended to be able to be implemented independently of or in combination with one another. Thus, embodiments can be implemented and examined in many different ways.

[0012] Furthermore, as used herein, words such as "a, an" generally mean "one or more" unless otherwise specified.

[0013] Neural networks are used in various medical imaging modalities to improve image quality. Figure 1 shows examples of different medical imaging domains according to embodiments of the present disclosure. These domains include, but are not limited to, 2D projection X-ray imaging, computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), ultrasound (US), etc. Each medical imaging modality can have its own unique characteristics. For example, compared with CT or US images, MRI images may have different texture patterns and noise characteristics.

[0014] In Figure 1, as examples of domains, the types of medical imaging modalities are listed. However, the embodiments are not limited thereto. For example, as the domain, the type of the imaging object may be used.

[0015] For example, depending on the site such as the head or abdomen, different features appear in the collected medical images, and the site of the imaging object may be used as the domain. Also, depending on the tissue such as organs, soft tissues, and bones, different features appear in the collected medical images, and the tissue of the imaging object may be used as the domain. Further, depending on the type of disease, different features appear in the collected medical images, and the type of disease that the subject is suffering from may be used as the domain.

[0016] In order to suppress the blurring of the final image estimated by the neural network, advanced training techniques such as perceptual loss and contrast learning are used. Figure 2 shows an exemplary scenario for training a noise removal neural network based on a contrast learning loss function. In contrast learning, both a positive target (P) and a negative target (N) are provided. The network 210 learns to take in the input image (X) and generate a predicted image (Y) that resembles the positive target (P) while avoiding the negative target (N). The positive target (P) is an image with less noise and sharpness. The negative target (N) is, for example, an image with less noise but blurred.

[0017] This is achieved by first extracting features from both positive and negative targets (P, N) via the encoder 220. Next, the network 210 learns to maintain the features extracted from the positive target (P) and avoid the features extracted from the negative target (N). Alternatively, the network 210 can also learn to only maintain certain features extracted from the positive target (P). Alternatively, the network 210 can also learn to only avoid certain features extracted from the negative target (N).

[0018] For example, for the encoder 220, the predicted image (Y) output from the network 210, the positive target (P), and the negative target (N) are input, and the differences between the images are emphasized. That is, the encoder 220 encodes the predicted image, the positive example image, and the negative example image to extract features, for example, increasing or decreasing the weights for certain features. Then, a contrastive loss is calculated based on the encoded various images. For example, it is assumed that the closer the features are, the smaller the distance between the images. The contrastive loss is calculated such that the smaller the distance between the predicted image (Y) and the positive target (P), and the larger the distance between the predicted image (Y) and the negative target (N), the smaller the loss.

[0019] That is, in the example shown in FIG. 2, the network 210 is trained based on the contrastive learning loss function using an image dataset. In the training, first, from the image dataset, the first image data "predicted image (Y)", the second image data "positive target (P)", and the third image data "negative target (N)" are obtained. Also, based on the contrastive learning loss function, using the first image data as input data and the second image data and the third image data as label data, the parameters of the neural network are updated until a predetermined criterion is satisfied.

[0020] Figure 3 shows an exemplary scenario for training a denoising neural network based on a perceptual loss function. The perceptual loss approach in Figure 3 is similar to the controlled learning approach, except that negative samples are not provided. Network 310 takes in an input image (X) and learns to generate a predicted image (Y) that resembles a positive target (P). Features are extracted from the positive target (P) by encoder 320. Network 310 then learns to maintain the features extracted from the positive target (P). For example, the closer the perceptual features are, the smaller the distance between images, and the smaller the distance between the predicted image (Y) and the positive target (P), the smaller the loss.

[0021] In other words, in the example shown in Figure 3, network 310 is trained using an image dataset based on a perceptual loss function. During training, first, a first image data "predicted image (Y)" and a second image data "positive target (P)" are obtained from the image dataset. Then, based on the perceptual loss function, the parameters of the neural network are updated using the first image data as input data and the second image data as label data until predetermined criteria are met.

[0022] Figure 3 illustrates the perceptual loss approach when negative samples are not provided, but the embodiments are not limited to this. That is, negative samples (negative example images) may also be used in the perceptual loss approach. For example, the distance between images may be smaller the more similar their perceptual features are, and the loss may be reduced as the distance between the predicted image (Y) and the positive target (P) decreases, and as the distance between the predicted image (Y) and the negative target (N) increases. In this case, the loss is both the perceptual loss and the control loss. Alternatively, the perceptual loss may be calculated without using positive samples, with the distance between images being smaller the more similar their perceptual features are, and the loss may be reduced as the distance between the predicted image (Y) and the negative target (N) increases.

[0023] Encoders 220 and 320 are typically neural networks pre-trained using a large number of natural images (e.g., photographs) intended for classification, such as VGG19 and VGG16. Generally, it is difficult and time-consuming for domain experts (e.g., radiologists with expertise in a specific medical imaging domain) to handcraft features for perceptual loss and control learning loss. Furthermore, it is not practical to have domain experts available to vote on image quality at each iteration during network training. Therefore, encoders 220 and 320 are not domain-specific and are typically not experts in a particular medical imaging domain.

[0024] This disclosure provides a method and apparatus for performing deep learning-based image data processing in a medical imaging system. A neural network for improving image quality is trained based on a loss function that has, for example, a perceptual element. The neural network for improving image quality is also referred to as the first neural network. By pre-training a Generative Adversarial Network (GAN), a feature extractor can learn to identify and extract relevant features specific to the domain of the medical imaging system. This pre-trained feature extractor can function as a domain expert and can be used, for example, for the perceptual element of a loss function. The feature extractor is an example of the second neural network.

[0025] Figure 4 shows a block diagram of an imaging data processing device 400 according to an embodiment of the present disclosure. The imaging data processing device 400 includes a training dataset acquisition circuit 410, a neural network training circuit 420, and an image quality improvement circuit 430.

[0026] The training dataset acquisition circuit 410 can acquire image data to create a dataset for training a neural network aimed at improving image quality in a medical imaging system. The image data can be data collected from the domain of the medical imaging system. For example, the training dataset may include image data generated by physical simulations, image data obtained from research experiments on phantoms and volunteers, and image data obtained during clinical procedures on patients.

[0027] The neural network training circuit 420 trains the neural network using the training dataset collected by the training dataset collection circuit 410. The neural network can be, for example, a denoising network, a deblurring network, or a network trained to remove image artifacts.

[0028] Once the network parameters are determined through training, the neural network can function as an image quality improvement circuit 430. It takes image data acquired by medical imaging systems, which are generally of low quality, as input and performs inference on high-quality image data as output.

[0029] Figure 5 shows an exemplary scenario for training a denoising neural network 510 based on a contrast learning loss function according to an embodiment of the present disclosure. A feature extractor 520 is used for the perceptual element of the contrast learning loss function. This feature extractor 520 is pre-trained to function as a domain expert. That is, training the denoising neural network 510 involves the process of extracting features from an image dataset using a feature extractor 520 specific to a particular domain. Using the trained denoising neural network 510, it is possible to infer output image data having better image quality than the input image data from input image data acquired by a medical imaging system.

[0030] Similarly, in the exemplary scenario shown in Figure 6, where the denoising neural network 610 is trained based on a perceptual loss function, a feature extractor 620 is included in the perceptual element of the perceptual loss. Like the feature extractor 520, this feature extractor 620 is also pre-trained as a domain expert. That is, training the denoising neural network 610 involves the process of extracting features from an image dataset using a feature extractor 620 specific to a particular domain. Using the trained denoising neural network 610, it is possible to infer output image data having better image quality than the input image data acquired by a medical imaging system. Although not shown in Figure 5 or 6, both the contrast learning loss function and the contrast learning loss function can be paired with an additional loss function that does not use perception, such as mean squared error. Conversely, a loss function different from the contrast learning loss function and the contrast learning loss function may be used. That is, the embodiments are not limited to perceptual loss or contrast learning, but can be similarly applied to any type of image processing including feature extraction.

[0031] Feature extractors 520 and 620 can be implemented by the pre-trained discriminator portion of a GAN. That is, a discriminator included in a generative adversarial neural network, trained by repeatedly alternating between training the generator and the discriminator, can be used as a feature extractor specific to a particular domain. Figure 7 shows an exemplary scenario for training a GAN to obtain a domain-specific feature extractor according to an embodiment of the present disclosure.

[0032] As shown in Figure 7, the GAN includes a generator 710 and a discriminator 720. The generator 710 generates images with the aim of deceiving the discriminator 720 into believing that the generated images are real. The discriminator 720 evaluates the images generated by the generator 710 and attempts to identify fake images by determining whether they are machine-generated images or real images from the training dataset.

[0033] For example, the generator 710 can accept random vectors as input. Typically, these random inputs follow a multidimensional Gaussian distribution. By introducing randomness into the model, we prevent it from simply memorizing the training data. Instead, the generator 710 maps the random inputs to new images that do not exist in the training dataset. Through adversarial training, the generator 710 learns to fill the distribution gap between these generated images and real images, making the generated images very realistic. The discriminator 720 learns to classify images based on their truth value, assigning a score (e.g., "0") to the generated images and a different score (e.g., "1") to real images from the training dataset. To successfully distinguish between real and synthetic images, the discriminator 720 needs to learn domain-specific features at a high level from the training dataset, thereby developing a comprehensive understanding of what features are important in a particular domain.

[0034] During the GAN training process, the generator 710 and the discriminator 720 are repeatedly trained alternately. Initially, the generator 710 may produce images that are easily identified as fake by the discriminator 720. However, with continuous training, the generator 710 improves its ability to create convincing images, eventually producing images that can deceive the discriminator 720. At this stage, the discriminator 720's ability to distinguish between real and generated images may decrease.

[0035] Once the generator 710 becomes sufficiently skilled at deceiving the discriminator 720, further training of the generator 710 is no longer beneficial. At this stage, the parameters of the generator 710 are fixed, and training of the discriminator 720 begins. Through continuous training, the discriminator 720 improves its ability to accurately identify false images. Once the discriminator 720 achieves a high level of proficiency, the generator 710 can no longer be deceived.

[0036] By repeating these two training stages, both the generator 710 and the discriminator 720 continuously improve their capabilities. The generator 710 becomes adept at creating realistic images, and the discriminator 720 becomes adept at identifying fake images. Once the GAN is trained, the discriminator portion of the GAN can be used as feature extractors 520 and 620. For example, in training feature extractors 520 and 620, the training of the generator 710 and the discriminator 720 is repeated alternately until a predetermined criterion is met.

[0037] There are no particular restrictions on the loss function or network architecture of the generator 710 or the discriminator 720. The discriminator portion can be any model suitable for a particular imaging task. For example, a U-net can be used, and the extracted features can be output from a single layer or a combination of layers within the trained U-net.

[0038] GAN training datasets can be collected from simulations, experiments, and / or clinical procedures in a specific medical imaging domain. For example, if the domain is a type of medical imaging modality, the GAN training dataset is an image dataset corresponding to that specific modality. This allows the classifier 720 included in the GAN to be trained to extract features specific to that particular modality and to be used as a modality-specific feature extractor in the process of inferring output image data with better image quality than the input image data. As mentioned above, the domain type is not limited to types of modalities. For example, if the domain is a body part to be imaged, the GAN training dataset is an image dataset corresponding to that specific body part.

[0039] It should be noted that a feature extractor specific to a particular domain means that it has high performance in extracting features specific to that domain, and does not mean that it cannot extract features other than those specific to that domain. For example, a feature extractor specific to 2D projected X-ray imaging can extract features specific to 2D projected X-ray imaging, as well as features that are common to both 2D projected X-ray imaging and CT imaging.

[0040] In one embodiment, a GAN can be trained using the same training dataset prepared to train a desired neural network aimed at improving image quality. For example, in the examples shown in Figures 5 and 6, training a denoising network may require pairs of clean and noisy image data. Therefore, clean and / or noisy image data can be used to train a GAN.

[0041] In one embodiment, the process involves training a neural network that infers output image data having better image quality than the input image data, and training a feature extractor that extracts features specific to a particular domain. In this case, the first dataset used to train the neural network can be used as the second dataset used to train the feature extractor. Alternatively, a subset of the first dataset used to train the neural network may be used as the second dataset used to train the feature extractor. For example, if a pair of clean image data and noisy image data is used to train a denoising network, only the clean image data may be used to train the GAN.

[0042] Figure 8 shows a flowchart of an exemplary procedure 800 for performing image data processing according to an embodiment of the present disclosure. Procedure 800 includes an offline portion (steps S810 and S820) and an online portion (steps S830 and S840). In step S810, a training dataset is collected for training a neural network aimed at improving image quality. In step S820, the neural network is trained using the training dataset based on a loss function with a perceptual element. A feature extractor is pre-trained as a domain expert and used for the perceptual element of the perceptual loss.

[0043] In step S830, image data acquired by the medical imaging system is received. In step S840, a trained neural network is used to infer high-resolution image data from the received low-resolution image data. Because the perceptual element of the loss function extracts relevant features specific to a particular imaging domain, the final output image is pushed to resemble domain-specific features rather than arbitrary features extracted from a network trained to perform other tasks.

[0044] This approach can be used in any imaging domain where sufficient real-world sample data to train a GAN can be collected. While this disclosure is described and illustrated for training a denoising neural network based on a controlled learning loss function or a perceptual loss function, experts in the art will recognize that any form of loss function with a perceptual element can be used.

[0045] Figure 9 is a schematic block diagram of an exemplary X-ray diagnostic system that can incorporate the technologies disclosed herein. For example, the processing circuit of the X-ray diagnostic apparatus in Figure 9 can perform various processes such as training the various neural networks described above and image processing using the trained neural networks. Figure 10 is a schematic diagram of an exemplary CT scanner implementation. This approach can be applied to any imaging modality, including but not limited to 2D projection X-ray imaging, CT, MRI, PET, and US.

[0046] As shown in Figure 10, the X-ray imaging gantry 1050 is shown in a side view and further includes an X-ray tube 1051, an annular frame 1052, and a multi-row or two-dimensional array type X-ray detector 1053. The X-ray tube 1051 and the X-ray detector 1053 are radially mounted on the annular frame 1052, which is rotatably supported around a rotation axis RA, with the subject's obj ophthalmology (OBJ) in between. A rotation unit 1057 rotates the annular frame 1052 at a high speed, such as 0.4 seconds / revolution, while the subject's OBJ moves in or out of the illustrated page along axis RA.

[0047] Embodiments of the X-ray CT apparatus of this disclosure will be described below with reference to the accompanying drawings. Note that the X-ray CT apparatus includes various types of devices, such as rotary / rotating devices in which the X-ray tube and X-ray detector rotate together around the subject being examined, and fixed / rotating devices in which many detector elements are arranged in annular or planar manner, and only the X-ray tube rotates around the subject being examined. This disclosure can be applied to either type. Here, the currently dominant rotary / rotating type is given as an example.

[0048] The multislice X-ray CT scanner further includes a high-voltage generator 1059, which generates a tube voltage applied to the X-ray tube 1051 through a slip ring 1058 so that the X-ray tube 1051 generates X-rays. The X-rays are irradiated toward the subject's obj, and the cross-sectional area of ​​the subject's obj is represented by a circle. For example, the X-ray tube 1051 has an average X-ray energy in the first scan that is lower than the average X-ray energy in the second scan. In this way, two or more scans can be obtained corresponding to different X-ray energies. An X-ray detector 1053 is located on the opposite side of the subject's obj from the X-ray tube 1051 to detect the irradiated X-rays that have propagated through the subject's obj. The X-ray detector 1053 further includes individual detection elements or units.

[0049] The CT scanner further includes other devices for processing detection signals from the X-ray detector 1053. The data acquisition circuit or data acquisition system (DAS) 1054 converts the signals output from each channel of the X-ray detector 1053 into voltage signals, amplifies the signals, and further converts the signals into digital signals. The X-ray detector 1053 and DAS 1054 are configured to process a predetermined total number of projections per rotation (TPPR).

[0050] The data described above is transmitted via a non-contact data transmitter 1055 to a preprocessing device 1056, which is housed in a console outside the radiation gantry 1050. The preprocessing device 1056 performs specific corrections on the raw data, such as sensitivity correction. Memory 1062 stores the resulting data, also called projection data, immediately before the reconstruction process. Memory 1062, along with the reconstruction device 1064, input device 1065, and display 1066, is connected to the system controller 1060 via a data / control bus 1061. The system controller 1060 controls a current regulator 1063 that limits the current to a level sufficient to drive the CT system.

[0051] In various generations of CT scanner systems, the detectors are rotated and / or fixed relative to the patient. In one embodiment, the CT system described above may be an example of a system combining third-generation and fourth-generation geometries. In the third-generation system, the X-ray tube 1051 and X-ray detector 1053 are radially mounted on an annular frame 1052 and rotate around the subject's oblique joint (OBJ) as the annular frame 1052 rotates around the rotation axis RA. In the fourth-generation geometry system, the detectors are fixedly positioned around the patient, and the X-ray tube rotates around the patient. In an alternative embodiment, the radiation gantry 1050 has a number of detectors arranged on the annular frame 1052, supported by C-arms and stands.

[0052] Memory 1062 can store measured values ​​indicating the X-ray irradiance in the X-ray detector unit 1053. Furthermore, memory 1062 can store dedicated programs for performing CT image reconstruction, material discrimination, and motion estimation and motion compensation methods, including those described herein.

[0053] The reconstruction device 1064 can perform the methods described above as described herein. For example, the reconstruction device 1064 can perform various processes such as training the various neural networks described above and image processing using the trained neural networks. Furthermore, the reconstruction device 1064 can perform pre-reconstruction image processing such as volume rendering and image difference processing as needed.

[0054] The pre-reconstruction processing of projection data performed by the pre-processing device 1056 may include, for example, detector calibration, correction for detector nonlinearity, and polarity effects.

[0055] The post-reconstruction processing performed by the reconstruction device 1064 may include, as necessary, image filtering and smoothing, volume rendering, and image subtraction. The image reconstruction process can be performed using filtered back projection, iterative reconstruction, or probabilistic reconstruction. The reconstruction device 1064 may use memory to store, for example, projection data, reconstructed images, calibration data and parameters, and computer programs.

[0056] The reconfigurable device 1064 may include a CPU (processing circuit) that can be executed as discrete logic gates, as an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), or other Complex Programmable Logic Device (CPLD). The FPGA or CPLD implementation may be coded in VDHL, Verilog, or other hardware description language, and the code may be stored directly in the electronic memory within the FPGA or CPLD, or in a separate electronic memory. Furthermore, the memory 1062 may be non-volatile, such as ROM, EPROM, EEPROM, or flash memory. The memory 1062 may also be volatile, such as static RAM or dynamic RAM, and a processor such as a microcontroller or microprocessor may be provided to manage the electronic memory and the interaction between the FPGA or CPLD and the memory.

[0057] Alternatively, the CPU within the reconfiguration device 1064 may execute a computer program comprising a set of computer-readable instructions that perform the functions described herein, the program being stored in any of the non-transient electronic memory and / or hard disk drives, CDs, DVDs, flash drives, or other known storage media described above. Furthermore, the computer-readable instructions may be provided as utility applications, background daemons, or components of an operating system, or a combination thereof, and may be executed in cooperation with processors such as Intel Xenon or AMD Opteron processors, and Microsoft 10, UNIX®, Solaris, LINUX®, Apple, MAC-OS, and other operating systems known to those skilled in the art. Furthermore, the CPU may be implemented as multiple processors that operate concurrently and cooperatively to execute instructions.

[0058] In one embodiment, the reconstructed image can be displayed on a display 1066. The display 1066 may be an LCD display, a CRT display, a plasma display, an OLED, an LED, or any other display known in the art.

[0059] The memory 1062 may be a hard disk drive, CD-ROM drive, DVD drive, flash drive, RAM, ROM, or other electronic storage device known in the art.

[0060] Figure 11 is a block diagram illustrating an exemplary computer system for implementing a machine learning training and inference method according to an exemplary embodiment of the present disclosure. In a non-limiting embodiment, the computer system may be an AI workstation running an operating system such as Ubuntu Linux® OS, Windows®, a version of Unix OS, or Mac OS. The computer system 1100 may include one or more processing circuits 1150 having multiple cores (multicores). The processing circuit 1150 is, for example, a Central Processing Unit (CPU). The computer system 1100 may also include a graphics board 1112 having multiple GPUs, each GPU having GPU memory. The graphics board 1112 can perform many of the mathematical operations of the disclosed machine learning method. For example, the processing circuits 1150 and the graphics board 1112 can perform various processes such as training the various neural networks described above and image processing using the trained neural networks. The computer system 1100 includes main memory 1102 containing software executed by processing circuits 1150 and a graphics board 1112, typically random access memory (RAM), and a non-volatile storage device 1104 for storing data and software programs. Several interfaces may be provided for interacting with the computer system 1100, including an I / O bus interface 1110, input / peripheral devices 1118 such as a keyboard, touchpad, and mouse, a display adapter 1116 and one or more displays 1108, and a network controller 1106 enabling wired or wireless communication over a network 99. The interfaces, memory, and processor may communicate via a system bus 1126. The computer system 1100 includes a power supply 1121, which may be a redundant power supply.

[0061] In some embodiments, the computer system 1100 includes a multi-core CPU and a graphics card from NVIDIA with multiple GPU cores. In other embodiments, the computer system 1100 may include a machine learning engine.

[0062] As explained above using several diagrams as examples, the various processes described above, such as training the neural networks and image processing using the trained neural networks, can be performed on a medical image diagnostic device (modality device) that collects medical images, or on other medical image processing devices. Other medical image processing devices include any computer equipment such as workstations or server devices. Some of the various processes described above may be implemented by a group of servers (cloud) connected to the medical image diagnostic device or medical image processing device via a network.

[0063] Of course, the various processes described above may be performed in a distributed manner across multiple devices. For example, the process of inferring output image data having better image quality than the input image data using a trained neural network, the process of training the neural network, and the process of training a feature extractor specific to a particular domain may each be performed on different devices.

[0064] In light of the above teachings, numerous modifications and variations of the embodiments presented herein are possible. Therefore, it should be understood that within the scope of the claims, this application may be implemented in ways other than those specifically described herein. The present invention is not limited to the embodiments described herein. In particular, features of the illustrated embodiments can be combined in variations not shown.

[0065] Each component of the apparatus according to the above embodiment is a functional concept and does not necessarily have to be physically configured as shown in the illustration. That is, the specific form of distribution and integration of each apparatus is not limited to that shown in the illustration, and all or part of it can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions. Furthermore, each processing function performed by each apparatus can be implemented in whole or in any part by a CPU and a program that is analyzed and executed by the CPU, or by hardware using wired logic.

[0066] Furthermore, the method described in the above-mentioned embodiments can be implemented by executing a pre-prepared program on a computer such as a personal computer or workstation. This program can be distributed via a network such as the Internet. Alternatively, this program can be recorded on a computer-readable non-transient recording medium such as a hard disk, flexible disk (FD), CD-ROM, MO, or DVD, and executed by reading it from the recording medium by a computer.

[0067] With respect to the above embodiments, the following additional notes are disclosed as aspects of the invention and selective features. (Note 1) A method for performing image data processing in a medical imaging system, wherein the method is: To collect the first image dataset, Using the first image dataset collected, a first neural network is trained based on a loss function with perceptual elements. The method includes using the trained first neural network to infer output image data from input image data acquired by the medical imaging system, such that the inferred output image data has better image quality than the acquired input image data. A method for training the first neural network, wherein the first neural network is trained using a pre-trained second neural network, and the pre-trained second neural network is specific to a particular domain to which the medical imaging system corresponds. (Note 2) To obtain a second image dataset, The method may further include using the acquired second image dataset to train a feature extractor, as the pre-trained second neural network, for extracting features specific to the particular domain. (Note 3) The step of training the feature extractor is: The generator and the discriminator included in the generative adversarial neural network are repeatedly and alternately trained until a predetermined criterion is met. This may further include using the trained classifier as the pre-trained second neural network. (Note 4) The aforementioned classifier may include one or more layers of the U-net. (Note 5) The collection step may further include collecting the first image dataset through simulations, experiments, and / or clinical procedures within the particular domain. (Note 6) The acquisition step may further include acquiring the second image dataset through simulations, experiments, and / or clinical procedures within the particular domain. (Note 7) The acquisition step may further include using the collected first image dataset, or a subset of the collected first image dataset, as the acquired second image dataset. (Note 8) The loss function is a contrast learning loss function, and the step of training the first neural network is: From the first image dataset collected, the first image data, the second image data, and the third image data are obtained. The method may further include updating the parameters of the first neural network using the first image data as input data and the second and third image data as label data, based on the contrast learning loss function, until a predetermined criterion is met. (Note 9) The loss function is a perceptual loss function, and the step of training the first neural network is: From the first image dataset collected, first image data and second image data are obtained, Based on the perception loss function, the parameters of the first neural network are updated using the first image data as input data and the second image data as label data until a predetermined criterion is met. It may also include the following. (Note 10) The aforementioned specific domain may be 2D projected X-ray imaging, computed tomography (CT) imaging, magnetic resonance imaging (MRI), positron emission tomography (PET) imaging, or ultrasound (US). (Note 11) A device for performing image data processing in a medical imaging system, wherein the device is We collected the first image dataset, Using the first image dataset collected, a first neural network is trained based on a loss function with perceptual elements. Using the trained first neural network, output image data is inferred from input image data acquired by the medical imaging system, and the inferred output image data is configured to have better image quality than the acquired input image data. It includes a processing circuit configured as follows: The training of the first neural network is performed using a pre-trained second neural network, the pre-trained second neural network being specific to the particular domain to which the medical imaging system corresponds. (Note 12) The aforementioned processing circuit is We obtained a second image dataset, Using the acquired second image dataset, a feature extractor is trained as the pre-trained second neural network to extract features specific to the particular domain. It may be further configured as follows. (Note 13) The aforementioned processing circuit is The generator and the discriminator included in the generative adversarial neural network are repeatedly and alternately trained until a predetermined criterion is met. The trained classifier is used as the pre-trained second neural network. The feature extractor may be further configured to be trained by the above. (Note 14) The aforementioned classifier may include one or more layers of the U-net. (Note 15) The processing circuit may be further configured to collect the first image dataset through simulations, experiments, and / or clinical procedures within the specific domain. (Note 16) The processing circuit may be further configured to acquire the second image dataset through simulation, experiment, and / or clinical procedures within the specific domain. (Note 17) The processing circuit may be further configured to use the collected first image dataset, or a subset of the collected first image dataset, as the acquired second image dataset. (Note 18) The loss function is a symmetric learning loss function, and the processing circuit is From the first image dataset collected, the first image data, the second image data, and the third image data are obtained. Based on the aforementioned controlled learning loss function, the parameters of the first neural network are updated using the first image data as input data and the second and third image data as label data until a predetermined criterion is met. The first neural network may be further configured to be trained by the above. (Note 19) The loss function is a perceptual loss function, and the processing circuit is From the first image dataset collected, first image data and second image data are obtained, Based on the perception loss function, the parameters of the first neural network are updated using the first image data as input data and the second image data as label data until a predetermined criterion is met. The first neural network may be further configured to be trained by the above. (Note 20) A non-temporary computer-readable medium, which, when executed by one or more processors, internally stores instructions causing the one or more processors to execute a method for performing image data processing in a medical imaging system, wherein the method is To collect the first image dataset, Using the first image dataset collected, a first neural network is trained based on a loss function with perceptual elements. The method includes using the trained first neural network to infer output image data from input image data acquired by the medical imaging system, such that the inferred output image data has better image quality than the acquired input image data. The first neural network is trained using a pre-trained second neural network, the pre-trained second neural network being specific to the particular domain to which the medical imaging system corresponds. Non-temporary computer-readable media.

[0068] According to at least one embodiment described above, the accuracy of image processing, including feature extraction, can be improved.

[0069] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents. [Explanation of Symbols]

[0070] 210: Network 220: Encoder 310: Neural Networks 320: Encoder 410: Training dataset acquisition circuit 420: Neural network training circuit 430: Image quality improvement circuit 510: Neural Networks 520: Feature Extractor 610: Neural Networks 620: Feature Extractor 710: Generator 720: Discriminator 1064: Reconfiguration device 1100: Computer System 1112: Graphics card 1150: Processing circuit

Claims

1. Using a trained neural network, infer output image data with better image quality than the input image data acquired by a medical imaging system. A medical image processing method including, The training of the aforementioned neural network includes the process of extracting features from an image dataset using a feature extractor specific to a particular domain. A medical image processing method that uses a classifier included in a generative adversarial neural network, which has been trained by repeatedly and alternately training a generator and a classifier, as the feature extractor.

2. The medical image processing method according to claim 1, further comprising training the neural network using the image dataset.

3. The medical image processing method according to claim 2, further comprising collecting the aforementioned image dataset.

4. The medical image processing method according to claim 2, wherein the neural network is trained using the aforementioned image dataset based on a loss function having perceptual elements.

5. The training of the neural network includes the process of extracting features from the first image dataset using the feature extractor. The medical image processing method according to claim 1, further comprising training the feature extractor for extracting features specific to the particular domain using a second image dataset.

6. The medical image processing method according to claim 1, wherein the training of the feature extractor includes repeatedly and alternately training the generator included in the generative adversarial neural network and the discriminator included in the generative adversarial neural network until a predetermined criterion is met.

7. The medical image processing method according to claim 1, wherein the classifier includes one or more layers of U-net.

8. The medical image processing method according to claim 5, wherein the first image dataset is collected through at least one of a simulation, experiment, and clinical procedure within the specific domain.

9. The medical image processing method according to claim 5, wherein the second image dataset is collected through at least one of the simulation, experiment, and clinical procedure within the specific domain.

10. The medical image processing method according to claim 8, wherein the first image dataset or a subset of the first image dataset is used as the second image dataset.

11. The method further includes training the neural network using the aforementioned image dataset based on a controlled learning loss function, The training of the neural network is From the aforementioned image dataset, obtain a first image data set, a second image data set, and a third image data set. Based on the aforementioned controlled learning loss function, the parameters of the neural network are updated using the first image data as input data and the second image data and the third image data as label data until a predetermined criterion is met. A medical image processing method according to claim 1, including the following:

12. The aforementioned loss function is a perceptual loss function, The training of the neural network is From the aforementioned image dataset, obtain the first image data and the second image data, Based on the aforementioned perceptual loss function, the parameters of the neural network are updated using the first image data as input data and the second image data as label data until a predetermined criterion is met. A medical image processing method according to claim 4, including the following:

13. The medical image processing method according to claim 1, wherein the specific domain is 2D projected X-ray imaging, computed tomography (CT) imaging, magnetic resonance imaging (MRI), positron emission tomography (PET) imaging, or ultrasound (US).

14. A medical image processing apparatus comprising a processing circuit that uses a trained neural network to infer output image data having better image quality than the input image data from input image data acquired by a medical imaging system, The training of the aforementioned neural network includes the process of extracting features from an image dataset using a feature extractor specific to a particular domain. A medical image processing apparatus that uses a classifier included in a generative adversarial neural network, which has been trained by repeatedly and alternately training a generator and a classifier, as the feature extractor.

15. The training of the neural network includes the process of extracting features from the first image dataset using the feature extractor. The medical image processing apparatus according to claim 14, wherein the processing circuit further trains the feature extractor for extracting features specific to the particular domain using a second image dataset.

16. The medical image processing apparatus according to claim 15, wherein the training of the feature extractor includes repeatedly and alternately training the generator included in the generative adversarial neural network and the discriminator included in the generative adversarial neural network until a predetermined criterion is met.

17. The processing circuit further trains the neural network based on the contrast learning loss function using the image dataset. The training of the neural network is From the aforementioned image dataset, obtain a first image data set, a second image data set, and a third image data set. Based on the aforementioned controlled learning loss function, the parameters of the neural network are updated using the first image data as input data and the second image data and the third image data as label data until a predetermined criterion is met. A medical image processing apparatus according to claim 14, including the following:

18. The processing circuit further trains the neural network based on the perceptual loss function using the image dataset. The training of the neural network is From the aforementioned image dataset, obtain the first image data and the second image data, Based on the aforementioned perceptual loss function, the parameters of the neural network are updated using the first image data as input data and the second image data as label data until a predetermined criterion is met. A medical image processing apparatus according to claim 14, including the following:

19. A program that uses a trained neural network to cause a computer to perform a process of inferring output image data having better image quality than the input image data acquired by a medical imaging system, The training of the aforementioned neural network includes the process of extracting features from an image dataset using a feature extractor specific to a particular domain. A program that uses a discriminator included in a generative adversarial neural network, which has been trained by repeatedly and alternately training a generator and a discriminator, as the feature extractor.

Images