Medical image processing equipment

Abstract

The present invention relates to an apparatus that can provide images, e.g., MR images, with improved quality when processed using a CNN. A providing unit 111 provides an image. Another providing unit 112 provides a machine learning-based processing model. The processing model is a CNN and is configured to use effective convolution that results in a loss of voxels at the margin of the resulting processed image. A patch image generation unit 113 generates patch images, which are divided to overlap at the margin by an amount of voxels equal to the amount of voxels lost at the margin during the effective convolution. An image generation unit 114 applies the processing model to each patch image and combines the processed patch images to generate a processed image.

Description

[Technical Field]

[0001] The present invention relates to an apparatus, a method, and a computer program product for processing medical images. Further, the present invention relates to a system for processing medical images comprising said apparatus. Furthermore, the present invention relates to a training apparatus, a training method, and a training computer program product for training a processing model that can be used in an apparatus, a method, and / or a computer program product for processing medical images. [Background technology]

[0002] Nowadays, machine learning methods, in particular convolutional neural networks, are often used to process medical images, in particular to improve the quality of the medical images, e.g., to reduce image artifacts or to improve spatial resolution, or to analyze medical images, e.g., to segment parts of anatomical structures within the medical images. Summary of the Invention [Problem to be solved by the invention]

[0003] However, currently, when using machine learning algorithms, all methods that deal with the huge amount of data that medical images can present, for example in the case of 3D CT or MRI images, either result in limitations in the applicability of the method, for example due to high computational costs, or introduce certain image artifacts that can even reduce the quality of the medical image.It would therefore be advantageous if medical images could be processed using convolutional neural networks, avoiding processing artifacts and at the same time keeping applicability, and in particular computational costs, low.

[0004] The object of the present invention is to provide a method that makes it possible to process medical images using convolutional neural networks so that the quality of the processed medical images is improved. Furthermore, the object of the present invention is to keep the required computational resources reasonable so that the method can also be applied to 3D or 4D images. Furthermore, the object of the present invention is to provide an apparatus, a method and a computer program product that makes it possible to provide a processing model that can be used in the above context. [Means for solving the problem]

[0005] In a first aspect of the present invention, an apparatus for processing medical images is provided, the apparatus comprising: a) an image providing unit for providing a medical image of a region of interest of a patient; and b) a model providing unit for providing a machine learning based processing model, the processing model being a convolutional neural network, configured to process the input medical image to generate a processed medical image when the processing model is applied to the input medical image, the processing model comprising: a) a convolutional neural network configured to provide a valid convolution as part of the convolutional neural network; a model providing unit configured to use a processing model (convolution) to result in a loss of voxels in a margin of the generated processed medical image compared to an individual input medical image; c) a patch image generation unit configured to generate patch images based on the medical image, wherein the medical image is divided into patch images, and such patch images overlap at the margin by an amount of voxels equal to the amount of voxels lost at the margin during an effective convolution used by the processing model; and d) an image generation unit configured to generate the processed medical image by applying the processing model to each of the patch images to generate a processed patch image and combining the generated processed patch images to generate the processed medical image.

[0006] The convolutional neural network utilizes efficient convolution and is applied in a patch-wise manner, where the medical image is divided into patch images, and such patch images overlap at the margin by an amount of voxels equal to the amount of voxels lost at the margin during efficient convolution, and the patch-wise processed medical images can be stitched together without stitching artifacts or convolution artifacts. Furthermore, since a patch-wise application of the convolutional neural network to the medical image is provided, the computational resources for performing the processing of the medical image can be kept within reasonable limits by adapting the patches accordingly.

[0007] Generally, the device is configured to process medical images. The device can be implemented in any form of hardware and / or software provided by a general or dedicated computer system. In particular, the device can be implemented in distributed computing, e.g., as part of a computer network in which the device's functionality is provided by different processors, servers, or computer systems. Medical images can refer to any two-dimensional or three-dimensional medical images. Furthermore, medical images can also refer to 4D medical images, e.g., time-series 3D medical images. Preferably, medical images refer to any of magnetic resonance images (MRI), computed tomography (CT), single photon emission computed tomography (SPECT), positron emission tomography (PET), and ultrasound (US) images. Medical image processing can refer to any processing that utilizes a convolutional neural network. In particular, medical image processing can refer to quality enhancement and / or analysis of medical images. Quality enhancement generally refers to improving some aspect of the quality of a medical image. In particular, quality enhancement preferably refers to any of noise removal, blur removal, resolution increase, artifact correction, and edge enhancement. Analysis of medical images may refer to any analysis that extracts one or more aspects of a medical image for further processing, e.g., user verification. Preferably, analysis of medical images refers to at least one of segmentation, medical labeling, and path extraction. Generally, medical labeling refers to any type of classification of content provided by a medical image, and may refer to, for example, tissue labels, anatomical labels, body type labels, etc.

[0008] The medical image providing unit is configured to provide medical images of a region of interest of a patient. Generally, the medical image providing unit can refer to or be communicatively coupled to a storage unit, where the medical images are already stored on the storage unit, and the medical image providing unit is configured to provide the medical images stored on the storage unit for further processing, for example, to the patch image generation unit. Furthermore, the medical image providing unit can be a receiving unit that receives medical images, for example, from an input unit or directly from a medical image acquisition unit, such as a CT unit, via an interface and provides the received medical images. The medical image providing unit can also be considered as a medical image acquisition unit or part of a medical image acquisition unit directly configured to provide medical images.

[0009] The model providing unit is configured to provide a processing model based on machine learning. For example, the model providing unit may also be a storage unit in which the processing model is already stored, or may be communicatively coupled to such a storage unit. Furthermore, the model providing unit may also refer to a receiving unit for receiving the processing model, for example, from a user input unit or from an interface connected to a training device for training the individual processing model. The model providing unit may then be configured to provide the machine learning-based processing model to, for example, the image generation unit.

[0010] The processing model is a convolutional neural network and is further configured to process an input medical image when the processing model is applied to the input medical image to produce a processed medical image. Generally, a convolutional neural network is a specialized type of artificial neural network that utilizes convolutions in at least one of the neural network layers. Preferably, the processing model is a fully convolutional neural network. This has the advantage that the mathematical results are equivalent when the processing model is applied to an image on a patch-by-patch basis and when it is not applied on a patch-by-patch basis. That is, the results are equivalent when applied to the entire image and when it is applied on a patch-by-patch basis. However, in some cases, a non-fully convolutional neural network can also be advantageously utilized. In the present invention, the processing model is configured to use effective convolutions as part of the convolutional neural network that result in a loss of voxels in the margins of the resulting processed medical image compared to the respective input medical image. Generally, a valid convolution is a type of convolution operation that does not use any padding before applying the convolution to an input matrix. The size of the input matrix to which the valid convolution is applied is reduced, as opposed to using a padded convolution, in which the size of the input matrix, e.g., an image, is the same as the size of the output matrix. However, using a valid convolution has the advantage of not using values ​​that do not exist in the image itself, particularly not using padding values ​​for the padding operation. In general, in the case of a padded convolution, a neural network can be trained to ignore the padded values ​​to some extent, thereby providing sufficiently accurate results. However, this additional task increases the computational resources used by the neural network, and since these computational resources are often limited, this can lead to a decrease in accuracy in other parts of the network. Therefore, the results provided by a valid convolution are more accurate and only take into account the actual values ​​of the input medical image.

[0011] The patch generation unit is configured to generate patch images based on the medical image. In particular, the medical image is divided into patch images, and the patch images overlap at the margin by an amount of voxels equal to the amount of voxels lost at the margin during effective convolution used by the processing model. In particular, the medical image is divided into patch images, and the patch images cover all of the medical image or a predetermined portion, such as a region of interest. The number and size of the patch images, e.g., voxel width, height, and depth, can be predetermined based on, for example, the size of the patch images on which the processing model was trained. In general, the amount and size of the patch images can depend on the size and / or resolution of the medical image. The overlap at the margin of the patch images is determined based on the amount of voxels lost at the margin during effective convolution of the utilized processing model. These amounts of voxels can be easily calculated based on predetermined knowledge about the effective convolution used by the processing model and can be stored in a storage device together with the processing model, for example, so as to be available to the patch image generation unit when generating the patch images. By segmenting the medical image in this way and generating patch images based on prior knowledge of the effective convolutions performed by the processing model, it becomes possible to generate processed medical images in the following steps without quality loss due to, for example, overlap errors or pattern convolution inaccuracies.

[0012] The image generation unit is configured to generate the processed medical image by applying the processing model to each of the patch images to generate a processed patch image. The processed patch images are then combined to generate the processed medical image. Preferably, combining the generated processed patch images to generate the processed medical image includes stitching the processed patch images together, with each processed patch image being provided at a respective generated patch image position within the medical image. Furthermore, the stitching of the processed patch images is preferably performed without overlapping of the processed patch images. This is made possible by utilizing effective convolution and simultaneously determining patch images with respective overlaps.

[0013] In one embodiment, processing of a medical image refers to improving quality, and the processing model is configured to enhance the quality of the medical image with respect to at least one of noise removal, blur removal, resolution increase, artifact correction, and edge enhancement.

[0014] In one embodiment, processing refers to analyzing the medical image, and the processing model is configured to analyze the medical image with respect to at least one aspect of segmentation, medical labels, and path extraction.

[0015] In another aspect of the present invention, an apparatus for training a machine learning-based processing model usable for processing medical images is provided, the apparatus comprising: a) a training data providing unit configured to provide training data including i) a plurality of images and ii) a processed image associated with each of the plurality of images; b) a model providing unit for providing a machine learning-based processing model, the processing model being a convolutional neural network that, when applied to the input image, is trainable to process the input image and generate a processed image, the processing model being configured, as part of the convolutional neural network, to use effective convolutions to result in a loss of voxels in the margin of the generated processed image compared to the individual input images; c) a training unit for training the provided machine learning-based processing model based on the training data; and d) a model providing unit for providing the trained processing model. The training images may generally be any images, particularly if the processing model is trained for image quality enhancement. Neural networks have a general potential for being trained so that they can be applied to images containing content not used during training. However, for medical applications, it is preferred that medical images are also used for training. This allows the processing model to be adapted very specifically to the characteristics of the medical image.

[0016] In another aspect of the present invention, a method for processing medical images is provided, the method comprising the steps of: a) providing a medical image of a region of interest of a patient; b) providing a machine learning based processing model, the processing model being a convolutional neural network configured to process the input medical image when applied to the input medical image to generate a processed medical image, the processing model being configured as part of the convolutional neural network to use effective convolutions to result in a loss of voxels at a margin of the generated processed medical image compared to each of the input medical images; c) generating patch images based on the medical image, the medical image being divided into patch images, the patch images overlapping at a margin by an amount of voxels equal to the amount of voxels lost at the margin during the effective convolutions used by the processing model; and d) generating the processed medical image by applying the processing model to each of the patch images to obtain a processed patch image and combining the obtained processed patch images to generate the processed medical image. The method is computer-implemented.

[0017] In another aspect of the present invention, a computer-implemented method is provided for training a machine learning-based processing model usable for processing medical images, the method comprising the steps of: a) providing training data including a plurality of images; and ii) a processed image associated with each of the plurality of images; b) providing a machine learning-based processing model, the processing model being a convolutional neural network that, when applied to an input image, is trainable to process the input image to generate a processed image, the processing model being configured, as part of the convolutional neural network, to use significant convolutions to result in a loss of voxels in a margin of the generated processed image compared to each individual input image; c) training the provided machine learning-based processing model based on the training data; and d) providing the trained processing model. The method is computer-implemented.

[0018] In another aspect of the present invention, a system for processing medical images is presented, said system comprising: a) a medical image acquisition device for acquiring medical images; and b) an apparatus as described above.

[0019] In another aspect, a computer program product for improving the quality of medical images is presented, said computer program product comprising program code means for causing an apparatus as described above to perform the method as described above.

[0020] In another aspect, there is provided a computer program product for training a machine learning based quality enhancement model that can be used to enhance the quality of medical images, said computer program product comprising program code means for causing an apparatus as described above to perform the method as described above.

[0021] It is to be understood that the above-mentioned devices, the above-mentioned methods, the above-mentioned systems and the above-mentioned computer program products have similar and / or identical preferred embodiments, as particularly defined in the dependent claims.

[0022] It is to be understood that a preferred embodiment of the invention may also be any combination of the dependent claims or the above embodiments with the respective independent claim.

[0023] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter. [Brief explanation of the drawings]

[0024] [Figure 1] 1 shows a schematic and exemplary system for processing medical images; [Figure 2] 1 shows, schematically and exemplarily, a flow chart of a method for processing medical images; [Figure 3] 1 shows, schematically and exemplarily, a flow chart of a method for training a processing model. [Figure 4] 1 shows, in a schematic and exemplary manner, the problem solved by the present invention; [Figure 5] FIG. 1 shows a schematic and exemplary illustration of checkerboard artifacts resulting from state-of-the-art learning-based approaches in medical image processing. [Figure 6] 1 shows a schematic and exemplary comparison of the computation time of the algorithm according to the invention with that of the state of the art algorithm; DETAILED DESCRIPTION OF THE INVENTION

[0025] 1 shows a schematic and exemplary system 100 for processing medical images. The system comprises an apparatus 110 for processing medical images and a medical image acquisition unit 130 for acquiring medical images. Optionally, the system 100 further comprises a processing model training apparatus 120 for training a processing model utilized in the apparatus 110.

[0026] The image acquisition unit 130 may be a device capable of acquiring medical images. For example, the image acquisition unit may be a CT unit or an MRI unit that acquires medical images of a region of interest on a patient 132 lying on a patient support 131. The acquired medical images may then be stored in a respective storage unit or may be transmitted directly to, for example, an interface of the device 110.

[0027] The apparatus 110 can be realized as general or dedicated software and / or hardware, and in particular, the apparatus 110 can also be realized by distributed computing. The apparatus 110 includes an image providing unit 111, a model providing unit 112, a patch image generation unit 113, and an image generation unit 114. The image providing unit 111 provides medical images of a region of interest of a patient 132 acquired by the image acquisition unit 130. For example, the image providing unit 111 can be configured to access a storage device in which the medical images are already stored. However, the image providing unit 111 can also be realized as an interface for directly receiving acquired medical images from the image acquisition unit 130, for example. For example, the image providing unit 111 can provide the medical images for further processing to the patch image generation unit 113.

[0028] The model providing unit 112 is configured to provide a machine learning-based processing model for processing medical images. For example, the processing model can be stored in a respective storage unit that can be accessed by the model providing unit 112. However, the model providing unit 112 can also receive the processing model directly from a training device for training the processing model, such as the training device 120. The processing model has a convolutional neural network and is configured to process input medical images to generate processed medical images. In particular, the processing model is configured to use effective convolutions as part of the convolutional neural network to result in a loss of voxels at the margin of the generated processed medical image compared to the respective input medical image. This loss of voxels at the margin when using effective convolutions is described in more detail with reference to FIG. 4 regarding a specific application of the processing model. In general, the processing model can be any model used to process medical images that uses a convolutional neural network. Such processing models are particularly utilized for quality enhancement such as noise removal, resolution enhancement, edge enhancement, etc., or for medical image analysis such as anatomical structure segmentation, tissue labeling, etc. Thus, a processing model can refer to an image enhancement model or an image analysis model.

[0029] The patch image generation unit 113 generates a patch image based on the medical image. A patch image refers to a portion of the medical image and is generated by dividing the medical image into patch images. In particular, the division is performed so that the patch images overlap at the margin by an amount of voxels equal to the amount of voxels lost at the margin during the effective convolution used by the processing model. The voxel loss at the margin due to the effective convolution used by the processing model can be predetermined, so that the voxel loss can be stored, for example, together with the processing model and used by the patch image generation unit to generate the patch image.

[0030] The image generation unit 114 is then configured to generate a processed medical image by applying the processing model to each of the patch images. The processing model then generates a processed patch image, for example, a high-quality patch image, based on each of the patch images, and the image generation unit 114 is configured to combine the generated processed patch images to generate the processed medical image. In particular, the processed patch images are combined by stitching the processed patch images without overlapping at their respective positions in the medical image. The generated patch images are generated to include overlap, and effective convolution of the processing model results in processed patch images in which the overlap is accurately lost, so the processed patch images fit snugly and no stitching or overlapping artifacts are generated.

[0031] The device 110 further comprises an input unit 115, e.g., a keyboard, a mouse, a touch screen, etc., and an output unit 116, e.g., a display, etc., to which the processed medical images are provided by the image generation unit 114, which displays the processed medical images, e.g., on a display.

[0032] The optional training device 120 is configured to train the machine learning-based processing models used by the device 110. For this purpose, the training device includes a training data providing unit 121, a model providing unit 122, a training unit 123, and a model providing unit 124. The training data providing unit 121 is configured to provide training data for training the respective machine learning-based processing models, in particular convolutional neural networks. The training data includes a plurality of images. In general, any images can be used to train the processing models, particularly if the processing models are trained as quality improvement models, and the trained processing models can also be applied to medical images with good results. However, preferably, for medical applications, medical images are also used, for example, multiple CT or MRI images of the same region of interest within a patient or different regions of interest within a patient, or multiple CT or MRI images from different patients. Furthermore, the training data includes associated processed images, for example, referring to medical images that exhibit desired processing characteristics. For example, to train a quality improvement, particularly a noise reduction, model, processed medical images for different regions of interest and / or different types of these regions of interest can be generated based on, for example, phantoms, anatomical knowledge, anatomical atlas images, etc. Based on these processed medical images that do not exhibit noise, medical images can be artificially generated by adding noise expected by the respective imaging modality. Training data for training a data analysis model, such as a segmentation model that segments an anatomical structure, can be generated by providing multiple medical images of a specific anatomical structure from different patients and / or from different viewpoints, and the processed medical images can be generated using, for example, known segmentation algorithms or by, for example, manually segmenting the medical images by an expert.

[0033] The model providing unit 122 is then configured to provide the respective machine learning-based processing models, as described above with respect to the device 110. The training unit 123 is then configured to train the provided machine learning-based processing models based on the training data. For example, known training algorithms, such as steepest gradient algorithms, can be used to parameterize, i.e., train, the processing models. During training, patch images can be generated and used to train the provided machine learning-based processing models. However, in some cases, training can also be performed without generating patch images. For example, general denoising of medical images may be independent of the regions represented by the medical images to which the respective processing models are applied, and as a result, the respective processing models can also be trained independently of the patch images. However, in the case of segmentation models, it may be more appropriate to utilize the same patch image generation for training the segmentation models later used by the device 110 during training. The model providing unit 124 is then configured to provide the trained processing models, for example, to a respective storage unit accessible by the device 110, or to provide the trained processing models directly to the device 110.

[0034] FIG. 2 schematically and exemplarily illustrates a method for processing a medical image. Method 200 is a computer-implemented method and can be executed, for example, by device 110 of FIG. 1 , for example, by executing respective program code. The method includes step 210 of providing a medical image of a region of interest of a patient. Furthermore, method 220 includes providing a processing model based on machine learning, for example, as described in more detail above with respect to FIG. 1 . In particular, steps 210 and 220 can be performed in any order or simultaneously. The method also includes step 230 of generating patch images based on the medical image. In particular, the medical image is divided into patch images, and the patch images overlap at the margin by an amount of voxels equal to the amount of voxels lost at the margin during the effective convolution used by the processing model. In final step 240, a processed medical image is generated by applying the processing model to each of the patch images to generate a processed patch image, and combining the resulting processed patch images to generate the processed medical image.

[0035] FIG. 3 schematically and exemplarily illustrates a flowchart of a method 300 for training a machine learning-based processing model usable for processing medical images. The method 300 is a computer-implemented method that can be executed, for example, by the device 120 described with reference to FIG. 1 . In a first step, the method 300 includes providing training data including a plurality of images, e.g., medical images, and a processed image associated with each of the plurality of images, as described above with reference to FIG. 1 . The method further includes a step 320 of providing a machine learning-based processing model, also as described above. Notably, steps 310 and 320 can be performed in any order or simultaneously. The method 300 further includes a step 330 of training the provided machine learning-based processing model based on the training data using any known training algorithm for training a convolutional neural network. Finally, the method 300 further includes, in a final step 340, providing the trained processing model for use, for example, in the device 110.

[0036] A more detailed description of preferred embodiments of the present application is provided below. In general, the present invention relates to the problem of improving the quality of medical images, particularly images of medical imaging, such as MRI, CT, PET / SPECT, and US. Quality enhancement is a broad term that typically describes the process of improving image quality (IQ) by improving characteristics such as signal-to-noise ratio (SNR), contrast-to-noise ratio (CNR), reducing / removing artifacts, improving spatial resolution, or improving the perceived sharpness, quality level, and details of the resulting image. IQ enhancement is an active research topic. Many publications are presented every year in both general and medical fields. In particular, deep learning algorithms are proposed in this context. Therefore, the majority of research papers and patents in this field are related to training IQ boosting models. Patch-wise inference is often not discussed. This is related to the fact that volumetric data is mostly viewed as a set of 2D slices, which allows for the use of 2D IQ enhancement models. In 2D, it is often, but not always, possible to apply a convolutional neural network to the entire image, making patch-wise inference often unnecessary. Changing the representation format to 3D and using a true 3D convolutional neural network typically introduces an enrichment problem that needs to be solved. In particular, most medical imaging modalities, e.g., MRI, CT, SPECT / PET, and US, can be represented as both a stack of 2D images or a 3D volume. Depending on the representation format chosen, appropriate IQ enhancement models can be applied. 3D representations typically allow the use of 3D models, which provide higher-quality results, at the cost of increased computational complexity and memory footprint. There are several ways to address the increased demands on hardware resources. The most commonly used methods are described below.

[0037] Typically, modern 3D IQ enhancement models are convolutional neural networks trained using a form of gradient descent optimization algorithm. Due to their high memory footprint, these models cannot be trained on the entire volume of data. Instead, small 3D patches of each image are used. The main idea of ​​patch-wise inference is to use the same strategy when the model is applied, i.e., during model inference. A typical patch-wise algorithm can be described as follows: First, the target volume, e.g., a medical image, is divided into arbitrarily overlapping patch images of a manageable size for the convolutional neural network. Then, the model is applied to all patch images sequentially or in parallel, and the resulting enhanced target volume, e.g., an enhanced medical image, is obtained by combining the individually processed and arbitrarily overlapping patch images.

[0038] The main aspect of any convolutional neural network is the convolution kernels, which are combined to generate layers. At each convolutional layer, these kernels are sequentially multiplied with portions of the feature map from the previous layer to obtain a single new value. If no additional transformations are performed, this operation results in reduced resolution after each convolutional layer. However, in practice, it is often desirable to maintain the size of the processed image or volume. To achieve this, so-called padded convolutions are used, as shown in Figure 4. Here, feature maps from previous layers are padded to generate new feature maps of the same size after processing with the convolution kernels. While padded convolutions are convenient for implementing convolutional neural network models using 3D images, they have two significant side effects during inference. Because convolutions are padded, and "0" is often used as a padding value to represent black, especially in grayscale images, small black borders remain in the resulting patch images during inference. After the patch images are combined, the resulting image typically has a characteristic black stitching artifact known as a checkerboard artifact, as shown in the example in Figure 5. To avoid checkerboard artifacts, inference with overlapping patches is typically used, as shown in Figure 4. However, overlapping does not eliminate stitching artifacts; it merely makes them less noticeable. Furthermore, it significantly increases the computational complexity of the resulting inference by up to 1.5 times compared to non-overlapping options. Therefore, it would be beneficial to discover suitable algorithms that can improve the image quality of medical images processed with convolutional neural networks, especially those that can avoid the respective patching artifacts.

[0039] In this context, the present invention proposes to use unpadded, i.e., effective, convolutions, as described above, in combination with a specific inference strategy to mathematically make the appearance of stitching artifacts impossible and reduce computational complexity. Figure 6 shows a schematic and illustrative comparison of computation time on a V100 GPU for the same 3D convolutional neural network model topology using padded and effective convolutions. The largest topology with 14 layers and 64 filters shows a performance improvement of approximately 1.5x.

[0040] To achieve this, the present invention combines a convolutional neural network architecture, such as 2D or 3D, with a specific inference strategy to efficiently obtain artifact-free resulting images. In particular, the present invention is based on the realization that the loss of resolution caused by effective convolution, which is usually considered a drawback, can be an advantage if used correctly.

[0041] In a preferred embodiment, the apparatus for processing medical images according to the present invention uses a 2D or 3D convolutional neural network model architecture designed to solve IQ enhancement problems, such as noise removal, super-resolution, and artifact correction, as a processing model, where the model uses effective convolution as its primary form of operation. Furthermore, a patch-wise inference strategy is used during inference to obtain a resulting IQ-enhanced version of the volume. In particular, padded convolutions, which are used by default in most deep learning frameworks, are replaced with effective, i.e., padded-free convolutions. Next, a convolutional neural network is trained to solve the respective processing model, e.g., the IQ enhancement problem, in a patch-wise manner. The trained convolutional neural network model is then applied to the data to be enhanced, i.e., the medical image, using the following patch-wise inference strategy: In this strategy, inference is performed on overlapping patches, and the size of the overlapping patches exactly matches the number of pixels lost by effective convolution during inference, given the network architecture. Non-overlapping patches can then be neatly stitched together without the need for averaging or other processing. This is the computationally fastest way to perform inference with limited GPU memory, and is mathematically guaranteed to produce the same results as applying a convolutional neural network model to the entire volume (which is typically computationally infeasible due to memory limitations).

[0042] Other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims.

[0043] In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality.

[0044] A single unit or device may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0045] The procedures performed by one or several units or devices, such as providing a medical image, providing a processing model, generating patch images, generating a processed medical image, etc., can be performed by any other number of units or devices, and these procedures can be realized as program code means of a computer program and / or as dedicated hardware.

[0046] The computer program may be stored / distributed on a suitable medium, such as an optical storage medium or a solid-state medium, supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems.

[0047] Any reference signs in the claims should not be construed as limiting the scope.

[0048] The present invention relates to an apparatus that provides images, e.g., MR images, with improved quality when processed using a CNN. A providing unit provides an image. Another providing unit provides a processing model based on machine learning. The processing model is a CNN and is configured to use effective convolution that results in a loss of voxels at the margin of the resulting processed image. A patch image generation unit generates patch images, which are divided such that the patch images overlap at the margin by an amount of voxels equal to the amount of voxels lost at the margin during the effective convolution. The image generation unit generates a processed image by combining the processed patch images obtained by applying the processing model to each patch image, respectively.

Claims

1. An apparatus for processing medical images, comprising: an image providing unit for providing a medical image of a region of interest of a patient; a model providing unit for providing a machine learning based processing model, the processing model being a convolutional neural network configured to process an input medical image to generate a processed medical image when the processing model is applied to the input medical image, the processing model being configured as part of the convolutional neural network to use effective convolutions to result in a loss of voxels in a margin of the generated processed medical image compared to the individual input medical image; a patch image generation unit for generating patch images based on the medical image, wherein the medical image is divided into the patch images, and the patch images overlap at the margin by an amount of voxels equal to an amount of voxels lost at the margin during an effective convolution used by the processing model; an image generation unit that generates a processed medical image by applying the processing model to each of the patch images to generate a processed patch image and combining the generated processed patch images to generate a processed medical image; A device having:

2. 2. The apparatus of claim 1, wherein combining the processed patch images to generate the processed medical image comprises stitching the processed patch images together, each processed patch image being provided at a position of the respective generated patch image within the medical image.

3. The apparatus of claim 2 , wherein the stitching of the processed patch images is performed without overlapping of the processed patch images.

4. The apparatus according to claim 1 , wherein said processing of said medical images refers to enhancement and / or analysis of said medical images.

5. 5. The apparatus of claim 1, wherein the processing of the medical image refers to quality enhancement, and the processing model is configured to enhance the quality of the medical image with respect to at least one of noise removal, blur removal, resolution increase, artifact correction, and edge enhancement.

6. 6. The apparatus of claim 1, wherein the processing refers to analyzing the medical image, and the processing model is configured to analyze the medical image with respect to at least one of segmentation, medical labeling, and path extraction.

7. The apparatus according to claim 1 , wherein the medical image is one of an MRI image, a CT image, a SPECT image, a PET image, and an US image.

8. 1. An apparatus for training a machine learning based processing model that can be used to process medical images, comprising: a training data providing unit for providing training data comprising: a) a plurality of images; and b) a processed image associated with each of the plurality of images; a model providing unit for providing a machine learning based processing model, the processing model being a convolutional neural network that is trainable to process an input image to generate a processed image when the processing model is applied to the input image, the processing model being configured as part of the convolutional neural network to use effective convolutions to result in a loss of voxels in a margin of the generated processed image compared to the respective input image; a training unit for training the provided machine learning based processing model based on the training data; a model providing unit for providing the trained processing model; A device having

9. 1. A method for processing medical images, comprising: providing a medical image of a region of interest of a patient; providing a machine learning based processing model, the processing model being a convolutional neural network configured to process an input medical image to generate a processed medical image when the processing model is applied to the input medical image, the processing model being configured as part of the convolutional neural network to use significant convolutions to result in a loss of voxels in a margin of the generated processed medical image compared to the respective input medical image; generating patch images based on the medical image, wherein the medical image is divided into the patch images, and the patch images overlap at the margin by an amount of voxels equal to an amount of voxels lost at the margin during the effective convolution used by the processing model; applying the processing model to each of the patch images to generate a processed patch image, and combining the processed patch images to generate a processed medical image; A method having the following.

10. 1. A method for training a machine learning based processing model that can be used to process medical images, comprising: providing training data having a) a plurality of images; and b) a processed image associated with each of the plurality of images; providing a machine learning based processing model, the processing model being a convolutional neural network that is trainable to process an input image to generate a processed image when the processing model is applied to the input image, the processing model being configured as part of the convolutional neural network to use effective convolutions that result in a loss of voxels in the margins of the resulting processed image compared to each input image; training the provided machine learning based processing model based on the training data; providing the trained processing model; A method having the following.

11. 1. A system for processing medical images, comprising: a medical image acquisition device for acquiring medical images; An apparatus according to any one of claims 1 to 7; A system having:

12. A computer program for improving the quality of medical images, comprising program code means for causing an apparatus according to any one of claims 1 to 7 to carry out the method according to claim 9.

13. 11. A computer program for training a machine learning based quality enhancement model that can be used to improve the quality of medical images, the computer program having program code means for causing an apparatus according to claim 8 to perform the method according to claim 10.

Images