Medical image analysis system

Abstract

The invention relates to an apparatus 10 for training a medical image analysis algorithm, comprising an input unit 20 and a processing unit 30. The input unit is configured to receive a medical image of a body part of a patient. The input unit is configured to provide the medical image to the processing unit. The processing unit is configured to generate a modified medical image. The generation of the modified medical image comprises modifying two or more spatial frequency bands associated with the medical image. The processing unit is configured to utilize the modified medical image to train a medical image analysis machine learning algorithm.

Description

[Technical field]

[0001] The present invention relates to a medical image analysis algorithm training device, a medical image generating device, a medical image analysis system, a medical image analysis algorithm training method, a medical image generating method, a medical image analysis method, a computer program, and a computer-readable medium. [Background technology]

[0002] Philipsen, RH, Maduskar, P., Hogeweg, L., Melendez, J., Sanchez, CI, & van Ginneken, B. (2015). Localized energy-based normalization of medical images: application to chest radiography. IEEE transactions on medical imaging, 34(9), 1965-1975 discloses a general method for normalizing images using energy band decomposition.

[0003] China Patent Publication No. 111428753 discloses a method for performing Gaussian pyramid processing and Laplace pyramid processing to obtain images for training a neural network model.

[0004] Xu, Z., Liu, D., Yang, J., Raffel, C., & Niethammer, M. (2020). Robust and generalizable visual representation learning via random convolutions. arXiv preprint arXiv:2007.13003 discloses a method for data augmentation using random convolutions. Summary of the Invention [Problem to be solved by the invention]

[0005] Supervised training of neural networks for classification, segmentation, or regression of medical images, such as X-ray and computed tomography (CT) images, requires a huge amount of training data to prevent overfitting. This is not only due to the anatomical and pathological variability in images of a particular examination type, but also due to the large number of prevalent image styles. Image styles are influenced by the imaging system used, parameter selection (e.g., voltage and power), as well as user- or manufacturer-dependent pre- and post-processing preference settings. In particular, when neural networks are applied to medical images originating from different manufacturers, style invariance becomes a prerequisite for good generalizability. Style invariance is not built into state-of-the-art neural network architectures and therefore needs to be learned during training. However, collecting the required appropriate training medical images is time-consuming or even impossible. This problem needs to be solved.

[0006] Improved medical image analysis modalities would be advantageous. [Means for solving the problem]

[0007] The invention is defined by the independent claims, and further embodiments are defined by the dependent claims. It should be noted that the following described aspects and embodiments of the invention apply to an apparatus for training a medical image analysis algorithm, a medical image generating apparatus, a medical image analysis system, a method for training a medical image analysis algorithm, a medical image generating method, a medical image analysis method, a computer program and a computer readable medium.

[0008] In a first aspect, there is provided an apparatus for training a medical image analysis algorithm, the apparatus comprising an input unit and a processing unit.

[0009] The input unit is configured to receive a medical image of a body part of a patient. The input unit is configured to provide said medical image to a processing unit. The processing unit is configured to generate a modified medical image, such as by modifying two or more spatial frequency bands associated with said medical image. The processing unit is configured to utilize the modified medical image to train a medical image analysis machine learning algorithm.

[0010] In this way, medical image analysis algorithms can be trained on a small set of real image data, where multi-scale data augmentation is used to generate what may be any number of realistic synthetic images for the real input image. Not only are fewer real images required, but the synthetic images also represent images that would have been acquired with different image acquisition settings, and even different image acquisition units. Thus, by taking into account the different image acquisition settings, the analysis algorithms trained on all these images are more robust and accurate.

[0011] In one example, the processing unit is configured to utilize the medical image to generate a scaled image set having a plurality of scaled images, each scaled image compromising the representation of spatial frequencies in the medical image, the representation of spatial frequencies in each of the plurality of scaled images being different. The processing unit is also configured to generate a modified scaled image set from the scaled image set, such generating including modifying two or more scaled images of the plurality of scaled images. The processing unit is configured to generate a modified medical image using the modified scaled image set.

[0012] In this way, a medical image of a patient's body part is used to generate multiple synthetic medical images that are similar to the original medical image but different from each other, and these multiple synthetic medical images can be used to train medical image analysis machine learning algorithms.

[0013] In one example, modifying two or more scaled images of the plurality of scaled images includes modifying representations of spatial frequencies contained in each of the two or more scaled images.

[0014] In one example, modifying two or more scaled images of the plurality of scaled images includes modifying representations of spatial frequencies contained in each of the two or more scaled images with a random modification factor.

[0015] In one example, generating the set of rectified scaled images includes rectifying a plurality of scaled images.

[0016] In one example, modifying the plurality of scaled images includes modifying the representation of spatial frequencies contained in each of the plurality of scaled images using a modification factor that increases with respect to spatial frequencies from one scaled image to the next scaled image.

[0017] In one example, modifying the plurality of scaled images includes modifying the representation of spatial frequencies contained in each of the plurality of scaled images using a modification factor that decreases with respect to spatial frequencies from one scaled image to the next scaled image.

[0018] In a second aspect, there is provided a medical imaging device comprising an input unit and a processing unit.

[0019] The input unit is configured to receive a medical image of a body part of a patient. The input unit is configured to provide the medical image to a processing unit. The processing unit is configured to generate a modified medical image, the generating including modifying two or more spatial frequency bands associated with the image.

[0020] In this way, a limited number of real images can be used to generate realistic synthetic images that can be used to train image analysis algorithms.

[0021] In a third aspect, there is provided a medical image analysis system having an input unit and a processing unit.

[0022] The input unit is configured to receive a review medical image of the patient's body part. The input unit is configured to provide the review medical image to a processing unit. The processing unit is configured to analyze the patient's body part. The analysis comprises an interrogation of the review medical image with a trained machine learning algorithm. The machine learning algorithm is trained, said training comprising utilizing at least one modified medical image generated by the generating device according to the second aspect.

[0023] In this way, the analysis of medical images is improved since the machine learning algorithms that analyze the images are trained on a comprehensive dataset of images that can reproduce different image acquisition settings and acquisitions by different acquisition units of the same type (hence different X-ray units or different CT units).

[0024] In this way, images are generated that effectively mimic different image styles depending on, for example, operator settings, mimic acquisition by units from different vendors, mimic different post-processing configurations, etc. Machine learning algorithms trained on this rich and diverse image dataset are then more robust in analyzing images acquired by acquisition units that are configured differently by different operators or from different vendors.

[0025] In a fourth aspect, there is provided a method for training a medical image analysis algorithm comprising the steps of receiving a medical image of a patient's body part by an input unit, providing the medical image by the input unit to a processing unit, generating by the processing unit a modified medical image, the generating comprising modifying two or more spatial frequency bands associated with the medical image, and training by the processing unit a medical image analysis machine learning algorithm comprising utilizing the modified medical image.

[0026] In one example, the method includes generating, by the processing unit, a scaled image set having a plurality of scaled images, the generating including utilizing the medical image, each scaled image compromising a representation of spatial frequencies in the medical image, such that the representation of spatial frequencies in each image of the plurality of scaled images is different; generating, by the processing unit, a modified scaled image set from the scaled image set, the generating including modifying two or more scaled images of the plurality of scaled images; and generating, by the processing unit, the modified scaled image set using the modified medical image.

[0027] In a fifth aspect, there is provided a method for generating a medical image comprising the steps of receiving, by an input unit, a medical image of a patient's body part, and generating, by the processing unit, a modified medical image, said generating including modifying two or more spatial frequency bands associated with the medical image.

[0028] In one example, the method includes the steps of: generating, by the processing unit, a scaled image set having a plurality of scaled images, the generating including utilizing the medical image, each scaled image compromising a representation of spatial frequencies in the medical image, such that the representation of spatial frequencies in each image of the plurality of scaled images is different; generating, by the processing unit, a modified scaled image set from the scaled image set, the generating including modifying two or more scaled images of the plurality of scaled images; and generating, by the processing unit, the modified scaled image set using the modified medical image.

[0029] In a sixth aspect, there is provided a method for medical image analysis comprising the steps of receiving, by an input unit, an examination medical image of a patient's body part; providing, by the input unit, the examination medical image to a processing unit; and analysing, by the processing unit, the patient's body part, said analysis comprising interrogating the examination medical image with a trained machine learning algorithm, the machine learning algorithm having been trained utilizing at least one modified medical image generated by the method according to the fifth aspect.

[0030] According to another aspect, there is provided a computer program element for controlling one or more of the aforementioned devices, generating devices, systems, the computer program element being adapted to perform the associated aforementioned method when the computer program element is executed by a processor.

[0031] According to another aspect, there is provided a computer readable medium having stored thereon the aforementioned computer program elements.

[0032] The computer program element may for example be a software program, but also an FPGA, PLD or any other suitable digital means.

[0033] Advantageously, advantages provided by any of the above aspects apply equally to all of the other aspects, and vice versa.

[0034] The above aspects and examples will be apparent from and elucidated with reference to the embodiments described hereinafter. [Brief description of the drawings]

[0035] [Figure 1] FIG. 1 is a schematic diagram illustrating an example of an apparatus for training a medical image analysis algorithm. [Diagram 2] FIG. 1 is a schematic diagram showing an example of a medical image generating apparatus. [Diagram 3] FIG. 1 is a schematic diagram showing an example of a medical image analysis system. [Figure 4] FIG. 1 illustrates an example method for training a medical image analysis algorithm. [Diagram 5] FIG. 1 is a diagram showing an example of a medical image generating method. [Figure 6] FIG. 1 is a diagram showing an example of a medical image analysis method. [Figure 7] An input image (left subplot) and a sequence of Laplace pyramid bands scaled to the dimensions of the original image (other subplots). [Figure 8]Diagram showing the input image (left subplot) and its eight enhanced versions (other subplots) obtained by randomly monotonically increasing pre-coefficients of the Laplace band in the pyramidal reconstruction. [Figure 9] Figure 1 shows the input image (left subplot) and its eight augmented versions (other subplots) obtained by random monotonically decreasing pre-coefficients of the Laplace band in the pyramidal reconstruction. [Figure 10] Diagram showing the input image (left subplot) and its eight enhanced versions (other subplots) obtained by random pre-coefficients of the Laplace band in the pyramidal reconstruction. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0036] Exemplary embodiments will now be described with reference to the accompanying drawings.

[0037] FIG. 1 shows an example of an apparatus 10 for training a medical image analysis algorithm. The apparatus comprises an input unit 20 and a processing unit 30. The input unit is configured to receive a medical image of a body part of a patient. The input unit is configured to provide said medical image to the processing unit. The processing unit is configured to generate a modified medical image, such generating comprising modifying two or more spatial frequency bands associated with the medical image. The processing unit is configured to train a medical image analysis machine learning algorithm, such training comprising utilizing the modified medical image.

[0038] In one example, the medical image is an x-ray image.

[0039] In one example, the medical image is a CT image.

[0040] In one example, the machine learning algorithm is a neural network.

[0041] In one example, each scaled image of two or more scaled images is modified differently.

[0042] In one example, the processing unit is configured to train a medical image analysis machine learning algorithm, where such training includes utilization of medical images.

[0043] According to one example, the processing unit is configured to generate a scaled image set having a plurality of scaled images. The generation of the scaled image set includes utilizing a medical image. Each scaled image compromises the representation of spatial frequencies in the medical image, and the representation of spatial frequencies in each image of the plurality of scaled images is different. The processing unit is configured to generate a modified scaled image set from the scaled image set. The generation of the modified scaled image set includes modifying two or more scaled images of the plurality of scaled images. The processing unit is configured to generate a modified medical image, and the generation of the modified medical image includes utilizing the modified scaled image set.

[0044] In one example, each scaled image of two or more scaled images is modified differently.

[0045] In one example, each scaled image corresponds to a band of frequencies.

[0046] In one example, the set of scaled images is a Laplacian pyramid with several elements, where each element is a scaled image and corresponds to a band of frequencies.

[0047] According to one example, modifying two or more of the plurality of scaled images includes modifying a representation of spatial frequencies contained in each of the two or more scaled images.

[0048] According to one example, modifying two or more of the plurality of scaled images includes modifying representations of spatial frequencies contained in each of the two or more scaled images using a random modification factor.

[0049] In one example, modifying spatial frequencies contained within the scaled image includes modifying intensity distributions associated with frequency bands of the scaled image.

[0050] In one example, the modification comprises modifying intensity distributions associated with frequency bands of elements of a Laplacian pyramid in the modified image.

[0051] In one example, modifying the representation of spatial frequencies contained within the scaled image includes modifying the intensity distribution of the scaled image.

[0052] In one example, the modification involves modifying the intensity distribution of the elements of the Laplacian pyramid itself (eg, by histogram matching against some randomly selected criteria).

[0053] In one example, the random modification factors include variable weighting factors that are applied to each different scaled image.

[0054] In one example, the weights of the elements of the Laplacian pyramid are used as random correction factors.

[0055] According to one example, generating the set of rectified scaled images includes rectifying a plurality of scaled images.

[0056] According to one example, modifying the plurality of scaled images includes modifying the representation of spatial frequencies contained within each scaled image of the plurality of scaled images using a modification factor that increases with respect to spatial frequencies from one scaled image to the next scaled image.

[0057] In one example, the correction coefficients are randomly generated with an expected mean value that increases with spatial frequency from one scaled image to the next.

[0058] According to one example, modifying the plurality of scaled images includes modifying the representation of spatial frequencies contained within each scaled image of the plurality of scaled images using a modification factor that decreases with respect to spatial frequency from one scaled image to the next scaled image.

[0059] In one example, the correction coefficients are randomly generated with an expected mean value that decreases with spatial frequency from one scaled image to the next.

[0060] In one example, the scaled image set includes a Laplacian pyramid.

[0061] In one example, the modified scaled image set is a first modified scaled image set. The processing unit is configured to generate a second modified scaled image set from the scaled image set. The generation of the second modified scaled image set includes modifying two or more scaled images of the plurality of scaled images. The modification is different from the modification for the generation of the first modified scaled image set. The modified medical image is a first modified medical image. The processing unit is configured to generate a second modified medical image, said generation of the second modified medical image includes utilization of the second modified scaled image set. The processing unit is configured to train a medical image analysis machine learning algorithm, said training includes utilization of the second modified medical image.

[0062] In one example, each scaled image of the two or more scaled images is modified differently to generate a second set of modified scaled images.

[0063] 2 shows an example of a medical image generating device 100. The generating device 100 comprises an input unit 120 and a processing unit 130. The input unit is configured to receive a medical image of a body part of a patient. The input unit is configured to provide the medical image to the processing unit. The processing unit is configured to generate a modified medical image. The generating includes modifying two or more spatial frequency bands associated with the medical image.

[0064] In one example, the processing unit is configured to generate a scaled image set having a plurality of scaled images. The generation of the scaled image set includes utilizing a medical image, where each scaled image compromises the representation of spatial frequencies in the medical image, and the representation of spatial frequencies in each of the plurality of scaled images is different. The processing unit is configured to generate a modified scaled image set from the scaled image set. The generation of the modified scaled image set includes modifying two or more scaled images of the plurality of scaled images. The processing unit is configured to generate a modified medical image utilizing the modified scaled image set.

[0065] In one example, the medical image is an x-ray image.

[0066] In one example, the medical image is a CT image.

[0067] In one example, each scaled image of two or more scaled images is modified differently.

[0068] In one example, each scaled frequency corresponds to a band of frequencies.

[0069] In one example, the set of scaled images is a Laplacian pyramid with several elements, where each element is a scaled image and corresponds to a band of frequencies.

[0070] In one example, modifying two or more of the plurality of scaled images includes modifying a representation of spatial frequencies contained in each of the two or more scaled images.

[0071] In one example, modifying two or more of the plurality of scaled images includes modifying representations of spatial frequencies contained within each of the two or more scaled images with a random modification factor.

[0072] In one example, modifying the representation of spatial frequencies contained within the scaled image includes modifying the intensity distribution associated with frequency bands in the scaled image.

[0073] In one example, the modification includes modifying the intensity distribution associated with the frequency bands of the elements of the Laplacian pyramid in the modified image.

[0074] In one example, modifying the representation of spatial frequencies contained within the scaled image includes modifying the intensity distribution of the scaled image.

[0075] In one example, the modification involves modifying the intensity distribution of the elements of the Laplacian pyramid itself (eg, by histogram matching to some randomly selected criteria).

[0076] In one example, the random modification factors include variable weighting factors that are applied to the different scaled images.

[0077] In one example, the weights of the elements of the Laplacian pyramid are used as random correction factors.

[0078] In one example, generating the set of rectified scaled images includes rectifying a plurality of scaled images.

[0079] In one example, modifying the plurality of scaled images includes modifying the representation of spatial frequencies contained in each of the plurality of scaled images using a modification factor that increases with respect to spatial frequencies from one scaled image to the next scaled image.

[0080] In one example, the correction coefficients are randomly generated with an expected mean value that increases with spatial frequency from one scaled image to the next.

[0081] In one example, modifying the plurality of scaled images includes modifying the representation of spatial frequencies contained in each of the plurality of scaled images using a modification factor that decreases with respect to spatial frequencies from one scaled image to the next scaled image.

[0082] In one example, the correction coefficients are randomly generated with an expected mean value that decreases with spatial frequency from one scaled image to the next.

[0083] In one example, the scaled image set includes a Laplacian pyramid.

[0084] In one example, the modified scaled image set is a first modified scaled image set. The processing unit is configured to generate a second modified scaled image set from the scaled image set. The generation of the second modified scaled image set includes modifying two or more scaled images of the plurality of scaled images. The modification is different from the modification for the generation of the first modified scaled image set. The modified medical image is a first modified medical image, and the processing unit is configured to generate a second modified medical image, and the generation of the second modified medical image includes utilizing the second modified scaled image set.

[0085] In one example, each scaled image of two or more scaled images of the scaled image set is differently modified to generate a second modified scaled image set.

[0086] Fig. 3 shows an example of a medical image analysis system 200. The system 200 comprises an input unit 220 and a processing unit 230. The input unit is configured to receive a review medical image of a patient's body part. The input unit is configured to provide the review medical image to the processing unit. The processing unit is configured to analyze the patient's body part. The analysis comprises interrogating the review medical image with a trained machine learning algorithm. The machine learning algorithm is trained as described with respect to Fig. 2, such training comprising the use of at least one modified medical image generated by the generating device.

[0087] In one example, the analysis of the patient's body part includes interrogating a modified version of the review medical image with a trained machine learning algorithm, the modified version of the review medical image being generated by a generation device as described with respect to FIG.

[0088] Thus, so-called "test-time augmentation" is provided, where the analysis algorithm is applied to both the original image being analyzed and its augmented version. A final prediction is formed by aggregating the predictions for the different input versions, which can typically be by taking an averaging. Thereby, the overall algorithm is made more robust and less sensitive to variations in the input image characteristics, by in effect averaging predictions of both the original input and its augmented version.

[0089] In one example, the diagnostic medical image is an X-ray image.

[0090] In one example, the examination medical image is a CT image.

[0091] In one example, the medical image analysis machine learning algorithm is a neural network.

[0092] In one example, medical image analysis machine learning algorithms are trained for automatic disease detection.

[0093] In one example, a medical image analysis machine learning algorithm is trained for image segmentation of a body part, e.g., bone.

[0094] In one example, a medical image analysis machine learning algorithm is trained for the regression of organ extents, such as fetal head diameter.

[0095] In one example, a medical image analysis machine learning algorithm is trained for an exam type or view classification.

[0096] In one example, a machine learning algorithm was trained using medical images.

[0097] In this way, a real image can be used to generate one or more synthetic images similar to the real image, which are used to train a medical image analysis machine learning algorithm, although the real image itself may be one of the images used directly to train the machine learning algorithm.

[0098] 4 shows an example of a method 300 for training a medical image analysis algorithm, the method comprising receiving, by an input unit, a medical image of a patient's body part, by the input unit, in a receiving step 310, providing, by the input unit, the medical image to a processing unit, in a providing step 320, generating, by the processing unit, a modified medical image, in a generating step 330, said generating comprising modifying two or more spatial frequency bands associated with the medical image, and training, by the processing unit, in a training step 340, a medical image analysis machine learning algorithm utilizing the modified medical image.

[0099] In one example, the medical image is an x-ray image.

[0100] In one example, the medical image is a CT image.

[0101] In one example, the machine learning algorithm is a neural network.

[0102] In one example, each scaled image of two or more scaled images is modified differently.

[0103] In one example, the processing unit is configured to train a medical image analysis machine learning algorithm, the training including utilization of medical images.

[0104] According to one example, the method comprises:

[0105] generating, by the processing unit, a scaled image set comprising a plurality of scaled images, the generating comprising utilizing the medical image, each scaled image compromising a representation of spatial frequencies in the medical image, the representation of spatial frequencies in each of the plurality of scaled images being different;

[0106] generating, by the processing unit, a modified scaled image set from the scaled image set, the generating including modifying two or more scaled images of the plurality of scaled images;

[0107] generating, by the processing unit, a modified medical image utilizing the modified scaled image set;

[0108] has.

[0109] In one example, the machine learning algorithm is a neural network.

[0110] In one example, each scaled image of two or more scaled images is modified differently.

[0111] In one example, each of the scaled images corresponds to a band of frequencies.

[0112] In one example, the set of scaled images is a Laplacian pyramid with several elements, where each element is a scaled image and corresponds to a band of frequencies.

[0113] In one example, the medical analysis machine learning algorithm includes utilizing medical images.

[0114] In one example, the method includes modifying a representation of spatial frequencies contained in each of the two or more scaled images.

[0115] In one example, the method includes modifying a representation of spatial frequencies contained in each of the two or more scaled images with a random modification factor.

[0116] In one example, modifying spatial frequencies contained in the scaled image includes modifying intensity distributions associated with frequency bands in the scaled image.

[0117] In one example, the modification includes modifying the intensity distribution associated with the frequency bands of the elements of the Laplacian pyramid in the modified image.

[0118] In one example, said modifying the representation of spatial frequencies contained within the scaled image comprises modifying the intensity distribution of the scaled image.

[0119] In one example, such modification involves modifying the intensity distribution of the elements of the Laplacian pyramid itself (eg, by histogram matching against some randomly selected criteria).

[0120] In one example, the random modification factors include variable weighting factors that are applied to the different scaled images.

[0121] In one example, the weights of the elements of the Laplacian pyramid are used as random correction factors.

[0122] In one example, the method includes rectifying the plurality of scaled images.

[0123] In one example, modifying the plurality of scaled images includes modifying the representation of spatial frequencies contained in each of the plurality of scaled images using a modification factor that increases with respect to spatial frequencies from one scaled image to the next scaled image.

[0124] In one example, the correction coefficients are randomly generated with an expected mean value that increases with spatial frequency from one scaled image to the next.

[0125] In one example, modifying the plurality of scaled images includes modifying representations of spatial frequencies contained in each of the plurality of scaled images with a modification factor that decreases with respect to spatial frequencies from one scaled image to the next scaled image.

[0126] In one example, the correction coefficients are randomly generated with an expected mean value that decreases with spatial frequency from one scaled image to the next.

[0127] In one example, the scaled image set includes a Laplacian pyramid.

[0128] In one example, the modified scaled image set is a first modified scaled image set. The method includes generating, by a processing unit, a second modified scaled image set from the scaled image set. The generating of the second modified scale image set includes modifying two or more scale images of the plurality of scaled images, the modification being different from the modification for generating the first modified scale image set. The modified medical image is also the first modified medical image. The method includes generating, by a processing unit, a second modified medical image, the generating including utilizing the second modified scaled image set. The method includes training, by a processing unit, a medical image analysis machine learning algorithm, the training including utilizing the second modified medical image.

[0129] In one example, each scaled image of the two or more scaled images is modified differently to generate a second set of modified scaled images.

[0130] 5 illustrates an example of a medical image generating method 400. The method 400 includes a receiving step 410, in which an input unit constructs a medical image of a patient's body part that is provided to a processing unit, and a generating step 420, in which a modified medical image is generated by the processing unit, the generating step including modifying two or more spatial frequency bands associated with the medical image.

[0131] In one example, the medical image is an x-ray image.

[0132] In one example, the medical image is a CT image.

[0133] According to one example, a method includes generating, by the processing unit, a scaled image set comprising a plurality of scaled images, the generating including utilizing the medical image, each scaled image compromising a representation of spatial frequencies in the medical image, such that the representation of spatial frequencies in each scaled image of the plurality of scaled images is different; generating, by the processing unit, a modified scaled image set from the scaled image set, the generating including modifying two or more scaled images of the plurality of scaled images; and generating, by the processing unit, the modified medical image, the generating including utilizing the modified scaled image set.

[0134] In one example, each scaled image of two or more scaled images is modified differently.

[0135] In one example, each of the scaled images corresponds to a band of frequencies.

[0136] In one example, the set of scaled images is a Laplacian pyramid with several elements, where each element is a scaled image and corresponds to a band of frequencies.

[0137] In one example, the method includes modifying a representation of spatial frequencies contained in each of the two or more scaled images.

[0138] In one example, the method includes modifying a representation of spatial frequencies contained within each scaled image of two or more scaled images with a random modification factor.

[0139] In one example, modifying the representation of spatial frequencies contained within the scaled image includes modifying intensity distributions associated with frequency bands of the scaled image.

[0140] In one example, the modifying includes modifying intensity distributions associated with frequency bands of elements of a Laplacian pyramid in the modified image.

[0141] In one example, modifying the representation of spatial frequencies contained within the scaled image includes modifying the intensity distribution of the scaled image.

[0142] In one example, the modifying involves modifying the intensity distribution of the elements of the Laplacian pyramid itself (eg, by histogram matching against some randomly selected criteria).

[0143] In one example, the random modification factors include variable weighting factors that are applied to the different scaled images.

[0144] In one example, the weights of the elements of the Laplacian pyramid are used as random correction factors.

[0145] In one example, the method includes rectifying the plurality of scaled images.

[0146] In one example, modifying the plurality of scaled images includes modifying a representation of spatial frequencies contained within each scaled image of the plurality of scaled images using a modification factor that increases with respect to spatial frequencies from one scaled image to the next scaled image.

[0147] In one example, the correction coefficients are randomly generated with an expected mean value that increases with spatial frequency from one scaled image to the next.

[0148] In one example, modifying the plurality of scaled images includes modifying a representation of spatial frequencies contained within each scaled image of the plurality of scaled images with a modification factor that decreases with respect to spatial frequencies from one scaled image to the next scaled image.

[0149] In one example, the correction coefficients are randomly generated with an expected mean value that decreases with spatial frequency from one scaled image to the next.

[0150] In one example, the scaled image set includes a Laplacian pyramid.

[0151] In one example, the modified scaled image set is a first modified scaled image set. The method includes generating, by a processing unit, a second modified scaled image set from the scaled image set, the generating including modifying two or more scaled images of the plurality of scaled images, the modifying being different than the modifying to generate the first modified scaled image set. The modified medical image is a first modified medical image. The method includes generating, by a processing unit, a second modified medical image, the generating including utilizing the second modified scaled image set.

[0152] In one example, each scaled image of the two or more scaled images is modified differently to generate a second set of modified scaled images.

[0153] Figure 6 illustrates an example of a medical image analysis method 500. The method 500 comprises: receiving, by an input unit, a review medical image of a patient's body part in a receiving step 510; providing, by the input unit, the review medical image to a processing unit in a providing step 520; and analyzing, by the processing unit, the patient's body part in an analyzing step 530, said analysis comprising interrogating the review medical image with a trained machine learning algorithm, said trained algorithm having been trained utilizing at least one modified medical image generated by the method described with respect to Figure 5.

[0154] In one example, analyzing the patient's body part includes interrogating a modified version of the examination medical image using a trained machine learning algorithm, where the modified version of the examination medical image was generated by the method described with respect to FIG.

[0155] In one example, the medical image is an x-ray image.

[0156] In one example, the medical image is a CT image.

[0157] In one example, the medical image analysis machine learning algorithm is a neural network.

[0158] In one example, a medical image analysis machine learning algorithm is trained using medical images.

[0159] In one example, medical image analysis machine learning algorithms are trained for automatic disease detection.

[0160] In one example, a medical image analysis machine learning algorithm is trained for image segmentation of a body part, e.g., bone.

[0161] In one example, a medical image analysis machine learning algorithm is trained on the regression of organ size, e.g. fetal head diameter.

[0162] In one example, a medical image analysis machine learning algorithm is trained for an exam type or view classification.

[0163] The apparatus for training a medical image analysis algorithm, the medical image generating apparatus, the medical image analysis system, the method for training a medical image analysis algorithm, the medical image generating method, and the medical image analysis method are described in further detail with reference to Figures 7 to 10. Here, the newly developed technology is described using a specific example of X-ray image analysis, but the new technology is also applicable to other image modalities such as CT.

[0164] The new development technique described herein avoids the need to collect annotated X-ray images of various image styles, which is time-consuming, expensive, or even impossible. This is achieved by multi-scale data augmentation. In this way, any number of random and realistic image styles can be fed into the training of the neural network without extra annotation effort, and other machine learning algorithms can also be utilized. Furthermore, unrealistic (i.e., not prevalent in nature) image styles can also be generated in a controlled manner to allow the neural network to learn a meaningful abstract representation of the input image.

[0165] Examples of today's image processing can be found, for example, in the following white papers: https: / / www.philips.de / c-dam / b2bhc / master / landing-pages / dynamic-unique / philips-dynamic-unique-white-paper.download.pdf https: / / www.philips.com / c-dam / b2bhc / us / Products / Category / radiography / radiography / unique-two-product-overview.pdf

[0166] The new developed technique described here mimics the image processing as if it were applied with a wide variety of different configuration parameters of these algorithms. In other words, the new technique provides multi-scale data augmentation that mimics different post-processing settings and thereby the corresponding image styles.

[0167] The following discussion focuses on an example involving detecting bone contours in ankle AP X-ray images using supervised learning and convolutional neural networks (e.g., U-Net) with reference to Figs. 7-10. Other learning tasks, such as disease / pathology classification or regression tasks (e.g., plane / pose estimation), can also benefit from this new development. Typically, only a few hundred images with annotated bone contours are available for training, such that large capacity neural networks generally tend to overfit to the training data distribution. Mechanisms for overfitting can include the neural network memorizing the specific training data noise / texture patterns, or the network implicitly exploiting the image style of the training data. Applying a trained neural network to unseen images with noise / texture patterns or image styles that differ significantly from the distribution of the training data typically results in poor prediction results. The new development techniques discussed here alleviate these problems.

[0168] A typical scenario that is highly susceptible to this effect is a classification task, where the positive training samples are of a different style than the negative training samples, and the network learns to detect the style instead of the desired discriminatory image features.

[0169] To alleviate this poor generalizability, the new technique is based on the following concept.

[0170] Each training image is decomposed into a number of scale components, which are stored in memory. For example, a Laplace pyramid can be used as a scale hierarchical representation (see Figure 7), where lower (higher) Laplace pyramid bands compromise larger (smaller) structures in the image.

[0171] At each training iteration, randomly drawn training samples are randomly augmented as follows:

[0172] The Laplace bands are randomly modified.

[0173] A new image is constructed from the modified Laplace bands using Laplace pyramid reconstruction (see Figures 8-10).

[0174] The corresponding annotations (eg, bone contour label masks) are kept unchanged.

[0175] Various variants of the random operation on the Laplace bands are possible. The simplest non-trivial variant consists of multiplying each band by a random coefficient, where the stylistic variability depends on the chosen range of random numbers centered at 1. Specifically, we will exemplify here three sampling variants:

[0176] Monotonically increasing these random coefficients along with the band index results in a different emphasis on fine and medium sized structures (see FIG. 8) and can be considered to emulate preprocessing given the raw input image.

[0177] Conversely, monotonically decreasing these random coefficients with the band index results in different aspects of blurring by emphasizing medium- and large-scale structures (see Fig. 9). This variant can be seen as undoing several preprocessing steps for a given preprocessed image.

[0178] Finally, not enforcing monotonicity of the random coefficients creates a mix of both realistic and artificial looking images (see Figure 10). While such artificial modifications may degrade performance in some tasks, training can benefit from them in other tasks because the neural network is forced to learn abstract representations / concepts of images in order to succeed in its learning task.

[0179] Besides modifying the Laplace bands only by a global random factor, more sophisticated approaches are also possible, such as matching the band histogram of the image to be augmented with that of a randomly drawn reference image (no annotation is required).

[0180] Below are some examples of image decomposition and recomposition:

[0181] The new technique developed is based on decomposing an image in general over a complete set of basis functions that are localized to some degree in both the spatial and frequency domains. The corresponding expansion coefficients are then modified to achieve different image styles. A modified version of the input image is then reconstructed / synthesized. This approach has the advantage that image contrast can be modified in a controlled manner at multiple scales.

[0182] In a particular example, a Laplacian image pyramid can be used to construct such an overcomplete base. Given an input image to be modified, a Gaussian pyramid is first constructed recursively by convolution with a blur filter B and subsampling every two pixels via an operator D. As the 0th element of this recursion, the i-th level is constructed as follows: x i =D(B * x i-1 )

[0183] where * denotes convolution. To rectify an image with respect to n scales, the Laplacian pyramid is x (assuming the image is large enough for n-fold subsampling). n The corresponding elements of the Laplacian pyramid are computed recursively by upsampling the elements of the Gaussian pyramid as follows: L i =x n-i -B*(Ux n-i+1 )), i=1,...,n where U denotes upsampling by inserting zeros between each pair of adjacent pixels. The Gaussian pyramid element corresponding to the maximum scale, and the elements L1, , L n Given, the augmented image can be synthesized recursively: y i =c n-i L n-i +B*(Uy i+1 ), i=0,···,n-1 where y n =c0x n and c0, ,c n denotes a sequence of real random numbers.

[0184] The augmented image is given by y=y0. If all ci are equal to 1, the input image is not modified at all, i.e. y=x. By randomly deriving the coefficients and sorting them in increasing (decreasing) order, small (large) scale contrast is enhanced, i.e. multi-scale sharpening (blurring). If the coefficients are not sorted, a wide variety of additional image styles can be achieved. Note that the composite image depends linearly on the coefficients, and by renormalizing the coefficients, any non-zero coefficient can be used as a global scaling coefficient.

[0185] In another exemplary embodiment, a computer program or a computer program element is provided, characterized in that it is configured to perform, on a suitable system, any of the method steps of the method according to one of the previous embodiments.

[0186] Thus, the computer program element can be stored in a computing unit which can be part of the present embodiment. This computing unit can be configured to execute or cause the execution of the steps of the above-mentioned method. Furthermore, the computing unit can be configured to operate the components of the above-mentioned devices and / or systems. The computing unit can be configured to operate automatically and / or to execute user instructions. The computer program can be loaded into a working memory of a data processor. The data processor can thus be equipped to execute a method according to one of the above-mentioned embodiments.

[0187] This exemplary embodiment of the invention encompasses both computer programs that use the invention from the outset and computer programs that, through updates, turn existing programs into programs that use the invention.

[0188] Moreover, a computer program element may provide all the steps required to perform the procedures of the exemplary embodiments of the methods described above.

[0189] According to another exemplary embodiment of the present invention, a computer readable medium such as a CD-ROM, a USB stick or the like is presented, said computer readable medium having a computer program element stored thereon, said computer program element being as described by the preceding section.

[0190] The computer program may be stored and / or distributed on a suitable medium, such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but may also be distributed in other forms or via the Internet or other wired or wireless telecommunications systems, etc.

[0191] However, the computer program can also be presented over a network such as the World Wide Web and downloaded into the working memory of a data processor from such a network. According to a further exemplary embodiment of the invention, a medium for making a computer program element available for downloading is provided, the computer program element being configured to perform a method according to one of the aforementioned embodiments of the invention.

[0192] It should be noted that the embodiments of the present invention are described with reference to different subject matters. In particular, some embodiments are described with reference to method type claims, and other embodiments are described with reference to device type claims. However, those skilled in the art will understand from the above and the following description that, unless otherwise indicated, any combination of features belonging to one type of subject matter, as well as any combination between features relating to different subject matters, is disclosed in the present application. However, all features can be combined to provide a synergistic effect that is higher than the simple sum of the features.

[0193] While the invention has been illustrated and described in detail in the drawings and the foregoing description, such illustration and description are to be considered exemplary or explanatory and not restrictive. The invention is not limited to the disclosed embodiments. Other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure and the dependent claims.

[0194] In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. Any reference signs in the claims should not be interpreted as limiting their scope.

Claims

1. An apparatus for training a medical image analysis algorithm, comprising: an input unit and a processing unit; the input unit is configured to receive a medical image of a body part of a patient; the processing unit is configured to generate a scaled image set having a plurality of scaled images, the generation including utilization of the medical image, each scaled image having a representation of a spatial frequency in the medical image, and the representation of the spatial frequency in each of the plurality of scaled images being different; configured to generate a modified scaled image set from the scaled image set, the generation including modifying two or more of the plurality of scaled images; configured to generate a modified medical image using the modified scaled image set, the generation of the modified medical image having a modification of two or more spatial frequency bands associated with the medical image; configured to train a medical image analysis machine learning algorithm using the modified medical image; the modification of the two or more images among the plurality of scaled images includes modifying the representation of the spatial frequency included in each of the two or more scaled images using a random modification coefficient, and / or the generation of the modified scaled image set includes modifying the plurality of scaled images, and the modification of the plurality of scaled images uses a modification coefficient that increases with respect to spatial frequency from one scaled image to the next scaled image, or uses a modification coefficient that decreases with respect to spatial frequency from one scaled image to the next scaled image to include modification of the representation of the spatial frequency included in each of the plurality of scaled images; an apparatus.

2. A medical image generation apparatus, comprising: an input unit and a processing unit; the input unit is configured to receive a medical image of a body part of a patient; the processing unit configured to generate a set of scaled images having a plurality of scaled images, said generation including utilization of the medical image, each scaled image having a representation of a spatial frequency in the medical image, the representation of the spatial frequency in each of the plurality of scaled images being different, configured to generate a set of modified scaled images from the set of scaled images, said generation including modification of two or more of the plurality of scaled images, configured to generate a modified medical image using the set of modified scaled images, said generation including modifying two or more spatial frequency bands associated with the medical image, the modification of the two or more of the plurality of scaled images includes modifying the representation of the spatial frequency included in each of the two or more scaled images using a random modification coefficient, and / or the generation of the set of modified scaled images includes modifying the plurality of scaled images, and the modification of the plurality of scaled images is a modification coefficient that increases with respect to spatial frequency from one scaled image to the next scaled image, or a modification coefficient that decreases with respect to spatial frequency from one scaled image to the next scaled image including modifying the representation of the spatial frequency included in each of the plurality of scaled images using, Medical image generation device. **Claim 3**: A medical image analysis system, having an input unit and a processing unit, the input unit being configured to receive a medical image of a body part of a patient, the processing unit being configured to analyze the body part of the patient, said analysis including interrogating the medical image of the examination by a trained machine learning algorithm, the machine learning algorithm being trained using at least one modified medical image generated by the medical image generation device according to claim 2, a medical image analysis system. **Claim 4**: A method for training a medical image analysis algorithm, receiving, by an input unit, a medical image of a body part of a patient, providing, by the input unit, the medical image to a processing unit; generating, by the processing unit, a scaled image set having a plurality of scaled images using the medical image, wherein each scaled image has a representation of a spatial frequency in the medical image and the representation of the spatial frequency in each of the plurality of scaled images is different; generating, by the processing unit, a modified scaled image set from the scaled image set, wherein the generating includes modifying two or more of the plurality of scaled images; generating, by the processing unit, a modified medical image using the modified scaled image set, wherein the generating has a modification of two or more spatial frequency bands associated with the medical image; training, by the processing unit, a medical image analysis machine learning algorithm using the modified medical image; comprising; the modification of the two or more of the plurality of scaled images includes modifying the representation of the spatial frequency included in each of the two or more scaled images using a random modification coefficient, and / or the generating of the modified scaled image set includes modifying the plurality of scaled images, and the modification of the plurality of scaled images includes an increasing modification coefficient with respect to spatial frequency from one scaled image to the next scaled image, or a decreasing modification coefficient with respect to spatial frequency from one scaled image to the next scaled image using to modify the representation of the spatial frequency included in each of the plurality of scaled images; a method.

5. A method for generating a medical image, comprising: receiving, by an input unit, a medical image of a body part of a patient; A step of generating a scaled image set having a plurality of scaled images by using the medical image by a processing unit, wherein each scaled image has a representation of a spatial frequency in the medical image, and the representation of the spatial frequency in each of the plurality of scaled images is different. A step of generating a corrected scaled image set from the scaled image set by the processing unit, wherein the generation includes correcting two or more of the plurality of scaled images. A step of generating a corrected medical image by using the corrected scaled image set by the processing unit, wherein the generation of the corrected medical image has corrections of two or more spatial frequency bands associated with the medical image. having The correction of the two or more images among the plurality of scaled images includes correcting the representation of the spatial frequency included in each of the two or more scaled images by using a random correction coefficient, and / or The generation of the corrected scaled image set includes correcting the plurality of scaled images, and the correction of the plurality of scaled images a correction coefficient that increases with respect to spatial frequency from one scaled image to the next scaled image, or a correction coefficient that decreases with respect to spatial frequency from one scaled image to the next scaled image is used to correct the representation of the spatial frequency included in each of the plurality of scaled images. Method. **Claim 6**: A medical image analysis method, comprising: a step of receiving a medical examination image of a patient's body part by an input unit; a step of providing the medical examination image to a processing unit by the input unit; a step of analyzing the body part of the patient by the processing unit, wherein the analysis includes interrogating the medical examination image by a trained machine learning algorithm. The method has, wherein the trained machine learning algorithm is trained by using at least one corrected medical image generated by the method according to claim 5. A computer program element for controlling the apparatus according to claim 1, and / or the generating apparatus according to claim 2, and / or the system according to claim 3, which, when executed by a processor, is configured to execute the method according to claim 4, and / or the method according to claim 5, and / or the method according to claim 6. A computer-readable medium storing the computer program element according to claim 7.

Images