Machine learning data augmentation using diffusion-based generative models

Through a diffusion-based generative model and denoising system, the artifact problem in medical imaging data expansion is solved, and high-quality expanded images consistent with the real image distribution is generated, solving the data acquisition cost and heterogeneity problems.

CN118941491APending Publication Date: 2024-11-12SIEMENS HEALTHINEERS AG
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410574552.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-05-10
Filing Date
2024-05-10
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The existing medical imaging data augmentation technology based on ML/AI models is prone to introduce artifacts, resulting in domain and distribution offsets of the training data set, and obtaining high-quality training data is expensive and heterogeneous.

Method used

A diffusion-based generation model is adopted to generate an expanded image through noise and transformation, and an iterative improvement is used to generate an expanded image consistent with the real image distribution and without artifacts.

Benefits of technology

It realizes the generation of realistic-looking expansion images, avoids the artifacts introduced by conventional expansion technologies, ensures the quality and consistency of training data, and reduces the cost of data acquisition.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941491A_ABST
    Figure CN118941491A_ABST
Patent Text Reader

Abstract

Systems and methods for generating an augmented image are provided. One or more input medical images are received. At least one of noise and the one or more transforms is applied to the one or more input medical images to generate one or more noisy augmented images. The one or more noisy augmented images are de-noised using a diffusion-based de-noising system to generate one or more de-noised augmented images. The application and denoising are repeated for one or more iterations using the one or more denoised augmented images as the one or more input medical images to generate one or more final augmented images. And outputting one or more final expanded images.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention generally relates to ML (machine learning) / AI (artificial intelligence) based data augmentation, and in particular to ML / AI data augmentation using diffusion-based generative models. Background Art

[0002] The performance of ML / AI-based models is highly dependent on the quality and amount of training data. In clinical practice, ML / AI-based models have been applied to perform various medical imaging analysis tasks, such as, for example, segmentation, classification, detection, quantification, etc. However, the acquisition and annotation of training data for training such ML / AI-based models to perform medical imaging analysis tasks is expensive and heterogeneous between manufacturers, centers, and acquisition protocols. Further, only a subset of the available training data may have appropriate clinical characteristics for training a given model, and this subset of the available data may often represent only a subset of manufacturers, centers, and acquisition protocols.

[0003] Conventionally, training datasets used to train such ML / AI-based models to perform medical imaging analysis are augmented by perturbing real data to generate synthetic data in a controlled manner. However, such conventional augmentation techniques may introduce artifacts that shift the domain and distribution of the training dataset. For example, rotation-based augmentation may introduce interpolation, boundary, and scale artifacts, intensity-based augmentation may introduce unrealistic noise patterns, and pathology-based synthesis-based augmentation may introduce unrealistic transitions between healthy and pathological tissues. Summary of the invention

[0004] According to one or more embodiments, a system and method for generating an augmented image is provided. One or more input medical images are received. At least one of noise and one or more transformations is applied to the one or more input medical images to generate one or more noisy augmented images. The one or more noisy augmented images are denoised using a diffusion-based denoising system to generate one or more denoised augmented images. The applying and denoising are repeated for one or more iterations using the one or more denoised augmented images as one or more input medical images to generate one or more final augmented images. One or more final augmented images are output.

[0005] In one embodiment, applying at least one of noise and one or more transforms to one or more input medical images to generate one or more noisy augmented images includes: applying one or more transforms to one or more input medical images to generate one or more transformed input medical images, adding noise to the one or more transformed input medical images to generate one or more noisy augmented images, and generating an uncertainty map of uncertainty introduced by the one or more transforms and the noise. The uncertainty map can be generated based on the difference between 1) the one or more transformed input medical images and 2) the one or more noisy augmented images. In one embodiment, the one or more noisy augmented images are denoised using a diffusion-based denoising system based on the uncertainty map.

[0006] In one embodiment, areas masked by noise are filled in during a first pass of denoising, and local texture features are captured during a second pass of denoising.

[0007] In one embodiment, one or more transformations are randomly applied to one or more input medical images within user defined parameters.

[0008] In one embodiment, a machine learning based model is trained based on one or more final augmented images for performing a medical imaging analysis task.

[0009] In one embodiment, the diffusion-based denoising system includes at least one of a diffusion probability model, a DDPM (denoising diffusion probability model) and a DDIM (denoising diffusion implicit model).

[0010] These and other advantages of the present invention will become apparent to those of ordinary skill in the art by reference to the following detailed description and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 A workflow for generating one or more augmented images according to one or more embodiments is shown; Figure 2 A method for generating one or more augmented images according to one or more embodiments is shown; Figure 3 An exemplary artificial neural network that may be used to implement one or more embodiments is shown; Figure 4 shows a convolutional neural network that can be used to implement one or more embodiments; and Figure 5 A high-level block diagram of a computer that can be used to implement one or more embodiments is shown. DETAILED DESCRIPTION

[0012] The present invention generally relates to methods and systems for machine learning data augmentation using diffusion-based generative models. Embodiments of the present invention are described herein to give a visual understanding of such methods and systems. A digital image is typically composed of digital representations of one or more objects (or shapes). Digital representations of objects are generally described herein in terms of identifying and manipulating objects. Such manipulations are virtual manipulations performed in the memory or other circuitry / hardware of a computer system. Therefore, it is to be understood that embodiments of the present invention may be performed within a computer system using data stored within the computer system.

[0013] Various augmentation techniques have been proposed to expand the training datasets used to train ML / AI-based models. However, conventional augmentation techniques may introduce artifacts that shift the domain and distribution of the training dataset.

[0014] According to the embodiments described herein, a diffusion-based generative model is used to augment training data used to train ML / AI-based models to perform medical imaging analysis tasks. The embodiments described herein apply a diffusion-based denoising system and a refinement denoising system for generating augmented medical images from real images according to an iterative refinement process. The iterative refinement of the augmented medical images results in images having image properties consistent with the distribution of the real images, but without the artifacts that are typically introduced by conventional augmentation techniques.

[0015] Figure 1 A workflow 100 for generating one or more augmented images according to one or more embodiments is shown. Certain elements of the workflow 100 (e.g., systems 104, 106, and / or 108) may be generated using one or more suitable computing devices (e.g., Figure 5 The computer 502) is used to implement it.

[0016] like Figure 1As shown, the workflow 100 includes a noisy data augmentation system 104, a diffusion-based denoising system 106, and a refinement noise system 108. The noisy data augmentation system 104 receives one or more input medical images as input and applies noise and one or more pseudo-random transformations (e.g., rotation or resampling) to the input medical images to generate one or more noisy augmented images. The diffusion-based denoising system 106 receives the noisy augmented images as input and at least partially denoises the noisy augmented images to generate one or more denoised augmented images consistent with the transformations applied by the noisy data augmentation system 104. The refinement noise system 108 receives the denoised augmented images as input and optionally applies further transformations and / or noise to generate one or more refined augmented images. The diffusion-based denoising system 106 receives the refined augmented images and at least partially denoises the refined augmented images to generate one or more final augmented images 110. The refinement noise system 108 and the diffusion-based denoising system 106 may be iteratively applied to the final augmented images 110 for one or more additional iterations.

[0017] Advantageously, the final augmented image 110 is consistent with the transformations applied by the noisy data augmentation system 104 and / or the refined noise system 108, and has imaging properties consistent with the distribution of the input medical image 102. The augmented image is iteratively refined by applying the refined noise system 108 (for applying further transformations and / or noise) and the diffusion-based denoising system 106 (for denoising the refined image) for one or more iterations, introducing small changes to the image at each iteration to achieve incremental imputation of masked regions, thereby obtaining an augmented image that looks realistic.

[0018] Figure 2 A method 200 for generating one or more augmented images is shown in accordance with one or more embodiments. Figure 2 Will refer to Figure 1 The steps of method 200 may be performed by one or more suitable computing devices (such as, for example, Figure 5 502) is executed by the computer.

[0019] exist Figure 2 At step 202, one or more input medical images are received. In one example, Figure 1 As shown in the workflow 100 of , the input medical image is an input medical image 102. The input medical image is a real image of a patient. In one embodiment, the input medical image includes multiple input medical images of the same imaging study.

[0020] The input medical image may be of any suitable modality, such as, for example, CT (computed tomography), MRI (magnetic resonance imaging), US (ultrasound), x-ray, or any other medical imaging modality or combination of medical imaging modalities. The input medical image may be a 2D (two-dimensional) image and / or a 3D (three-dimensional) volume, and may include a single input medical image or multiple input medical images. Therefore, references herein to pixels of a 2D image refer equivalently to voxels of a 3D volume. The input medical image may be received directly from an image acquisition device (such as, for example, a CT scanner) when acquiring the medical image, by loading a previously acquired medical image from a storage device or memory of a computer system, or by receiving a medical image from a remote computer system.

[0021] exist Figure 2 At step 204, at least one of the noise and the one or more transformations is applied to the one or more input medical images to generate one or more noisy augmented images. Figure 1 As shown in workflow 100 of , noise and one or more initial transformations are applied to an input medical image 102 by a noisy data augmentation system 104 .

[0022] The noise may include any suitable noise, such as, for example, Gaussian noise, salt and pepper noise, speckle noise, etc. The noise is applied to the input medical image to obtain a masked portion of the input medical image. Such a masked portion may mask undesirable structures, artifacts, or regions in the input medical image. The transformation may include any suitable image expansion transformation. Examples of such image expansion transformations may include geometric transformations (e.g., rotation, deformation, resampling), color space transformations (intensity transformations), kernel filters, mixed images, random erasure, feature space transformations, lesion synthesis, etc.

[0023] The transformation may be applied to the input medical image in any suitable manner. In one embodiment, the initial transformation is randomly applied to the input medical image (e.g., by randomly rotating the input medical image, or by inserting lesions in random regions of the input medical image). In another embodiment, the initial transformation is applied to the input medical image based on user-defined parameters (e.g., by rotating the input medical image according to a user-defined rotation degree, or by inserting lesions in user-defined regions of the input medical image). In another embodiment, the initial transformation is applied to the input medical image based on parameters extracted from medical data (e.g., by inserting lesions in regions of the input medical image based on the spatial distribution of lesions in a lesion database). In a further embodiment, the initial transformation is applied to the input medical image in a pseudo-random manner, such as by randomly applying the initial transformation within some user-defined parameters (e.g., by randomly rotating the input medical image within a user-defined range), or by randomly applying the initial transformation based on parameters extracted from medical data (e.g., by inserting lesions in regions of the input medical image randomly sampled according to a spatial distribution).

[0024] In one embodiment, a noisy augmented image is generated by first applying a transform to an input medical image to generate a transformed input medical image. Then, noise is added to the transformed input medical image to generate a noisy augmented image. In addition to applying the transform and noise, an uncertainty map of the uncertainty introduced by the transform and noise is also generated. For example, the uncertainty map is calculated as the Gaussian log-likelihood of the noise, that is, the square of the absolute difference between the transformed input medical image and the noisy augmented image. In one embodiment, the uncertainty map is a voxel-by-voxel (or pixel-by-pixel) uncertainty map, wherein each corresponding voxel in the uncertainty map has an intensity value ranging, for example, between 0 and 1, which represents the uncertainty associated with the corresponding voxel. However, the uncertainty map can be in any other suitable form. The uncertainty map can be calculated based on interpolation of the transform. In one example, in the case where the transform includes a rotation and / or deformation transform, the rotation and / or deformation transform can be applied in a region outside the image space of the input medical image. Such a region has a high uncertainty in the uncertainty map. In another example, where at least one transform includes a lesion synthesis transform, the lesion synthesis transform may have a high uncertainty in a region at a boundary of the synthesized lesion, a medium uncertainty within the synthesized lesion, and a low uncertainty in a region far from the synthesized lesion in the uncertainty map. In a further example, where at least one transform includes an interpolation transform, the interpolation transform may be a low uncertainty when the interpolated voxel is relatively close to the voxel center of the input medical image, and a high uncertainty in the uncertainty map when the interpolated voxel is relatively far from the voxel center of the input medical image.

[0025] exist Figure 2 At step 206, one or more noisy extended images are denoised using a diffusion-based denoising system to generate one or more denoised extended images. Figure 1 As shown in the workflow 100 of , the noisy augmented image generated by the noisy data augmentation system 104 is denoised by the diffusion-based denoising system 106 .

[0026] The diffusion-based denoising system can be implemented based on a 2D or 3D diffusion model trained to support the expected distribution of the input medical image. For example, the diffusion model can be a diffusion probability model, a DDPM (denoising diffusion probability model) or a DDIM (denoising diffusion implicit model). However, the diffusion model can be implemented according to any other diffusion-based architecture.

[0027] The diffusion-based denoising system reverses the diffusion process of gradually adding noise to the image in a series of T denoising steps. In one embodiment, the diffusion-based denoising is separated into two passes. In the first pass of the diffusion-based denoising, from step T to step t n Execute the diffusion model. In the second pass of diffusion-based denoising, from step t m Go to step 1 1 A diffusion model is executed. The first pass captures the most challenging aspects of denoising, such as, for example, filling in missing regions of the image (e.g., regions masked by noise) and long-range correlations, and therefore involves a larger number of short steps for stability. By filling in missing regions in the noisy augmented image, the diffusion-based denoising system generates new imaging data in the regions masked by noise. The second pass captures local texture features, involves a smaller number of steps, and is faster to execute. In one embodiment, the first pass computes an intermediate latent representation at a smaller size (e.g., an image with a lower resolution) to reduce the cost of a large number of steps, and the second pass expands the size from the small size of the intermediate latent representation to the target size. In another embodiment, the diffusion-based denoising system denoises the noisy augmented image in the first pass and performs no further processing.

[0028] In one embodiment, the noisy augmented image is denoised based on the uncertainty map. In this embodiment, the diffusion model introduces constraints, such as through the formulation of conditional probability distributions or through additional algorithmic steps, to further constrain the output of the denoised augmented image using the uncertainty map. In particular, by inverting the transform from the input medical image to the final augmented medical image (in Figure 2The diffusion is regulated by the restored constraint (generated at step 208) to ensure that the information present in the input medical image is retained in the generated denoised augmented image. If the transformation is linear, the constraint generates an additional least squares gradient during the diffusion.

[0029] Depending on the constraints, the first pass of diffusion-based denoising can be performed during a previous offline phase, while the second pass can be performed during an online phase during training using the noise generated by perfecting the noise system (in Figure 2 If the diffusion-based denoising system is implemented using DDPM, the diffusion-based denoising system will produce different images at each instance execution because DDPM performs denoising during its own execution, thereby obtaining further data augmentation.

[0030] In one embodiment, where the noisy augmented image is in a different coordinate system (e.g., where the input medical image belongs to an imaging study with a different coordinate system), the diffusion-based denoising system includes an intermediate latent representation that is resampled to an image-specific coordinate system (e.g., the coordinate system of the target denoised augmented image or a corresponding lower resolution coordinate system). The coordinate system of the intermediate latent representation can be defined relative to an original image from the input medical image, an image derived via data augmentation, or an anatomical reference.

[0031] exist Figure 2 At step 208, steps 204 and 206 are repeated for one or more iterations using the one or more denoised augmented images as one or more input medical images to generate one or more final augmented images. Figure 1 As shown in the workflow 100 of , step 204 is repeated to apply the transformation by the noise refinement system 108, and step 206 is repeated to denoise the noisy augmented image by the diffusion-based denoising system 106. By iteratively repeating steps 204 and 206, noise is incrementally added at each iteration of step 204, and masked areas are incrementally filled at each iteration of step 206, thereby iteratively refining the augmented image to generate an image that looks realistic.

[0032] In one embodiment, the noise applied to the input medical image while repeating step 204 comprises an appropriate level of noise. In another embodiment, the one or more transformations applied to the input medical image while repeating step 204 comprise transformations such as, for example, resampling the denoised augmented image to a different coordinate grid that does not introduce substantial uncertainty. During the repetition of step 204 (by Figure 1 The noise and transform applied by the perfect noise system 108 of step 204 can be compared with the noise and transform applied by the perfect noise system 108 of step 204 during the first iteration (by step 204). Figure 1The noise and transformation applied by the noisy data augmentation system 104 belong to the same category, but are constrained to have a smaller impact on the image (eg, rotation is performed only at a relatively small angle).

[0033] exist Figure 2 At step 210, one or more final augmented images are output. In one embodiment, only the final augmented image generated at the last iteration of step 208 is output. In another embodiment, one or more of 1) the denoised augmented image generated at step 206, and 2) the final augmented image generated at some (e.g., each) iteration of step 208 are output. The image may be output by displaying the final augmented image on a display device of the computer system, storing the image on a memory or storage device in the computer system, or by transmitting the image to a remote computer system.

[0034] exist Figure 2 At step 212, a machine learning-based model is trained based on the output image for performing a medical imaging analysis task. In one example, the machine learning-based model is trained based on the final augmented image generated at the last iteration of step 208. In another example, the machine learning-based model is trained based on one or more of 1) the denoised augmented image generated at step 206, and 2) the final augmented image generated at some (e.g., each) iteration of step 208. The output image can augment a training data set for training a machine learning-based model. The medical imaging analysis task can be, for example, segmentation, classification, detection, quantification, or any other suitable medical imaging analysis task.

[0035] Advantageously, method 200 enables generation of realistic looking final augmented images having properties consistent with the distribution of the input medical image and without artifacts that may be introduced using conventional data augmentation techniques. Further, iterative repetition of steps 204 and 206 enables rapid generation of augmented medical images that are structurally similar but distinct images.

[0036] The embodiments described herein are described with respect to the claimed system and the claimed method. Features, advantages, or alternative embodiments herein may be assigned to other claimed objects, and vice versa. In other words, the features described or claimed in the context of the method may be utilized to improve the claims for the system. In this case, the functional features of the method are embodied by providing the target unit of the system.

[0037] In addition, certain embodiments described herein are described with respect to methods and systems for utilizing trained machine learning-based models, and with respect to methods and systems for training machine learning-based models. Features, advantages, or alternative embodiments herein may be assigned to other claimed objects, and vice versa. In other words, features described or claimed in the context of methods and systems for utilizing trained machine learning-based models may be utilized to improve claims for methods and systems for training machine learning-based models, and vice versa.

[0038] In particular, the trained machine learning-based models applied in the embodiments described herein can be adapted by methods and systems for training machine learning-based models. In addition, the input data of the trained machine learning-based models can include advantageous features and embodiments of the training input data, and vice versa. In addition, the output data of the trained machine learning-based models can include advantageous features and embodiments of the output training data, and vice versa.

[0039] Generally speaking, trained machine learning-based models mimic the cognitive functions of humans in connection with other human minds. In particular, through training based on training data, trained machine learning-based models are able to adapt to new situations and detect and extrapolate patterns.

[0040] In general, the parameters of the machine learning based model can be adapted by means of training. In particular, supervised training, semi-supervised training, unsupervised training, reinforcement learning and / or active learning can be used. In addition, representation learning (an alternative term is "feature learning") can be used. In particular, the parameters of the trained machine learning based model can be iteratively adapted through several training steps.

[0041] In particular, the trained machine learning based model may include a neural network, a support vector machine, a decision tree and / or a Bayesian network, and / or the trained machine learning based model may be based on k-means clustering, Q learning, a genetic algorithm and / or an association rule. In particular, the neural network may be a deep neural network, a convolutional neural network or a convolutional deep neural network. In addition, the neural network may be an adversarial network, a deep adversarial network and / or a generative adversarial network.

[0042] Figure 3 An embodiment of an artificial neural network 300 according to one or more embodiments is shown. Alternative terms for "artificial neural network" are "neural network", "artificial neural network" or "neural network". The machine learning networks described herein (such as, for example, Figure 1 The diffusion-based denoising system 106 and Figure 2The diffusion-based denoising system utilized at step 206 and the machine learning-based model utilized at step 212 ) can be implemented using an artificial neural network 300 .

[0043] The artificial neural network 300 includes nodes 302-322 and edges 332, 334, ... 336, wherein each edge 332, 334, ... 336 is a directed connection from a first node 302-322 to a second node 302-322. Generally speaking, the first node 302-322 and the second node 302-322 are different nodes 302-322, and it is also possible that the first node 302-322 and the second node 302-322 are the same. For example, in Figure 3 , edge 332 is a directed connection from node 302 to node 306, and edge 334 is a directed connection from node 304 to node 306. Edges 332, 334, ... 336 from a first node 302-322 to a second node 302-322 are also denoted as "incoming edges" to the second node 302-322, and as "outgoing edges" to the first node 302-322.

[0044] In this embodiment, the nodes 302-322 of the artificial neural network 300 may be arranged in layers 324-330, wherein the layers may include an inherent order introduced by edges 332, 334 ... 336 between the nodes 302-322. In particular, the edges 332, 334 ... 336 may only exist between adjacent layers of nodes. Figure 3 In the embodiment shown in , there is an input layer 324 that includes only nodes 302 and 304 without incoming edges, an output layer 330 that includes only nodes 322 without outgoing edges, and hidden layers 326, 328 between the input layer 324 and the output layer 330. In general, the number of hidden layers 326, 328 can be selected arbitrarily. The number of nodes 302 and 304 within the input layer 324 is generally related to the number of input values ​​of the neural network 300, and the number of nodes 322 within the output layer 330 is generally related to the number of output values ​​of the neural network 300.

[0045] In particular, a (real) number may be assigned as a value to each node 302-322 of the neural network 300. Here, x (n) i represents the value of the i-th node 302-322 of the n-th layer 324-330. The value of the node 302-322 of the input layer 324 is equivalent to the input value of the neural network 300, and the value of the node 322 of the output layer 330 is equivalent to the output value of the neural network 300. In addition, each edge 332, 334, ... 336 may include a weight as a real number, in particular, the weight is a real number in the interval [-1, 1] or in the interval [0, 1]. Here, w(m,n) i,j represents the weight of the edge between the i-th node 302-322 of the m-th layer 324-330 and the j-th node 302-322 of the n-th layer 324-330. (n) i,j is defined for the weight w (n,n+1) i,j .

[0046] In particular, in order to calculate the output value of the neural network 300, the input value is propagated through the neural network. In particular, the values ​​of the nodes 302-322 of the (n+1)th layer 324-330 can be calculated based on the values ​​of the nodes 302-322 of the nth layer 324-330 by the following formula:

[0047] In this article, function f is a transfer function (another term is "activation function"). Known transfer functions are step functions, sigmoid functions (e.g., logistic function, generalized logistic function, hyperbolic tangent function, inverse tangent function, error function, smooth step function), or rectifier functions. Transfer functions are mainly used for normalization purposes.

[0048] In particular, these values ​​are propagated through the neural network layer by layer, where the value of the input layer 324 is given by the input of the neural network 300, where the value of the first hidden layer 326 can be calculated based on the value of the input layer 324 of the neural network, where the value of the second hidden layer 328 can be calculated based on the value of the first hidden layer 326, and so on.

[0049] To set the value of edge w (m,n) i,j , training data must be used to train the neural network 300. In particular, the training data includes training input data and training output data (denoted as t i ). For the training step, the neural network 300 is applied to the training input data to generate the calculated output data. In particular, the training data and the calculated output data include a certain number of values, which is equal to the number of nodes of the output layer.

[0050] In particular, the comparison between the calculated output data and the training data is used to recursively adapt the weights within the neural network 300 (back propagation algorithm). In particular, the weights are changed according to the following formula: where γ is the learning rate and if the (n+1)th layer is not the output layer, it can be based on δ (n+1) jRecursively add the number δ (n) j Calculated as: And if the (n+1)th layer is the output layer 330, it is calculated as: where f' is the first derivative of the activation function, and y (n+1) j is the comparative training value of the j-th node of the output layer 330.

[0051] Figure 4 A convolutional neural network 400 is shown according to one or more embodiments. The machine learning networks described herein (such as, for example, Figure 1 The diffusion-based denoising system 106 and Figure 2 The diffusion-based denoising system utilized at step 206 and the machine learning-based model utilized at step 212 ) can be implemented using the convolutional neural network 400 .

[0052] exist Figure 4 In the embodiment shown in , the convolutional neural network 400 includes an input layer 402, a convolutional layer 404, a pooling layer 406, a fully connected layer 408, and an output layer 410. Alternatively, the convolutional neural network 400 may include several convolutional layers 404, several pooling layers 406, and several fully connected layers 408, as well as other types of layers. The order of these layers can be arbitrarily selected, and the fully connected layer 408 is usually used as the last layer before the output layer 410.

[0053] In particular, within the convolutional neural network 400, the nodes 412-420 of a layer 402-410 may be considered to be arranged as a d-dimensional matrix or a d-dimensional image. In particular, in the two-dimensional case, the values ​​of the nodes 412-420 indexed by i and j in the nth layer 402-410 may be represented as x (n) [i,j] However, the arrangement of the nodes 412-420 of a layer 402-410 has no effect on the computations themselves performed within the convolutional neural network 400, since they are given only by the structure and weights of the edges.

[0054] In particular, the convolution layer 404 is characterized by the structure and weights of the incoming edges that form the convolution operation based on a certain number of kernels. In particular, the structure and weights of the incoming edges are selected so that the value x of the node 414 of the convolution layer 404 is (n) k Based on the value x of the node 412 of the previous layer 402 (n-1) is calculated as the convolution x (n) k =Kk *x (n-1) , where the convolution* in two dimensions is defined as:

[0055] Here, the kth kernel K k Is a d-dimensional matrix (a two-dimensional matrix in this embodiment), which is typically small compared to the number of nodes 412-418 (e.g., a 3×3 matrix or a 5×5 matrix). In particular, this means that the weights of the incoming edges are not independent, but are selected so that they produce the convolution formula. In particular, for the kernel as a 3×3 matrix, there are only 9 independent weights (each entry in the kernel matrix corresponds to an independent weight), regardless of the number of nodes 412-420 in the corresponding layer 402-410. In particular, for a convolutional layer 404, the number of nodes 414 in the convolutional layer is equal to the number of nodes 412 in the previous layer 402 multiplied by the number of kernels.

[0056] If the nodes 412 of the previous layer 402 are arranged as a d-dimensional matrix, the use of multiple kernels can be interpreted as adding an additional dimension (denoted as the "depth" dimension) so that the nodes 414 of the convolution layer 414 are arranged as a (d+1)-dimensional matrix. If the nodes 412 of the previous layer 402 are already arranged as a (d+1)-dimensional matrix including the depth dimension, the use of multiple kernels can be interpreted as expanding along the depth dimension so that the nodes 414 of the convolution layer 404 are also arranged as a (d+1)-dimensional matrix, where the size of the (d+1)-dimensional matrix with respect to the depth dimension is the number of kernels times the size in the previous layer 402.

[0057] An advantage of using convolutional layers 404 is that spatially local correlations in the input data can be exploited by enforcing local connectivity patterns between nodes in adjacent layers, in particular by connecting each node only to a small region of nodes in the previous layer.

[0058] exist Figure 4 In the embodiment shown in , the input layer 402 includes 36 nodes 412 arranged as a two-dimensional 6×6 matrix. The convolution layer 404 includes 72 nodes 414 arranged as two two-dimensional 6×6 matrices, each of which is the result of convolving the values ​​of the input layer with the kernel. Equivalently, the nodes 414 of the convolution layer 404 can be interpreted as being arranged as a three-dimensional 6×6×2 matrix, where the last dimension is the depth dimension.

[0059] The pooling layer 406 is characterized by the activation function of its nodes 416 and the structure and weights of the incoming edges that form the pooling operation based on the nonlinear pooling function f. For example, in the two-dimensional case, the value x of the node 416 of the pooling layer 406 (n) It can be based on the value x of the node 414 of the previous layer 404 (n-1) and is calculated as: x (n) [i,j]=f(x (n-1) [id1,jd2],...,x (n-1) [id1+d1-1,jd2+d2-1]).

[0060] In other words, by using the pooling layer 406, the number of nodes 414, 416 can be reduced by replacing the number d1·d2 of neighboring nodes 414 in the previous layer 404 with a single node 416 in the pooling layer, which is calculated based on the values ​​of the number of neighboring nodes. In particular, the pooling function f can be a maximum function, an average value, or an L2-norm. In particular, for the pooling layer 406, the weights of the incoming edges are fixed and are not modified by training.

[0061] The advantage of using the pooling layer 406 is that the number of nodes 414, 416 and the number of parameters are reduced. This results in a reduction in the amount of computation in the network and in control of overfitting.

[0062] exist Figure 4 In the embodiment shown in , pooling layer 406 is a maximum pooling, which replaces four neighboring nodes with only one node whose value is the maximum of the values ​​of the four neighboring nodes. Maximum pooling is applied to each d-dimensional matrix of the previous layer; in this embodiment, maximum pooling is applied to each of the two two-dimensional matrices, thereby reducing the number of nodes from 72 to 18.

[0063] The fully connected layer 408 is characterized by the fact that most of the edges, in particular all of the edges, between the nodes 416 of the previous layer 406 and the nodes 418 of the fully connected layer 408 exist, and the weight of each of the edges can be adjusted individually.

[0064] In this embodiment, the nodes 416 of the previous layer 406 of the fully connected layer 408 are displayed both as a two-dimensional matrix and additionally as unrelated nodes (which are indicated as a row of nodes with a reduced number of nodes for better presentability). In this embodiment, the number of nodes 418 in the fully connected layer 408 is equal to the number of nodes 416 in the previous layer 406. Alternatively, the number of nodes 416, 418 may be different.

[0065] Furthermore, in this embodiment, the values ​​of the nodes 420 of the output layer 410 are determined by applying the Softmax function to the values ​​of the nodes 418 of the previous layer 408. By applying the Softmax function, the sum of the values ​​of all the nodes 420 of the output layer 410 is 1, and all the values ​​of all the nodes 420 of the output layer are real numbers between 0 and 1.

[0066] The convolutional neural network 400 may also include a ReLU (rectified linear unit) layer or an activation layer having a nonlinear transfer function. In particular, the number of nodes and the structure of the nodes included in the ReLU layer are equivalent to the number of nodes and the structure of the nodes included in the previous layer. In particular, the value of each node in the ReLU layer is calculated by applying a rectification function to the value of the corresponding node of the previous layer.

[0067] The inputs and outputs of different CNN blocks can be wired using summation (residual / dense NNs), element-wise multiplication (attention), or other differentiable operators. Therefore, if the entire pipeline is differentiable, the CNN architecture can be nested instead of sequential.

[0068] In particular, the convolutional neural network 400 can be trained based on a back-propagation algorithm. To prevent overfitting, regularization methods such as dropout of nodes 412-420, random pooling, use of artificial data, weight decay based on L1 or L2 norms, or maximum norm constraints can be used. Different loss functions can be combined for training the same neural network to reflect joint training objectives. A subset of neural network parameters can be excluded from optimization to retain weights pre-trained on another data set.

[0069] The systems, devices, and methods described herein may be implemented using digital circuits, or may be implemented using one or more computers using known computer processors, memory units, storage devices, computer software, and other components. Typically, a computer includes a processor for executing instructions and one or more memories for storing instructions and data. A computer may also include or may be coupled to one or more mass storage devices, such as one or more disks, internal hard disks and removable disks, magneto-optical disks, optical disks, and the like.

[0070] The systems, devices, and methods described herein can be implemented using computers operating in a client-server relationship. Typically, in such a system, the client computer is located remotely from the server computer and interacts via a network. The client-server relationship can be defined and controlled by computer programs running on the respective client and server computers.

[0071] The systems, devices, and methods described herein can be implemented within a network-based cloud computing system. In such a network-based cloud computing system, a server or another processor connected to the network communicates with one or more client computers via the network. The client computer can communicate with the server via, for example, a web browser application that resides and operates on the client computer. The client computer can store data on the server and access the data via the network. The client computer can transmit a request for data or a request for an online service to the server via the network. The server can perform the requested service and provide the data to (one or more) client computers. The server can also transmit data adapted to cause the client computer to perform a specified function (e.g., perform a calculation, display specified data on a screen, etc.). For example, the server can transmit data adapted to cause the client computer to perform one or more steps or functions (including Figure 1 or 2) of one or more steps or functions). Certain steps or functions of the methods and workflows described herein (including Figure 1 One or more steps or functions of 2) can be performed by a server or by another processor in a network-based cloud computing system. Certain steps or functions of the methods and workflows described herein (including Figure 1 One or more steps of 2 or 3) can be performed by a client computer in a network-based cloud computing system. The steps or functions of the methods and workflows described herein (including Figure 1 Or one or more steps of 2) can be performed by a server and / or by a client computer in a network-based cloud computing system in any combination.

[0072] The systems, apparatus, and methods described herein may be implemented using a computer program product tangibly embodied in an information carrier (e.g., embodied in a non-transitory machine-readable storage device) for execution by a programmable processor; and the methods and workflow steps described herein (including Figure 1 One or more steps or functions of the above-mentioned processing unit (or 2) can be implemented using one or more computer programs executable by such a processor. A computer program is a set of computer program instructions that can be used directly or indirectly in a computer to perform a certain activity or bring about a certain result. A computer program can be written in any form of programming language (including compiled or interpreted languages), and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0073] Figure 5A high-level block diagram of an example computer 502 that can be used to implement the systems, apparatus, and methods described herein is depicted in FIG. The computer 502 includes a processor 504 operably coupled to a data storage device 512 and a memory 510. The processor 504 controls the overall operation of the computer 502 by executing computer program instructions that define such operation. The computer program instructions may be stored in the data storage device 512, or other computer-readable medium, and loaded into the memory 510 when execution of the computer program instructions is desired. Thus, Figure 1 The method and workflow steps or functions of the method 502 may be defined by computer program instructions stored in the memory 510 and / or the data storage device 512 and may be controlled by the processor 504 executing the computer program instructions. For example, the computer program instructions may be implemented as a program programmed by a person skilled in the art to execute Figure 1 or 2 method and workflow steps or functions of computer executable code. Therefore, by executing the computer program instructions, the processor 504 performs Figure 1 The computer 502 may also include one or more network interfaces 506 for communicating with other devices via a network. The computer 502 may also include one or more input / output devices 508 (e.g., a display, keyboard, mouse, speaker, buttons, etc.) that enable a user to interact with the computer 502.

[0074] The processor 504 may include both general-purpose and special-purpose microprocessors, and may be the sole processor or one of multiple processors of the computer 502. For example, the processor 504 may include one or more central processing units (CPUs). The processor 504, the data storage device 512, and / or the memory 510 may include, be supplemented by, or be incorporated in one or more application-specific integrated circuits (ASICs) and / or one or more field-programmable gate arrays (FPGAs).

[0075] The data storage device 512 and the memory 510 each include a tangible, non-transitory computer-readable storage medium. The data storage device 512 and the memory 510 may each include a high-speed random access memory, such as a dynamic random access memory (DRAM), a static random access memory (SRAM), a double data rate synchronous dynamic random access memory (DDR RAM), or other random access solid-state memory devices, and may include non-volatile memory, such as one or more magnetic disk storage devices (such as an internal hard disk and a removable magnetic disk), a magneto-optical disk storage device, an optical disk storage device, a flash memory device, a semiconductor memory device (such as an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a compact disk read-only memory (CD-ROM), a digital versatile disk read-only memory (DVD-ROM) disk) or other non-volatile solid-state storage devices.

[0076] Input / output devices 508 may include peripheral devices such as printers, scanners, display screens, etc. For example, input / output devices 508 may include a display device (such as a cathode ray tube (CRT) or liquid crystal display (LCD) monitor) for displaying information to a user, a keyboard, and a pointing device such as a mouse or trackball through which a user can provide input to computer 502.

[0077] The image acquisition device 514 can be connected to the computer 502 to input image data (e.g., medical images) to the computer 502. It is possible to implement the image acquisition device 514 and the computer 502 as one device. It is also possible that the image acquisition device 514 and the computer 502 communicate wirelessly via a network. In a possible embodiment, the computer 502 can be remotely located relative to the image acquisition device 514.

[0078] One or more computers, such as computer 502, may be used to implement any or all of the systems and apparatus discussed herein.

[0079] Those skilled in the art will recognize that actual computer or computer system implementations may have other structures and may include other components, and for illustrative purposes, Figure 5 is a high-level representation of some of the components in such a computer.

[0080] Regardless of the use of grammatical terms, individuals with male or female identities are also included in the term. The foregoing specific embodiments should be understood in every respect to be illustrative and exemplary, rather than restrictive, and the scope of the invention disclosed herein is not determined by the specific embodiments, but rather from the claims interpreted in accordance with the full scope allowed by patent law. It is to be understood that the embodiments shown and described herein are merely illustrative of the principles of the invention, and that various modifications may be implemented by those skilled in the art without departing from the scope and spirit of the invention. Various other feature combinations may be implemented by those skilled in the art without departing from the scope and spirit of the invention.

Claims

1. A computer-implemented method comprising: receiving one or more input medical images; applying at least one of noise and one or more transforms to the one or more input medical images to generate one or more noisy augmented images; denoising the one or more noisy extended images using a diffusion-based denoising system to generate one or more denoised extended images; repeating the applying and the denoising for one or more iterations using the one or more denoised augmented images as the one or more input medical images to generate one or more final augmented images; as well as The one or more final augmented images are output.

2. The computer-implemented method of claim 1 , wherein applying at least one of noise and one or more transformations to the one or more input medical images to generate one or more noisy augmented images comprises: applying the one or more transformations to the one or more input medical images to generate one or more transformed input medical images; adding the noise to the one or more transformed input medical images to generate the one or more noisy augmented images; as well as An uncertainty map is generated of uncertainty introduced by the one or more transforms and the noise.

3. The computer-implemented method of claim 2, wherein generating an uncertainty map of uncertainty introduced by the one or more transformations and the noise comprises: The uncertainty map is generated based on a difference between 1) the one or more transformed input medical images and 2) the one or more noisy augmented images.

4. The computer-implemented method of claim 2, wherein denoising the one or more noisy augmented images using a diffusion-based denoising system to generate one or more denoised augmented images comprises: The one or more noisy augmented images are denoised using the diffusion-based denoising system based on the uncertainty map.

5. The computer-implemented method of claim 1 , wherein denoising the one or more noisy augmented images using a diffusion-based denoising system to generate one or more denoised augmented images comprises: filling in areas masked by the noise during a first pass of the denoising; as well as Local texture features are captured during the second pass of the denoising.

6. The computer-implemented method of claim 1 , wherein applying at least one of noise and one or more transformations to the one or more input medical images to generate one or more noisy augmented images comprises: The one or more transformations are randomly applied to the one or more input medical images within user defined parameters.

7. The computer-implemented method of claim 1 , further comprising: A machine learning based model is trained based on the one or more final augmented images for performing a medical imaging analysis task.

8. The computer-implemented method of claim 1, wherein the diffusion-based denoising system comprises at least one of a diffusion probabilistic model, a DDPM (denoising diffusion probabilistic model), and a DDIM (denoising diffusion implicit model).

9. A device comprising: means for receiving one or more input medical images; means for applying at least one of noise and one or more transforms to the one or more input medical images to generate one or more noisy augmented images; means for denoising the one or more noisy extended images using a diffusion-based denoising system to generate one or more denoised extended images; means for repeating said applying and said denoising for one or more iterations using said one or more denoised augmented images as said one or more input medical images to generate one or more final augmented images; as well as Means for outputting the one or more final augmented images.

10. The apparatus of claim 9, wherein the means for applying at least one of noise and one or more transformations to the one or more input medical images to generate one or more noisy augmented images comprises: means for applying the one or more transformations to the one or more input medical images to generate one or more transformed input medical images; means for adding said noise to said one or more transformed input medical images to generate said one or more noisy augmented images; as well as Means for generating an uncertainty map of uncertainty introduced by the one or more transforms and the noise.

11. The apparatus of claim 10, wherein the means for generating an uncertainty map of the uncertainty introduced by the one or more transformations and the noise comprises: Means for generating the uncertainty map based on a difference between 1) the one or more transformed input medical images and 2) the one or more noisy augmented images.

12. The apparatus of claim 10, wherein the means for denoising the one or more noisy extended images using a diffusion-based denoising system to generate one or more denoised extended images comprises: Means for denoising the one or more noisy augmented images using the diffusion-based denoising system based on the uncertainty map.

13. The apparatus of claim 9, wherein the means for denoising the one or more noisy extended images using a diffusion-based denoising system to generate one or more denoised extended images comprises: means for filling in areas masked by said noise during a first pass of said denoising; as well as Means for capturing local texture features during a second pass of said denoising.

14. The apparatus of claim 9, wherein the means for applying at least one of noise and one or more transformations to the one or more input medical images to generate one or more noisy augmented images comprises: Means for randomly applying the one or more transformations to the one or more input medical images within user defined parameters.

15. A non-transitory computer readable medium storing computer program instructions that, when executed by a processor, cause the processor to perform operations comprising: receiving one or more input medical images; applying at least one of noise and one or more transforms to the one or more input medical images to generate one or more noisy augmented images; denoising the one or more noisy extended images using a diffusion-based denoising system to generate one or more denoised extended images; repeating the applying and the denoising for one or more iterations using the one or more denoised augmented images as the one or more input medical images to generate one or more final augmented images; as well as The one or more final augmented images are output.

16. The non-transitory computer-readable medium of claim 15, wherein applying at least one of noise and one or more transformations to the one or more input medical images to generate one or more noisy augmented images comprises: applying the one or more transformations to the one or more input medical images to generate one or more transformed input medical images; adding the noise to the one or more transformed input medical images to generate the one or more noisy augmented images; as well as An uncertainty map is generated of uncertainty introduced by the one or more transforms and the noise.

17. The non-transitory computer readable medium of claim 16, wherein generating an uncertainty map of uncertainty introduced by the one or more transformations and the noise comprises: The uncertainty map is generated based on a difference between 1) the one or more transformed input medical images and 2) the one or more noisy augmented images.

18. The non-transitory computer-readable medium of claim 16, wherein denoising the one or more noisy augmented images using a diffusion-based denoising system to generate one or more denoised augmented images comprises: The one or more noisy augmented images are denoised using the diffusion-based denoising system based on the uncertainty map.

19. The non-transitory computer readable medium of claim 15, the operations further comprising: A machine learning based model is trained based on the one or more final augmented images for performing a medical imaging analysis task.

20. The non-transitory computer readable medium of claim 15, wherein the diffusion-based denoising system comprises at least one of a diffusion probability model, a DDPM (denoising diffusion probability model), and a DDIM (denoising diffusion implicit model).