Systems and methods for domain generalization across variations in medical images
Patent Information
- Application Number
- JP2023528749
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2020-11-13
- Filing Date
- 2021-11-15
- Publication Date
- 2024-11-22
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Current data augmentation strategies for ultrasound images are inadequate in generalizing across time-based imaging modalities, spatial anatomical variations, and scanning procedures, limiting the performance of deep learning models in real-world scenarios due to distribution mismatches and the inability to handle fast and unpredictable changes during image acquisition.
Implementing advanced data augmentation techniques that include modifying ultrasound image data by altering the temporal and spatial relationships between frames, such as removing or adding subsequences, applying geometric distortions, and using Bayesian neural networks to generate uncertainty maps, thereby enhancing the model's ability to handle diverse types of motion and tissue features.
The proposed methods enable deep learning models to better adapt to real-world ultrasound imaging scenarios, improving generalization across various domains and enhancing performance in emergency and inconsistent scanning conditions.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
Technical Field
[0001] Cross - reference to Related Applications This application claims the priority of U.S. Provisional Patent Application No. 63 / 113,397, filed on November 13, 2020, the disclosure of which is hereby incorporated by reference in its entirety.
[0002] Government licensing rights This invention was made with U.S. government support under patent number W81XWH-19-C-0083, granted by U.S. military medical research activities. The U.S. government has certain rights in this invention.
Background Art
[0003] 1. Field The present disclosure generally relates to ultrasonic image processing, and in non - limiting embodiments or aspects, to systems and methods for domain generalization across variations in medical images.
[0004] 2. Technical Considerations Ultrasound has become an increasingly popular technique for medical imaging. For example, ultrasound can be relatively low - risk (e.g., having relatively few potential side effects, etc.), portable, radiation - free, and relatively inexpensive (e.g., compared to other types of medical images).
[0005] Many medical learning techniques, such as deep learning, have been used to analyze ultrasound images in various scenarios. Due to deep learning techniques and recent advancements, it has become possible to increase the performance of models in real - time ultrasound imaging scenarios to or beyond the level of human skills. However, this performance increase is generally applicable only within one domain, such as anatomical structures, scanner types, imaging settings, probe positions, and external artifacts. In particular, within the field of medical imaging using ultrasound, real - life scenarios can present problems to deep learning models trained in the conventional way.
[0006] Certain types of anatomical structures, imaging settings, or injury scenarios present new data that trained deep learning models have not previously encountered. For example, in emergency scenarios, medical professionals may use faster and less consistent scanning methods to capture real-time ultrasound images. Training a model across all conceivable scenarios requires a vast amount of medical imaging data. However, medical imaging datasets are often limited in quantity and distribution. Deep learning models trained on existing medical imaging data often perform poorly when tested with data from different anatomical sites, scanning settings, or imaging scenarios that are more commonly seen in real-world situations. The models cannot be used to generalize to out-of-training distribution data and unseen images. This may limit the real-world use cases of such deep learning models. Furthermore, collecting the data necessary for appropriate training of deep learning models across all conceivable scenarios is extremely time-consuming and expensive. Obtaining labeled ultrasound data with different scanning speeds and imaging conditions for model training is not feasible.
[0007] Some temporal data augmentation methods include window warping, where a time shift is applied to the image data by dropping a certain number of data points within each data sequence, thereby extending the temporal horizon of the data. However, window warping and other methods do not address the rapid and / or unpredictable changes in ultrasound imaging. [Overview of the project] [Problems that the invention aims to solve]
[0008] To address some of these problems, different data augmentation strategies can be used on ultrasound image datasets. However, there are currently no data augmentation strategies that actively generalize across time-based imaging modalities, spatial anatomical variations, scanning procedures, imaging settings, or other perceptible differences in real-time medical imaging. Current data augmentation techniques are generally limited to simple static transformations such as rotation, inversion, skew, and blurring. However, these simple transformations do not allow data augmentation strategies to be fitted to current-state deep learning models. Ultrasound images present a specific set of problems because the training data distribution can differ significantly from the test distribution. Current simple transformation data augmentation strategies cannot accurately capture these distributions.
[0009] A non-limiting embodiment or aspect provides a method for training a machine learning-based artificial intelligence (AI) model to handle various types of motion occurring during image acquisition, the method comprising: capturing image data using an imaging device, thereby generating multiple frames of captured image data, the captured image data including motion between the imaging device and tissue; modifying the captured image data using a computing device, thereby modifying the captured image data to generate modified image data by at least one of: modifying the amount of time between any two frames of the captured image data, removing a subsequence of frames from the captured image data, and adding one or more subsequences of new frames to the captured image data, each of the one or more new frames including at least one of: a copy of at least one frame of the captured image data, a composite frame created from other frames in the captured image data, an interpolated frame generated to appear between other frames in the captured image data, a frame not included in the captured image data, or any combination thereof; and training a machine learning-based AI model based on the modified image data using a computing device.
[0010] In non-limiting embodiments or embodiments, the image data is captured in real time or has been captured in advance. In non-limiting embodiments or embodiments, modifying the captured image data includes removing a subsequence of frames from the captured image data, the method further includes identifying the probability value of each frame in the captured image data, and identifying the subsequence of frames in the captured image data by comparing the probability value of each frame with a predetermined probability threshold. In non-limiting embodiments or embodiments, modifying the captured image data includes removing a subsequence of frames from the captured image data, the method further includes selecting a range of frames from the captured image data, generating a random integer having a value greater than or equal to the first frame number in the range and less than or equal to the last frame number in the range plus 1, and determining the frames to be selected for the subsequence of frames based on the random integer. In a non-limiting embodiment or aspect, modifying the captured image data includes adding one or more new frame subsequences to the captured image data, the method further including randomly generating a time shift of a predetermined length; dividing the captured image data into a plurality of subsequences equal to the predetermined length; generating a random integer for each of the plurality of subsequences; and spatially shifting at least one of the width and height of at least one frame of each of the plurality of subsequences based on the random integer corresponding to the subsequence.In non-limiting embodiments or aspects, the captured image data includes at least one of the following: ultrasound images, optical coherence tomography (OCT) images, CT images, MRI images, PET images, SPECT images, fluorescence fluoroscopy images, X-ray images, mammography images, tomosynthesis images, photoacoustic images, acousto-optics images, endoscopic images, microscopic images, fundus images, scanning laser ophthalmoscope (SLO) images, smartphone images, 3D (depth) images, focal stack images, brightfield images, visible light images, infrared images, ultraviolet images, thermal images, multispectral images, tomography images, projection images, integrated images, reconstructed images, or any combination thereof, and the machine learning-based AI model includes at least one of the following: convolutional neural networks, recurrent neural networks, long short-term memory neural networks, Bayesian networks, Bayesian neural networks, transformer networks, autoencoder networks, deep Boltzmann machines, deep belief networks, random forests, Bayesian random forests, gradient-boosted regression trees, or any combination thereof.
[0011] In non-limiting embodiments or aspects, a system is provided for training a machine learning-based artificial intelligence (AI) to handle various types of motion occurring during video acquisition, the system comprising at least one computing device, wherein the at least one computing device is programmed or configured to capture image data using an imaging device, thereby generating multiple frames of captured image data, the captured image data including motion between the imaging device and tissue; modify the captured image data, thereby modifying the captured image data by at least one of the following: modifying the amount of time between any two images of the captured image data, removing a subsequence of frames from the captured image data, and adding one or more subsequences of new frames to the captured image data, each of the one or more new frames including at least one of a copy of at least one frame of the captured image data, a composite frame created from other frames in the captured image data, an interpolated frame generated to appear between other frames in the captured image data, a frame not included in the captured image data, or any combination thereof, thereby modifying the captured image data to generate modified image data; and training a machine learning-based AI model based on the modified image data.
[0012] In non-limiting embodiments or embodiments, the image data is captured in real time or has been captured in advance. In non-limiting embodiments or embodiments, modifying the captured image data includes removing a subsequence of frames from the captured image data, the method further includes identifying the probability value of each frame in the captured image data, and identifying the subsequence of frames in the captured image data by comparing the probability value of each frame with a predetermined probability threshold. In non-limiting embodiments or embodiments, modifying the captured image data includes removing a subsequence of frames from the captured image data, the method further includes selecting a range of frames from the captured image data, generating a random integer having a value greater than or equal to the first frame number in the range and less than or equal to the last frame number in the range plus 1, and determining the frames to be selected for the subsequence of frames based on the random integer. In a non-limiting embodiment or aspect, modifying the captured image data includes adding one or more new frame subsequences to the captured image data, the method further including randomly generating a time shift of a predetermined length; dividing the captured image data into a plurality of subsequences equal to the predetermined length; generating a random integer for each of the plurality of subsequences; and spatially shifting at least one of the width and height of at least one frame of each of the plurality of subsequences based on the random integer corresponding to the subsequence.In non-limiting embodiments or aspects, the captured image data includes at least one of the following: ultrasound images, optical coherence tomography (OCT) images, CT images, MRI images, PET images, SPECT images, fluorescence fluoroscopy images, X-ray images, mammography images, tomosynthesis images, photoacoustic images, acousto-optics images, endoscopic images, microscopic images, fundus images, scanning laser ophthalmoscope (SLO) images, smartphone images, 3D (depth) images, focal stack images, brightfield images, visible light images, infrared images, ultraviolet images, thermal images, multispectral images, tomography images, projection images, integrated images, reconstructed images, or any combination thereof, and the machine learning-based AI model includes at least one of the following: convolutional neural networks, recurrent neural networks, long short-term memory neural networks, Bayesian networks, Bayesian neural networks, transformer networks, autoencoder networks, deep Boltzmann machines, deep belief networks, random forests, Bayesian random forests, gradient-boosted regression trees, or any combination thereof.
[0013] According to non-limiting embodiments or aspects, a computer program product for training machine learning-based artificial intelligence (AI) to handle various types of motion occurring during video acquisition, comprising at least one non-temporary computer-readable medium including program instructions, the program instructions, when executed by at least one computing device, to capture image data using an imaging device, thereby generating multiple frames of captured image data, the captured image data including motion between the imaging device and the tissue, and to modify the captured image data, thereby altering the amount of time between any two images of the captured image data, and subsequences of frames from the captured image data. A computer program product is provided that modifies the captured image data by at least one of the following: removing a frame and adding one or more subsequences of new frames, each of which includes at least one of the following: a copy of at least one frame of the captured image data, a composite frame created from other frames in the captured image data, an interpolated frame generated to appear between other frames in the captured image data, a frame not included in the captured image data, or any combination thereof; to generate modified image data; and to train a machine learning-based AI model based on the modified image data.
[0014] In non-limiting embodiments or embodiments, the image data is captured in real time or has been captured in advance. In non-limiting embodiments or embodiments, modifying the captured image data includes removing a subsequence of frames from the captured image data, and the program instruction further causes the at least one computing device to identify the probability value of each frame of the captured image data, and to identify the subsequence of frames of the image data by comparing the probability value of each frame with a predetermined probability threshold. In non-limiting embodiments or embodiments, modifying the captured image data includes removing a subsequence of frames from the captured image data, and the program instruction further causes the at least one computing device to select a range of frames from the captured image data, to generate a random integer having a value greater than or equal to the first frame number of the range and less than or equal to the last frame number of the range plus 1, and to determine the frames to be selected for the subsequence of frames based on the random integer. In a non-limiting embodiment or aspect, modifying the captured image data includes adding one or more new frame subsequences to the captured image data, wherein the program instruction causes the at least one computing device to randomly generate the magnitude of the time shift of each subsequence of a frame of a predetermined length; to divide the captured image data into a plurality of subsequences equal to the predetermined length; to generate a random integer for each subsequence of the plurality of subsequences; and to spatially shift at least one of the width and height of at least one frame of each subsequence of the plurality of subsequences based on the random integer corresponding to the subsequence.In non-limiting embodiments or aspects, the captured image data includes at least one of the following: ultrasound images, optical coherence tomography (OCT) images, CT images, MRI images, PET images, SPECT images, fluorescence fluoroscopy images, X-ray images, mammography images, tomosynthesis images, photoacoustic images, acousto-optics images, endoscopic images, microscopic images, fundus images, scanning laser ophthalmoscope (SLO) images, smartphone images, 3D (depth) images, focal stack images, brightfield images, visible light images, infrared images, ultraviolet images, thermal images, multispectral images, tomography images, projection images, integrated images, reconstructed images, or any combination thereof, and the machine learning-based AI model includes at least one of the following: convolutional neural networks, recurrent neural networks, long short-term memory neural networks, Bayesian networks, Bayesian neural networks, transformer networks, autoencoder networks, deep Boltzmann machines, deep belief networks, random forests, Bayesian random forests, gradient-boosted regression trees, or any combination thereof. In a non-limiting embodiment or aspect, modifying the captured image data includes generating a copy of the captured image data and modifying the copy of the captured image data to form the modified image data.
[0015] A non-limiting embodiment or aspect provides a method for training a machine learning-based artificial intelligence (AI) model to handle diverse types of tissue appearances during medical imaging, comprising: capturing a series of training medical images using an imaging device, wherein the series of training medical images includes a plurality of frames in the sequence; generating a set of control points before and after each frame of at least a subset of the plurality of frames and each classified object in each frame using a computing device; generating a set of deformation control points for each frame of the at least subset of frames using a convolutional neural network based on the set of control points and uncertainty maps using a computing device; applying geometric distortion to each frame of the at least subset of frames based on the set of deformation control points, thereby generating a new training image; and training a machine learning-based AI model based on the new training image using a computing device.
[0016] In a non-limiting embodiment or embodiment, the set of deformation control points is generated based on an affine transformation. In a non-limiting embodiment or embodiment, the set of deformation control points is generated by automatically assessing which portion of each frame in the at least subset of frames satisfies an uncertainty threshold. In a non-limiting embodiment or embodiment, the automatic assessment of which portion of each frame in the at least subset of frames satisfies the uncertainty threshold is based on a recognition and / or randomness uncertainty map. In a non-limiting embodiment or embodiment, the recognition and / or randomness uncertainty map is output from the machine learning-based AI model being trained. In a non-limiting embodiment or embodiment, the machine learning-based AI model being trained includes a Bayesian artificial neural network. In a non-limiting embodiment or embodiment, the set of deformation control points is generated based on the output of an individual-agent adversarial augmented artificial neural network configured to process the output of the automatic assessment as input.
[0017] In non-limiting embodiments or aspects, a method is provided for training a machine learning-based artificial intelligence (AI) model to handle diverse types of tissue appearances during medical imaging, the method comprising: capturing a series of training medical images using an imaging device; extending the training medical images into new training images, wherein different distortions target different parts within individual frames of the series of training medical images and / or images; applying the distortions to each new training image of the new training images; and iteratively calculating the distortions during AI learning, wherein the iterative calculation is based on an automated assessment of which parts of the training images are less visually understandable to the AI compared to parts that are more visually understandable to the AI at that moment in training, and / or the degree of difficulty of the AI for each of the different parts of the training images at that moment in training.
[0018] In non-limiting embodiments or embodiments, the distortion is geometric and is applied using moving least-squares image deformation. In non-limiting embodiments or embodiments, the distortion is computed based on the output of an independent agent adversarial augmented artificial neural network that processes an assessment of the degree of difficulty of different image regions as input. In non-limiting embodiments or embodiments, the augmentation system augments a training medical image to a new training image, and the augmentation system is integrated with one or more data augmentation processes such that the augmentation system spot augments specific regions of the image data during training.
[0019] In non-limiting embodiments or aspects, a system is provided which includes at least one computing device programmed or configured to perform the method. In non-limiting embodiments or aspects, a computer program product is provided which, when executed by the at least one computing device, includes at least one non-temporary computer-readable medium which includes program instructions causing the at least one computing device to perform the method.
[0020] In non-limiting embodiments or aspects, a system is provided for training a machine learning-based artificial intelligence (AI) model to handle diverse types of tissue appearances during medical imaging, the system comprising at least one computing device, the at least one computing device being programmed or configured to capture a series of training medical images using an imaging device, the series of training medical images comprising a plurality of frames in a sequence, to generate a set of control points before and after each frame of at least a subset of the frames and each classified object in each frame, to generate a set of deformation control points for each frame of the at least subset of frames using a convolutional neural network based on the set of control points and uncertainty maps, and to apply geometric distortion to each frame of the at least subset of frames based on the set of deformation control points, thereby generating, applying, and training a machine learning-based AI model based on the new training images.
[0021] In a non-limiting embodiment or embodiment, the set of deformation control points is generated based on an affine transformation. In a non-limiting embodiment or embodiment, the set of deformation control points is generated by automatically assessing which portion of each frame in the at least subset of frames satisfies an uncertainty threshold. In a non-limiting embodiment or embodiment, the automatic assessment of which portion of each frame in the at least subset of frames satisfies the uncertainty threshold is based on a recognition and / or randomness uncertainty map. In a non-limiting embodiment or embodiment, the recognition and / or randomness uncertainty map is output from the machine learning-based AI model being trained. In a non-limiting embodiment or embodiment, the machine learning-based AI model being trained includes a Bayesian artificial neural network. In a non-limiting embodiment or embodiment, the set of deformation control points is generated based on the output of an individual-agent adversarial augmented artificial neural network configured to process automatic assessment as input.
[0022] In non-limiting embodiments or aspects, a computer program product is provided for training a machine learning-based artificial intelligence (AI) model to handle diverse types of tissue appearances during medical imaging, comprising at least one non-temporary computer-readable medium containing program instructions, wherein, when executed by at least one computing device, the computer program product causes the at least one computing device to capture a series of training medical images using an imaging device, the series of training medical images comprising a plurality of frames in the sequence, generate a set of control points before and after each frame of at least a subset of the frames and each classified object in each frame, generate a set of deformation control points for each frame of the at least subset of frames using a convolutional neural network based on the set of control points and uncertainty maps, and apply geometric distortion to each frame of the at least subset of frames based on the set of deformation control points, thereby generating, applying, and training a machine learning-based AI model based on the new training images. In a non-limiting embodiment or embodiment, the set of deformation control points is generated based on an affine transformation.
[0023] In a non-limiting embodiment or aspect, the set of deformation control points is generated by automatically assessing which portions of each frame of the at least subset of frames satisfy an uncertainty threshold. In a non-limiting embodiment or aspect, automatically assessing which portions of each frame of the at least subset of frames satisfy the uncertainty threshold is based on an epistemic and / or aleatoric uncertainty map. In a non-limiting embodiment or aspect, the epistemic and / or aleatoric uncertainty map is output from the machine learning-based AI model during training. In a non-limiting embodiment or aspect, the machine learning-based AI model during training includes a Bayesian artificial neural network. In a non-limiting embodiment or aspect, the set of deformation control points is generated based on the output of an individual agent adversarial extended artificial neural network configured to process the output of the automatic assessment as an input.
[0024] According to a non-limiting embodiment or aspect, a system for training a machine learning-based artificial intelligence (AI) model to handle diverse types and / or tissue appearances during medical imaging by acquiring new training data during training, the system including a robot configured to autonomously acquire training image data of a subject, and at least one computing device in communication with the robot, the at least one computing device being configured to, when the training image data is acquired by the robot, assess the image quality of the training image data in real time, create at least one map of the training image data based on the image quality of the training image data, and adjust the acquisition of new training image data by the robot based on the at least one map of the training image data.
[0025] In non-limiting embodiments or embodiments, assessing the image quality includes identifying a mean estimate and an uncertainty estimate of the mean estimate. In non-limiting embodiments or embodiments, identifying the mean estimate and the uncertainty estimate is based on a Gaussian process or a covariance model. In non-limiting embodiments or embodiments, coordinating the acquisition of new training image data includes optimizing at least one of the following image quality parameters: best mean, confidence upper limit, expected improvement, or any combination thereof. In non-limiting embodiments or embodiments, the at least one computing device is further configured to identify the mean estimate and the uncertainty estimate based on a combination of recognizability and randomness uncertainty outputs from a Bayesian AI model being trained. In non-limiting embodiments or embodiments, the organization is autonomously and physically operated by the robot or a second device to alter the appearance of the captured image while the training image data is being acquired. In non-limiting embodiments or embodiments, the at least one computing device is further configured to train a machine learning-based AI model based on the new training image data, and the robot or the second device is configured to operate the organization during runtime use of the machine learning-based AI model. In a non-limiting embodiment or embodiment, the tissue is physically manipulated to appear as tissue captured in the new training image data. In a non-limiting embodiment or embodiment, the at least one computing device is further configured to incrementally modify one or more aspects of the training image data to improve the image quality. In a non-limiting embodiment or embodiment, the at least one computing device includes a first AI system and a second AI system, the first AI system including a student AI system, and the second AI system including a teacher AI system configured to adjust the student AI system based on a set of intended actions.In non-limiting embodiments or embodiments, the training image data is represented in a latent space of independent vectors, each latent vector representing one aspect of the change in the training image data, and the acquisition of new training image data or the expansion of the training image data is based on the latent vector representation. In non-limiting embodiments or embodiments, the at least one computing device is further configured to create or infer the independent latent vectors based on the unraveled data representation. In non-limiting embodiments or embodiments, the training image data includes: ultrasound images, optical coherence tomography (OCT) images, CT images, MRI images, PET images, SPECT images, fluoroscopy images, X-ray images, mammography images, tomosynthesis images, photoacoustic images, acousto-optics images, endoscopic images, microscopic images, fundus images, scanning laser ophthalmoscope (SLO) images, smartphone images, 3D (depth) images, focal stack images, brightfield images, visible light images, infrared images, ultraviolet images, thermal images, multispectral images, tomography images, and projection images. The at least one computing device includes, at least one of the following: integrated image, reconstructed image, or any combination thereof, and the at least one computing device includes: convolutional neural network, recurrent neural network, long short-term memory neural network, Bayesian network, Bayesian neural network, transformer network, autoencoder network, deep Boltzmann machine, deep belief network, random forest, Bayesian random forest, gradient boost machine, gradient boost regression tree, or at least one combination thereof. In a non-limiting embodiment or aspect, the at least one computing device includes a first AI system and a second AI system, the first AI system configured to assess the image quality of the training image data in real time as the training image data is acquired by the robot and to create the at least one map of the training image data based on the image quality of the training image data, and the second AI system configured to coordinate the acquisition of the new training image data based on the at least one map of the training image data.
[0026] According to a non-limiting embodiment or aspect, a method for training a machine learning-based artificial intelligence (AI) model to handle diverse types and / or appearances of tissues during medical imaging by acquiring new training data during training, the method comprising: autonomously acquiring training image data of a subject using a robot using a computing device; assessing the image quality of the training image data in real time as the training image data is acquired by the robot using the computing device; creating at least one map of the training image data based on the image quality of the training image data using the computing device; and adjusting the acquisition of new training image data by the robot based on the at least one map of the training image data using the computing device.
[0027] In non-limiting embodiments or embodiments, assessing the image quality includes identifying a mean estimate and an uncertainty estimate of the mean estimate. In non-limiting embodiments or embodiments, identifying the mean estimate and the uncertainty estimate is based on a Gaussian process or a covariance model. In non-limiting embodiments or embodiments, coordinating the acquisition of new training image data includes optimizing at least one of the following image quality parameters: best mean, confidence upper limit, expected improvement, or any combination thereof. In non-limiting embodiments or embodiments, further includes identifying the mean estimate and the uncertainty estimate based on a combination of recognizability and randomness uncertainty outputs from a Bayesian AI model during training. In non-limiting embodiments or embodiments, the organization is autonomously and physically operated by the robot or a second device to alter the appearance of the captured image while acquiring the training image data. In non-limiting embodiments or embodiments, further includes training a machine learning-based AI model based on the new training image data, wherein the robot or the second device is configured to operate the organization during runtime use of the machine learning-based AI model. In non-limiting embodiments or embodiments, the tissue is physically manipulated to appear as tissue captured in the new training image data. In non-limiting embodiments or embodiments, further includes incrementally modifying one or more aspects of the training image data to improve the image quality. In non-limiting embodiments or embodiments, the training image data is represented in a latent space of independent vectors, each latent vector representing one aspect of change in the training image data, and the acquisition of the new training image data or the expansion of the training image data is based on the latent vector representation. In non-limiting embodiments or embodiments, the at least one computing device is further configured to create or infer the independent latent vectors based on the unraveled data representation.In non-limiting embodiments or aspects, the training image data includes at least one of the following: ultrasound images, optical coherence tomography (OCT) images, CT images, MRI images, PET images, SPECT images, fluoroscopy images, X-ray images, mammography images, tomosynthesis images, photoacoustic images, acousto-optics images, endoscopic images, microscopic images, fundus images, scanning laser ophthalmoscope (SLO) images, smartphone images, 3D (depth) images, focal stack images, brightfield images, visible light images, infrared images, ultraviolet images, thermal images, multispectral images, tomography images, projection images, integrated images, reconstructed images, or any combination thereof, and the computing device includes at least one of the following: convolutional neural networks, recurrent neural networks, long short-term memory neural networks, Bayesian networks, Bayesian neural networks, transformer networks, autoencoder networks, deep Boltzmann machines, deep belief networks, random forests, Bayesian random forests, gradient boosted machines, gradient boosted regression trees, or any combination thereof.
[0028] In non-limiting embodiments or aspects, a computer program product is provided for training a machine learning-based artificial intelligence (AI) model to handle diverse types and / or tissue appearances during medical imaging by acquiring new training data during training, the product comprising at least one non-temporary computer-readable medium including program instructions, the program instructions, when executed by at least one computing device, cause the at least one computing device to: autonomously acquire training image data of a target using a robot; assess the image quality of the training image data in real time as the training image data is acquired by the robot; create at least one map of the training image data based on the image quality of the training image data; and coordinate the acquisition of new training image data by the robot based on the at least one map of the training image data.
[0029] In non-limiting embodiments or embodiments, assessing the image quality includes identifying a mean estimate and an uncertainty estimate of the mean estimate. In non-limiting embodiments or embodiments, identifying the mean estimate and the uncertainty estimate is based on a Gaussian process or a covariance model. In non-limiting embodiments or embodiments, coordinating the acquisition of new training image data includes optimizing at least one of the following image quality parameters: best mean, confidence upper limit, expected improvement, or any combination thereof. In non-limiting embodiments or embodiments, the program instructions cause the at least one computing device to further identify the mean estimate and the uncertainty estimate based on a combination of recognizability and randomness uncertainty outputs from a Bayesian AI model being trained. In non-limiting embodiments or embodiments, the organization is autonomously and physically operated by the robot or a second device to alter the appearance of the captured image while acquiring the training image data. In a non-limiting embodiment or embodiment, the program instructions further cause the at least one computing device to train a machine learning-based AI model based on the new training image data, and the robot or the second device is configured to manipulate the tissue during runtime use of the machine learning-based AI model. In a non-limiting embodiment or embodiment, the tissue is physically manipulated to appear as tissue captured in the new training image data. In a non-limiting embodiment or embodiment, the program instructions further cause the at least one computing device to incrementally modify one or more aspects of the training image data to improve the image quality. In a non-limiting embodiment or embodiment, the training image data is represented in a latent space of independent vectors, each latent vector representing one aspect of change in the training image data, and the acquisition of new training image data or the expansion of the training image data is based on the latent vector representation. In a non-limiting embodiment or embodiment, the program instructions further cause the at least one computing device to create or infer the independent latent vectors based on a disassembled data representation.In non-limiting embodiments or aspects, the training image data may include: ultrasound images, optical coherence tomography (OCT) images, CT images, MRI images, PET images, SPECT images, fluorescence fluoroscopy images, X-ray images, mammography images, tomosynthesis images, photoacoustic images, acousto-optical images, endoscopic images, microscopic images, fundus images, scanning laser ophthalmoscope (SLO) images, smartphone images, 3D (depth) images, focal stack images, brightfield images, visible light images, infrared images, ultraviolet images, thermal images, multispectral images, tomography images, and projection images. The at least one computing device includes, at least one of the following: a combined image, a reconstructed image, or any combination thereof, and the at least one computing device includes, at least one of the following: a convolutional neural network, a recurrent neural network, a long short-term memory neural network, a Bayesian network, a Bayesian neural network, a transformer network, an autoencoder network, a deep Boltzmann machine, a deep belief network, a random forest, a Bayesian random forest, a gradient boosted machine, a gradient boosted regression tree, or any combination thereof.
[0030] A non-limiting embodiment or aspect provides a method for training a machine learning-based artificial intelligence (AI) model to handle various types of changes occurring during image acquisition, the method comprising: capturing image data using an imaging device to generate captured image data of multiple frames, the captured image data including movement between the imaging device and tissue; identifying at least one latent vector representation; inputting the at least one latent vector representation and the captured image data into a variation autoencoder model; decoding the at least one latent vector representation and the captured image data into adversarial image data using the variation autoencoder model; and training a machine learning-based AI model on the adversarial image data. In a non-limiting embodiment or aspect, each vector dimension of the latent vector representation represents one degree of change in a target domain. In a non-limiting embodiment or aspect, each latent vector representation is independent of each other latent vector representation.
[0031] In non-limiting embodiments or aspects, a system is provided for training a machine learning-based artificial intelligence (AI) model to handle various types of changes occurring during image acquisition, the system comprising at least one computing device, the at least one computing device being programmed or configured to capture image data using an imaging device, thereby generating captured image data of multiple frames, the captured image data including movement between the imaging device and tissue, to identify at least one latent vector representation, to input the at least one latent vector representation and the captured image data into a variation autoencoder model, to decode the at least one latent vector representation and the captured image data into adversarial image data using the variation autoencoder model, and to train a machine learning-based AI model based on the adversarial image data. In non-limiting embodiments or aspects, each vector dimension of the latent vector representation represents one degree of change in a target domain. In non-limiting embodiments or aspects, each latent vector representation is independent of each other latent vector representation.
[0032] In non-limiting embodiments or aspects, a computer program product is provided for training a machine learning-based artificial intelligence (AI) model to handle various types of changes occurring during image acquisition, comprising at least one non-temporary computer-readable medium including program instructions, wherein, when executed by at least one computing device, the computer program product causes the at least one computing device to: capture image data using an imaging device, thereby generating captured image data of multiple frames, the captured image data including movement between the imaging device and tissue; identify at least one latent vector representation; input the at least one latent vector representation and the captured image data into a variation autoencoder model; decode the at least one latent vector representation and the captured image data into adversarial image data using the variation autoencoder model; and train a machine learning-based AI model based on the adversarial image data. In non-limiting embodiments or aspects, each vector dimension of the latent vector representation represents one degree of change of a target domain. In non-limiting embodiments or aspects, each latent vector representation is independent of each other latent vector representation.
[0033] Further embodiments or aspects are described in the following numbered clauses:
[0034] Clause 1: A method for training a machine learning-based artificial intelligence (AI) model to handle diverse types of motion occurring during image acquisition, comprising: capturing image data using an imaging device, thereby generating multiple frames of captured image data, the captured image data including motion between the imaging device and tissue; modifying the captured image data using a computing device, thereby modifying the captured image data to generate modified image data by at least one of: altering the amount of time between any two frames of the captured image data, removing a subsequence of frames from the captured image data, and adding one or more subsequences of new frames to the captured image data, each of the one or more new frames including at least one of: a copy of at least one frame of the captured image data, a composite frame created from other frames in the captured image data, an interpolated frame generated to appear between other frames in the captured image data, a frame not included in the captured image data, or any combination thereof; and training a machine learning-based AI model based on the modified image data using a computing device.
[0035] Clause 2: The image data is captured in real time or previously captured, as described in Clause 1.
[0036] Clause 3: The method according to Clause 1 or 2, wherein modifying the captured image data includes removing a subsequence of frames from the captured image data, the method further comprising identifying the probability value of each frame of the captured image data, and identifying the subsequence of frames of the captured image data by comparing the probability value of each frame with a predetermined probability threshold.
[0037] Clause 4: The method according to any one of Clauses 1 to 3, wherein modifying the captured image data includes removing a subsequence of frames from the captured image data, the method further comprising selecting a frame range from the captured image data, generating a random integer having a value greater than or equal to the first frame number of the frame range and less than or equal to the last frame of the frame range plus 1, and determining the frames to be selected for the subsequence of frames based on the random integer.
[0038] Clause 5: The method according to any one of Clauses 1 to 4, wherein modifying the captured image data includes adding one or more new frame subsequences to the captured image data, the method further comprising: randomly generating a time shift of a predetermined length; dividing the captured image data into a plurality of subsequences equal to the predetermined length; generating a random integer for each of the plurality of subsequences; and spatially shifting at least one of the width and height of at least one frame of each of the plurality of subsequences based on the random integer corresponding to the subsequence.
[0039] Clause 6: The captured image data includes: ultrasound images, optical coherence tomography (OCT) images, CT images, MRI images, PET images, SPECT images, fluorescence fluoroscopy images, X-ray images, mammography images, tomosynthesis images, photoacoustic images, acousto-optical images, endoscopic images, microscopic images, fundus images, scanning laser ophthalmoscope (SLO) images, smartphone images, 3D (depth) images, focal stack images, brightfield images, visible light images, infrared images, ultraviolet images, thermal images, multispectral images, tomography images, projection images, integrated images, reconstructed images, or The method described in any one of Clauses 1 to 5, wherein the machine learning-based AI model includes at least one of any combination thereof, and the machine learning-based AI model includes at least one of the following: convolutional neural networks, recurrent neural networks, long short-term memory neural networks, Bayesian networks, Bayesian neural networks, transformer networks, autoencoder networks, deep Boltzmann machines, deep belief networks, random forests, Bayesian random forests, gradient-boosted regression trees, or any combination thereof.
[0040] Clause 7: A system for training a machine learning-based artificial intelligence (AI) to handle diverse types of motion occurring during video acquisition, the system comprising at least one computing device, wherein the at least one computing device is programmed or configured to capture image data using an imaging device, thereby generating multiple frames of captured image data, the captured image data including motion between the imaging device and tissue; modify the captured image data, thereby modifying the captured image data by at least one of the following: altering the amount of time between any two images of the captured image data, removing a subsequence of frames from the captured image data, and adding one or more subsequences of new frames to the captured image data, each of the one or more new frames including at least one of the following: a copy of at least one frame of the captured image data, a composite frame created from other frames in the captured image data, an interpolated frame generated to appear between other frames in the captured image data, a frame not included in the captured image data, or any combination thereof, thereby modifying the captured image data to generate modified image data; and training a machine learning-based AI model based on the modified image data.
[0041] Clause 8: The image data is captured in real time or previously captured by the system described in Clause 7.
[0042] Clause 9: The system according to Clause 7 or 8, wherein modifying the captured image data includes removing a subsequence of frames from the captured image data, and the at least one computing device is further configured to identify the probability value of each frame of the captured image data, and to identify the subsequence of frames of the image data by comparing the probability value of each frame with a predetermined probability threshold.
[0043] Clause 10: The system according to any one of Clauses 7 to 9, wherein modifying the captured image data includes removing a subsequence of frames from the captured image data, and the at least one computing device is further configured to: select a frame range from the captured image data; generate a random integer having a value greater than or equal to the first frame number of the frame range and less than or equal to the last frame of the frame range plus 1; and determine, based on the random integer, the frames to be selected for the subsequence of frames.
[0044] Clause 11: The system according to any one of Clauses 7 to 10, wherein modifying the captured image data includes adding one or more new frame subsequences to the captured image data, and the at least one computing device is further configured to randomly generate a time shift magnitude of a predetermined length, divide the captured image data into a plurality of subsequences equal to the predetermined length, generate a random integer for each of the plurality of subsequences, and spatially shift at least one of the width and height of at least one frame of each of the plurality of subsequences based on the random integer corresponding to the subsequence.
[0045] Clause 12: The captured image data includes: ultrasound images, optical coherence tomography (OCT) images, CT images, MRI images, PET images, SPECT images, fluorescence fluoroscopy images, X-ray images, mammography images, tomosynthesis images, photoacoustic images, acousto-optical images, endoscopic images, microscopic images, fundus images, scanning laser ophthalmoscope (SLO) images, smartphone images, 3D (depth) images, focal stack images, brightfield images, visible light images, infrared images, ultraviolet images, thermal images, multispectral images, tomography images, projection images, integrated images, reconstructed images, or The machine learning-based AI model includes at least one of any combination thereof, and the system described in any one of the following: convolutional neural networks, recurrent neural networks, long short-term memory neural networks, Bayesian networks, Bayesian neural networks, transformer networks, autoencoder networks, deep Boltzmann machines, deep belief networks, random forests, Bayesian random forests, gradient-boosted regression trees, or any combination thereof, as described in any one of the systems described in any one of the clauses 7 to 11.
[0046] Clause 13: A computer program product for training machine learning-based artificial intelligence (AI) to handle diverse types of motion occurring during video acquisition, comprising at least one non-temporary computer-readable medium including program instructions, wherein the program instructions, when executed by at least one computing device, instruct the at least one computing device to capture image data using an imaging device, thereby generating multiple frames of captured image data, the captured image data including motion between the imaging device and the tissue, and to modify the captured image data, thereby altering the amount of time between any two images of the captured image data, and subsequencing the frames from the captured image data. A computer program product that modifies the captured image data by at least one of the following: removing frames and adding one or more new frame subsequences, each of which includes at least one of the following: a copy of at least one frame of the captured image data, a composite frame created from other frames in the captured image data, an interpolated frame generated to appear between other frames in the captured image data, a frame not included in the captured image data, or any combination thereof; modifying the captured image data to generate modified image data; modifying; and training a machine learning-based AI model based on the modified image data.
[0047] Clause 14: The image data is captured in real time or previously captured by the computer program product described in Clause 13.
[0048] Clause 15: The computer program product according to Clause 13 or 14, wherein modifying the captured image data includes removing a subsequence of frames from the captured image data, and the program instruction further causes the at least one computing device to identify the probability value of each frame of the captured image data, and to identify the subsequence of frames of the image data by comparing the probability value of each frame with a predetermined probability threshold.
[0049] Clause 16: The computer program product according to any one of Clauses 13 to 15, wherein modifying the captured image data includes removing a subsequence of frames from the captured image data, and the program instruction further causes the at least one computing device to select a range of frames from the captured image data, generate a random integer having a value greater than or equal to the number of the first frames in the range and less than or equal to the last frame of the range plus 1, and determine which frames are selected for the subsequence of frames based on the random integer.
[0050] Clause 17: The computer program product according to any one of Clauses 13 to 16, wherein modifying the captured image data includes adding one or more new frame subsequences to the captured image data, and the program instruction further causes the at least one computing device to randomly generate the magnitude of the time shift of each subsequence of a frame of a predetermined length; to divide the captured image data into a plurality of subsequences equal to the predetermined length; to generate a random integer for each subsequence of the plurality of subsequences; and to spatially shift at least one of the width and height of at least one frame of each subsequence of the plurality of subsequences based on the random integer corresponding to the subsequence.
[0051] Clause 18: The captured image data includes: ultrasound images, optical coherence tomography (OCT) images, CT images, MRI images, PET images, SPECT images, fluorescence fluoroscopy images, X-ray images, mammography images, tomosynthesis images, photoacoustic images, acousto-optical images, endoscopic images, microscopic images, fundus images, scanning laser ophthalmoscope (SLO) images, smartphone images, 3D (depth) images, focal stack images, brightfield images, visible light images, infrared images, ultraviolet images, thermal images, multispectral images, tomography images, projection images, integrated images, reconstructed images, or any of the above. A computer program product as described in any one of Clauses 13 to 17, comprising at least one of the following combinations, wherein the machine learning-based AI model comprises at least one of the following: convolutional neural networks, recurrent neural networks, long short-term memory neural networks, Bayesian networks, Bayesian neural networks, transformer networks, autoencoder networks, deep Boltzmann machines, deep belief networks, random forests, Bayesian random forests, gradient-boosted regression trees, or any combination thereof.
[0052] Clause 19: The method according to any one of Clauses 1 to 6, wherein modifying the captured image data includes generating a copy of the captured image data and modifying the copy of the captured image data to form the modified image data.
[0053] Clause 20: A method for training a machine learning-based artificial intelligence (AI) model to handle diverse types of tissue appearances during medical imaging, comprising: capturing a series of training medical images using an imaging device, the series of training medical images comprising a plurality of frames in a sequence; generating a set of control points before and after each frame of at least a subset of the frames and each classified object in each frame using a computing device; generating a set of deformation control points for each frame of the at least subset of frames using a convolutional neural network based on the set of control points and uncertainty maps using a computing device; applying geometric distortion to each frame of the at least subset of frames based on the set of deformation control points, thereby generating a new training image; and training a machine learning-based AI model based on the new training image using a computing device.
[0054] Clause 21: The method according to Clause 20, wherein the set of deformation control points is generated based on an affine transformation.
[0055] Clause 22: The method according to Clause 20 or 21, wherein the set of deformation control points is generated by automatically assessing which portion of each frame of the at least subset of the frame satisfies the uncertainty threshold.
[0056] Clause 23: Automatically assessing which portion of each frame in at least a subset of the frame satisfies the uncertainty threshold, based on a recognition and / or chance uncertainty map, as described in any one of Clauses 20-22.
[0057] Clause 24: The recognition and / or randomness uncertainty map is the method described in any one of Clauses 20 to 23, output from the machine learning-based AI model during training.
[0058] Clause 25: The machine learning-based AI model being trained includes a Bayesian artificial neural network, as described in any one of Clauses 20-24.
[0059] Clause 26: The method according to any one of Clauses 20 to 25, wherein the set of deformation control points is generated based on the output of an individual agent adversarial augmented artificial neural network configured to process the output of the auto-assessment as input.
[0060] Clause 27: A method for training a machine learning-based artificial intelligence (AI) model to handle diverse types of tissue appearances during medical imaging, comprising: capturing a series of training medical images using an imaging device; extending the training medical images into new training images, wherein different distortions target different parts within individual frames of the series of training medical images and / or images; applying the distortions to each new training image of the new training images; and iteratively calculating the distortions during AI learning, wherein the iterative calculation is based on an automated assessment of which parts of the training images are less visually understandable to the AI compared to parts of the training images that are more visually understandable to the AI at that moment in training, and / or the degree of difficulty of the AI for each of the different parts of the training images at that moment in training.
[0061] Clause 28: The method according to Clause 27, wherein the distortion is geometric and is applied using moving least squares image deformation.
[0062] Clause 29: The method according to Clause 27 or 28, wherein the distortion is calculated based on the output of an independent agent adversarial augmented artificial neural network that processes an assessment of the degree of difficulty in different image regions, as input.
[0063] Clause 30: The method according to any one of Clauses 27 to 29, wherein the augmentation system augments a training medical image to a new training image, and the augmentation system is integrated with one or more data augmentation processes such that the augmentation system spot augments specific regions of the image data during training.
[0064] Clause 31: A system including at least one computing device programmed or configured to perform the method described in any one of Clauses 27-30.
[0065] Clause 32: A computer program product comprising at least one non-temporary computer-readable medium which, when executed by at least one computing device, program instructions causing the at least one computing device to perform the actions described in any one of Clauses 27 to 30.
[0066] Clause 33: A system for training a machine learning-based artificial intelligence (AI) model to handle diverse types of tissue appearances during medical imaging, comprising at least one computing device, the at least one computing device being programmed or configured to capture a series of training medical images using an imaging device, the series of training medical images comprising a plurality of frames in a sequence, to generate a set of control points before and after each frame of at least a subset of the frames and each classified object in each frame, to generate a set of deformation control points for each frame of the at least subset of frames using a convolutional neural network based on the set of control points and uncertainty maps, and to apply geometric distortion to each frame of the at least subset of frames based on the set of deformation control points, thereby generating, applying, and training a machine learning-based AI model on the new training images.
[0067] Clause 34: The system described in Clause 33, wherein the set of deformation control points is generated based on an affine transformation.
[0068] Clause 35: The system according to Clause 33 or 34, wherein the set of deformation control points is generated by automatically assessing which portion of each frame in the at least subset of the frame satisfies an uncertainty threshold.
[0069] Clause 36: Automatically assessing which portion of each frame in at least a subset of the frame satisfies the uncertainty threshold is a system described in any one of Clauses 33-35, based on a recognition and / or chance uncertainty map.
[0070] Clause 37: The recognition and / or randomness uncertainty map is the system described in any one of Clauses 33 to 36, output from the machine learning-based AI model during training.
[0071] Clause 38: The machine learning-based AI model being trained is a system described in any one of Clauses 33 to 37, including a Bayesian artificial neural network.
[0072] Clause 39: The system according to any one of Clauses 33 to 38, wherein the set of deformation control points is generated based on the output of an individual agent adversarial augmented artificial neural network configured to process automated assessment as input.
[0073] Clause 40: A computer program product for training a machine learning-based artificial intelligence (AI) model to handle diverse types of tissue appearances during medical imaging, comprising at least one non-temporary computer-readable medium including program instructions, wherein, when executed by at least one computing device, the computer program product causes the at least one computing device to capture a series of training medical images using an imaging device, the series of training medical images comprising multiple frames in a sequence, to generate a set of control points before and after each frame of at least a subset of the frames and each classified object in each frame, to generate a set of deformation control points for each frame of the at least subset of frames using a convolutional neural network based on the set of control points and uncertainty maps, and to apply geometric distortion to each frame of the at least subset of frames based on the set of deformation control points, thereby generating, applying, and training a machine learning-based AI model based on the new training images.
[0074] Clause 41: The set of deformed control points described herein is a computer program product described in Clause 40, generated based on an affine transformation.
[0075] Clause 42: The computer program product described in Clause 40 or 41, wherein the set of deformation control points is generated by automatically assessing which portion of each frame of the at least subset of the frame satisfies the uncertainty threshold.
[0076] Clause 43: A computer program product described in any one of Clauses 40-42, based on a recognition and / or chance uncertainty map, that automatically assesses which portion of each frame of at least a subset of the frame satisfies the uncertainty threshold.
[0077] Clause 44: The Recognition and / or Chance Uncertainty Map is a computer program product described in any one of Clauses 40 to 43, which is output from the machine learning-based AI model during training.
[0078] Clause 45: The machine learning-based AI model being trained includes the computer program products described in Clauses 40-44, including Bayesian artificial neural networks.
[0079] Clause 46: The computer program product described in any one of Clauses 40 to 45, wherein the set of deformation control points is generated based on the output of an individual agent adversarial augmented artificial neural network configured to process the output of an automated assessment as input.
[0080] Clause 47: A system for training a machine learning-based artificial intelligence (AI) model to handle diverse types and / or tissue appearances during medical imaging by acquiring new training data during training, comprising: a robot configured to autonomously acquire training image data of a subject; and at least one computing device communicating with the robot, wherein the at least one computing device is configured to assess the image quality of the training image data in real time when the training image data is acquired by the robot; to create at least one map of the training image data based on the image quality of the training image data; and to coordinate the acquisition of new training image data by the robot based on the at least one map of the training image data.
[0081] Clause 48: The system described in Clause 47, which assesses the image quality, including identifying the mean estimate and the uncertainty estimate of the mean estimate.
[0082] Clause 49: Identifying the mean estimate and the uncertainty estimate is to be done using a system based on a Gaussian process or covariance model, as described in Clause 47 or 48.
[0083] Clause 50: Adjusting the acquisition of new training image data is the system described in any one of Clauses 47-49, which includes optimizing at least one of the following image quality parameters: best mean, confidence limit, expected improvement, or any combination thereof.
[0084] Clause 51: The system according to any one of Clauses 47-50, wherein the at least one computing device is further configured to determine the mean estimate and the uncertainty estimate based on a combination of recognition and randomness uncertainty outputs from a Bayesian AI model under training.
[0085] Clause 52: The Organization shall have a system described in any one of Clauses 47 to 51, which is autonomously and physically operated by the robot or a second device to alter the appearance of the captured image while the training image data is being acquired.
[0086] Clause 53: The system according to any one of Clauses 47 to 52, wherein the at least one computing device is further configured to train a machine learning-based AI model based on the new training image data, and the robot or the second device is configured to operate the organization while the machine learning-based AI model is running.
[0087] Clause 54: The system described in any one of Clauses 47-53, wherein the tissue is physically manipulated to appear as tissue captured in the new training image data.
[0088] Clause 55: The system according to any one of Clauses 47 to 54, wherein the at least one computing device is further configured to incrementally modify one or more aspects of the training image data to improve the image quality.
[0089] Clause 56: The system described in any one of Clauses 47 to 55, wherein the at least one computing device includes a first AI system and a second AI system, the first AI system including a student AI system, and the second AI system including a teacher AI system configured to adjust the student AI system based on a set of intended actions.
[0090] Clause 57: The training image data is represented in a latent space of independent vectors, each latent vector representing one mode of change in the training image data, and the acquisition of new training image data or the expansion of the training image data is performed in any one of the systems described in Clauses 47 to 56, based on the latent vector representation.
[0091] Clause 58: The system according to any one of Clauses 47 to 57, wherein the at least one computing device is further configured to create or infer the independent latent vectors based on the unraveled data representation.
[0092] Clause 59: The training image data includes: ultrasound images, optical coherence tomography (OCT) images, CT images, MRI images, PET images, SPECT images, fluorescence fluoroscopy images, X-ray images, mammography images, tomosynthesis images, photoacoustic images, acousto-optics images, endoscopic images, microscopic images, fundus images, scanning laser ophthalmoscope (SLO) images, smartphone images, 3D (depth) images, focal stack images, brightfield images, visible light images, infrared images, ultraviolet images, thermal images, multispectral images, tomography images, projection images, integrated images, reconstructed images, or any of the above. A system as described in any one of Clauses 47 to 58, comprising at least one of any combination thereof, wherein the at least one computing device comprises: a convolutional neural network, a recurrent neural network, a long short-term memory neural network, a Bayesian network, a Bayesian neural network, a transformer network, an autoencoder network, a deep Boltzmann machine, a deep belief network, a random forest, a Bayesian random forest, a gradient boost machine, a gradient boost regression tree, or at least one of any combination thereof.
[0093] Clause 60: The system according to any one of Clauses 47 to 59, wherein the at least one computing device comprises a first AI system and a second AI system, the first AI system configured to assess the quality of the training image data in real time as the training image data is acquired by the robot and to create the at least one map of the training image data based on the quality of the training image data, and the second AI system configured to coordinate the acquisition of the new training image data based on the at least one map of the training image data.
[0094] Clause 61: A method for training a machine learning-based artificial intelligence (AI) model to handle diverse types and / or tissue appearances during medical imaging by acquiring new training data during training, the method comprising: using a computing device to autonomously acquire training image data of a target using a robot; using a computing device to assess the image quality of the training image data in real time as the training image data is acquired by the robot; using a computing device to create at least one map of the training image data based on the image quality of the training image data; and using a computing device to coordinate the acquisition of new training image data by the robot based on the at least one map of the training image data.
[0095] Clause 62: The method according to Clause 61, wherein assessing the image quality includes identifying the mean estimate and the uncertainty estimate of the mean estimate.
[0096] Clause 63: Identifying the mean estimate and the uncertainty estimate is to be done in accordance with the method described in Clause 61 or 62, based on a Gaussian process or a covariance model.
[0097] Clause 64: Adjusting the acquisition of new training image data is the method described in any one of Clauses 61-63, including optimizing at least one of the following image quality parameters: best mean, confidence limit, expected improvement, or any combination thereof.
[0098] Clause 65: The method of any one of Clauses 61 to 64, further comprising identifying the mean estimate and the uncertainty estimate based on a combination of recognition and randomness uncertainty outputs from a Bayesian AI model during training.
[0099] Clause 66: The method according to any one of Clauses 61 to 65, wherein the organization is autonomously and physically operated by the robot or a second device to change the appearance of the captured image while acquiring the training image data.
[0100] Clause 67: The method according to any one of Clauses 61 to 66, further comprising training a machine learning-based AI model based on the new training image data, wherein the robot or the second device is configured to operate the organization while the machine learning-based AI model is running.
[0101] Clause 68: The method according to any one of Clauses 61 to 67, wherein the tissue is physically manipulated to appear as tissue captured in the new training image data.
[0102] Clause 69: The method according to any one of Clauses 61 to 68, further comprising incrementally modifying one or more aspects of the training image data to improve the image quality.
[0103] Clause 70: The training image data is represented in a latent space of independent vectors, each latent vector representing one mode of change in the training image data, and the acquisition of new training image data or the expansion of the training image data is carried out according to any one of Clauses 61 to 69, based on the latent vector representation.
[0104] Clause 71: The method according to any one of Clauses 61 to 70, wherein the at least one computing device is further configured to create or infer the independent latent vectors based on the unraveled data representation.
[0105] Clause 72: The training image data includes: ultrasound images, optical coherence tomography (OCT) images, CT images, MRI images, PET images, SPECT images, fluorescence fluoroscopy images, X-ray images, mammography images, tomosynthesis images, photoacoustic images, acousto-optical images, endoscopic images, microscopic images, fundus images, scanning laser ophthalmoscope (SLO) images, smartphone images, 3D (depth) images, focal stack images, brightfield images, visible light images, infrared images, ultraviolet images, thermal images, multispectral images, tomography images, projection images, integrated images, reconstructed images, or The method according to any one of the following clauses 61 to 71, wherein the computing device includes at least one of any combination thereof, and the computing device includes at least one of the following: a convolutional neural network, a recurrent neural network, a long short-term memory neural network, a Bayesian network, a Bayesian neural network, a transformer network, an autoencoder network, a deep Boltzmann machine, a deep belief network, a random forest, a Bayesian random forest, a gradient boost machine, a gradient boost regression tree, or any combination thereof.
[0106] Clause 73: A computer program product for training a machine learning-based artificial intelligence (AI) model to handle diverse types and / or tissue appearances during medical imaging by acquiring new training data during training, comprising at least one non-temporary computer-readable medium including program instructions, wherein, when executed by at least one computing device, the program instructions cause the at least one computing device to: autonomously acquire training image data of a target using a robot; assess the image quality of the training image data in real time as the training image data is acquired by the robot; create at least one map of the training image data based on the image quality of the training image data; and coordinate the acquisition of new training image data by the robot based on the at least one map of the training image data.
[0107] Clause 74: The computer program product described in Clause 73, which includes assessing the image quality by identifying the mean estimate and the uncertainty estimate of the mean estimate.
[0108] Clause 75: The computer program product described in Clause 73 or 74, which identifies the mean estimate and the uncertainty estimate, based on a Gaussian process or covariance model.
[0109] Clause 76: Adjusting the acquisition of new training image data includes optimizing at least one of the following image quality parameters: best mean, confidence limit, expected improvement, or any combination thereof, as described in any one of Clauses 73 to 75 of the Computer Program Product.
[0110] Clause 77: The computer program product according to any one of Clauses 73 to 76, wherein the program instructions cause the at least one computing device to further determine the mean estimate and the uncertainty estimate based on a combination of recognition and randomness uncertainty outputs from a Bayesian AI model being trained.
[0111] Clause 78: The Organization shall use a computer program product described in any one of Clauses 73 to 77, which is autonomously and physically operated by the robot or a second device to alter the appearance of the captured image while acquiring the training image data.
[0112] Clause 79: The computer program product according to any one of Clauses 73 to 78, wherein the program instructions cause the at least one computing device to further train a machine learning-based AI model based on the new training image data, and the robot or the second device is configured to operate the organization while the machine learning-based AI model is running.
[0113] Clause 80: The computer program product described in any one of Clauses 73 to 79, which is physically manipulated to appear as an organization captured in the new training image data.
[0114] Clause 81: The computer program product according to any one of Clauses 73 to 80, wherein the program instructions further cause the at least one computing device to incrementally modify one or more aspects of the training image data to improve the image quality.
[0115] Clause 82: The training image data is represented in a latent space of independent vectors, each latent vector representing one mode of change in the training image data, and the acquisition of new training image data or the expansion of the training image data is performed by a computer program product as described in any one of Clauses 73 to 81, based on the latent vector representation.
[0116] Clause 83: The computer program product according to any one of Clauses 73 to 82, wherein the program instructions cause the at least one computing device to further create or infer the independent latent vectors based on the unraveled data representation.
[0117] Clause 84: The training image data includes: ultrasound images, optical coherence tomography (OCT) images, CT images, MRI images, PET images, SPECT images, fluoroscopy images, X-ray images, mammography images, tomosynthesis images, photoacoustic images, acousto-optics images, endoscopic images, microscopic images, fundus images, scanning laser ophthalmoscope (SLO) images, smartphone images, 3D (depth) images, focal stack images, brightfield images, visible light images, infrared images, ultraviolet images, thermal images, multispectral images, tomography images, projection images, integrated images, reconstructed images, or any combination thereof. A computer program product as described in any one of Clauses 73 to 83, comprising at least one of the following, wherein the at least one computing device comprises at least one of the following: convolutional neural networks, recurrent neural networks, long short-term memory neural networks, Bayesian networks, Bayesian neural networks, transformer networks, autoencoder networks, deep Boltzmann machines, deep belief networks, random forests, Bayesian random forests, gradient boosted machines, gradient boosted regression trees, or any combination thereof.
[0118] Clause 85: A method for training a machine learning-based artificial intelligence (AI) model to handle diverse types of changes occurring during image acquisition, comprising: capturing image data using an imaging device, thereby generating captured image data of multiple frames, the captured image data including movement between the imaging device and tissue; identifying at least one latent vector representation; inputting the at least one latent vector representation and the captured image data into a variation autoencoder model; decoding the at least one latent vector representation and the captured image data into adversarial image data using the variation autoencoder model; and training a machine learning-based AI model on the adversarial image data.
[0119] Clause 86: The method according to Clause 85, wherein each vector dimension of the latent vector representation represents one degree of change of the target domain.
[0120] Clause 87: Each latent vector representation is independent of each other latent vector representation, as described in Clause 85 or 86.
[0121] Clause 88: A system for training a machine learning-based artificial intelligence (AI) model to handle diverse types of changes occurring during image acquisition, comprising at least one computing device, the at least one computing device being programmed or configured to capture image data using an imaging device, thereby generating captured image data of multiple frames, the captured image data including movement between the imaging device and tissue, to identify at least one latent vector representation, to input the at least one latent vector representation and the captured image data into a variation autoencoder model, to decode the at least one latent vector representation and the captured image data into adversarial image data using the variation autoencoder model, and to train a machine learning-based AI model based on the adversarial image data.
[0122] Clause 89: The system described in Clause 88, in which each vector dimension of the latent vector representation represents one degree of change of the target domain.
[0123] Clause 90: Each latent vector representation is independent of each other latent vector representation in the system described in Clause 88 or 89.
[0124] Clause 91: A computer program product for training a machine learning-based artificial intelligence (AI) model to handle diverse types of changes occurring during image acquisition, comprising at least one non-temporary computer-readable medium including program instructions, wherein, when executed by at least one computing device, the computer program product causes the at least one computing device to: capture image data using an imaging device, thereby generating captured image data of multiple frames, the captured image data including movement between the imaging device and tissue; identify at least one latent vector representation; input the at least one latent vector representation and the captured image data into a variation autoencoder model; decode the at least one latent vector representation and the captured image data into adversarial image data using the variation autoencoder model; and train a machine learning-based AI model based on the adversarial image data.
[0125] Clause 92: The computer program product described in Clause 91, wherein each vector dimension of the latent vector representation represents one degree of change of the target domain.
[0126] Clause 93: Each latent vector representation is independent of each other latent vector representation in the computer program product described in Clause 91 or 92.
[0127] The features and characteristics of these and other features and properties of this disclosure, as well as the operating methods, functions, and manufacturing economies of the combinations of related elements and components of the structure, will become more apparent from the following description and the examination of the appended claims with reference to the accompanying drawings, all of which form part of this specification, and similar reference numerals indicate corresponding parts in various figures. However, it will be clearly understood that the drawings are for illustrative and explanatory purposes only and are not intended to define the scope of the invention.
[0128] Further advantages and details are described in more detail below with reference to the non-limiting exemplary embodiments shown in the attached figures. [Brief explanation of the drawing]
[0129] [Figure 1] This document describes a system for training machine learning-based artificial intelligence (AI) models in a non-limiting embodiment or manner. [Figure 2] This document describes a system for training machine learning-based AI models using spatial image transformation, in a non-limiting embodiment or manner. [Figure 3] This document describes a system for training machine learning-based AI models using adversarial image data modification, in a non-limiting embodiment or manner. [Figure 4] This document describes a system for training a machine learning-based AI model using active data modification in conjunction with robot control, in a non-limiting embodiment or manner. [Figure 5] This document describes a method for training a machine learning-based AI model according to a non-limiting embodiment or aspect. [Figure 6] This is a flowchart illustrating a method for training a machine learning-based AI model to handle various types of motion that occur during image acquisition, in a non-limiting embodiment or manner. [Figure 7] This is a flowchart illustrating a method for training a machine learning-based AI model to handle diverse types of tissue appearances during medical imaging, according to a non-limiting embodiment or aspect. [Figure 8] This is a flowchart illustrating a method for training a machine learning-based AI model to handle diverse types and / or tissue appearances during medical imaging by acquiring new training image data during training, according to a non-limiting embodiment or aspect. [Figure 9] This is a flowchart illustrating a method for training a machine learning-based AI model to handle various types of changes that occur during medical image acquisition, in a non-limiting embodiment or manner. [Figure 10] This is a schematic diagram of a computing device according to a non-limiting embodiment or aspect. [Modes for carrying out the invention]
[0130] It should be understood that embodiments may take various alternative modifications and step sequences unless expressly specified otherwise. It should also be understood that certain devices and processes described in the following specification are merely exemplary embodiments or aspects of the disclosure. Therefore, certain dimensions and other physical characteristics relating to embodiments or aspects disclosed herein should not be considered limiting. As used herein, aspects, components, elements, structures, operations, steps, functions, instructions, etc., should not be construed as such unless expressly stated to be extremely important or essential. Furthermore, as used herein, the articles "a" and "an" are intended to include one or more items and can be used synonymously with "one or more" and "at least one". Furthermore, as used herein, terms such as "has," "have," and "having" are intended to be open-ended words. In addition, the phrase "based on" is intended to mean "at least partially based" unless expressly specified otherwise.
[0131] As used herein, the term “computational device” may refer to one or more electronic devices configured to process data. In some examples, a computing device may include components necessary for receiving, processing, and outputting data, such as a processor, display, memory, input devices, and network interfaces. A computing device may be a central processing unit (CPU), a graphics processing unit (GPU), etc. A computing device may be a mobile device. A computing device may be a desktop computer or other form of non-mobile computer. In non-limiting embodiments or aspects, a computing device may include an AI accelerator, including an application-specific integrated circuit (ASIC) neural engine such as Apple’s “Neural Engine” or Google’s Tensor Processing Unit. In non-limiting embodiments or aspects, a computing device may consist of a plurality of individual circuits representing each connection in a neural network, such that each circuit is configured to weight the input from each node in the neural network. In such a configuration, logic gates and / or analog circuits may be used without requiring software, a processor, or memory.
[0132] Non-limiting embodiments or aspects provide systems and methods for training machine learning-based artificial intelligence (AI) models using ultrasound image data. Non-limiting embodiments or aspects use image segmentation and data modification methodologies to randomly modify (e.g., expand, alter, edit, distort, etc.) medical images, such as ultrasound images, without limitation, and train a deep learning model (e.g., convolutional neural network (CNN), Bayesian neural network (BNN), etc.) to generalize to various domains across multiple images using the modified images as training data. Non-limiting embodiments enable the AI model to actively learn sequential feature invariance and features generated by different scanning procedures, so that the AI model may be able to generalize features captured in untrained or real-world scenarios. It has been found that AI models have difficulty dealing with temporal features such as fast and inconsistent vascular pulsations and ultrasound scanning during emergency scenarios.
[0133] Non-limiting embodiments or aspects may provide modifications to medical images through probabilistic time data augmentation, and the modified images may be used to train an AI model toward generalization across various domains of medical images. In some non-limiting embodiments or aspects, probabilistic time data augmentation may include probabilistically dropping frames within each sequence of captured image data. For example, each frame of captured image data may be assigned a probability value, and if the probability value assigned to a frame satisfies or exceeds a threshold probability value, that frame may be dropped. Non-limiting embodiments or aspects may modify the captured image data by changing the amount of time between any two frames of the captured image data. Non-limiting embodiments or aspects may modify the captured image data by removing subsequences of frames from the captured image data and creating a generated image sequence that can be used for training. In some non-limiting embodiments or aspects, subsequences of image data may be generated by probabilistically dropping every other frame or randomly selected frames. For example, a sequence of training image data may be generated by selecting random frames from a sequence of captured image data and combining the selected frames into a new sequence. In non-limiting embodiments or aspects, image data may be modified by adding one or more new frames to a sequence of captured image data. In some non-limiting embodiments or aspects, the new frames may include a copy of at least one frame of the captured image data, a composite frame created from other frames in the captured image data, an interpolated frame generated to appear as if it were between other frames, a frame not included in the original captured image data, or any combination of such frames. In non-limiting embodiments or aspects, a sequence of captured images may be divided into multiple subsequences, each subsequence having different spatial and / or temporal shifts applied to the frames in the sequence.Therefore, non-limiting embodiments or aspects provide a method for synthesizing motion across spatial-temporal differences between ultrasonic imaging frames into existing image data.
[0134] Non-limiting embodiments or aspects may adaptively generate spatial deformations that challenge the AI model in areas of its strengths and / or weaknesses. Non-limiting embodiments or aspects may use a dropout-based Bayesian formulation of a 3D U-net encoder-decoder architecture. In some non-limiting embodiments or aspects, a Bayesian temporal segmentation network (BTSN) or other BNN may be used to generate uncertainty maps to identify the model's strengths and / or weaknesses in an image or image data sequence. For example, a BNN may be used to output a recognizability uncertainty map. In some non-limiting embodiments or aspects, the BTSN or BNN may output image segmentation. In some non-limiting embodiments or aspects, the uncertainty map may be input to another artificial neural network (ANN) to adversarially generate geometrically distorted images. An ANN may spatially deform and / or distort an existing image using methods such as moving least-squares image deformation to generate an image having one or more regions of geometric compression, stretching, twisting, shearing, displacement, and / or other geometric distortion. Non-limiting embodiments or aspects may use a CNN or other neural network to generate synthetic and / or modified images including image deformation for training. Non-limiting embodiments may integrate adaptive generation of spatial deformation with other data modification techniques. The described adaptive generation of spatial deformation may be performed in real time or applied to a previously captured image sequence. Thus, in non-limiting embodiments or aspects, an AI model can automatically segment regions of image and anatomical landmarks in ultrasound images and adaptively generate modified images to learn invariance across various image artifacts.
[0135] Non-limiting embodiments or aspects may provide active data modification of ultrasonic image data using robotic control. Non-limiting embodiments or aspects may train a segmentation model online or in real time during scanning and image data acquisition. Captured image data may be used to model locations to search for new data and drive the movement of a robot or robotic system. In some non-limiting embodiments, modeling information about where to search next for additional image data may use a Gaussian process or other relevant processes that represent the mean and uncertainty of the current belief in the captured image data. In some non-limiting embodiments or aspects, the best mean, confidence upper limit, or expected improvement measure may be used. As new image data is acquired, the mean map and uncertainty map are updated. In some non-limiting embodiments or aspects, the mean map and uncertainty map may be used as input to a CNN or other AI model to identify locations to drive the movement of a robotic system to acquire additional image data. In some non-limiting embodiments or aspects, the uncertainty map may be output from a BTSN or other BNN. Non-limiting embodiments or aspects provide a robot or robot system that includes robot control that can learn based on captured image data to manipulate the object of image capture so that the robot system represents images previously captured in the model's training domain or captures image data outside the model's training domain. In some non-limiting embodiments or aspects, if the AI model's path deviates from a desired course, the teacher AI model can be used to correct the AI model driving the robot system's movement. Thus, in non-limiting embodiments or aspects, the AI model can be trained in real time while capturing image data and training the robot control system to help improve the AI model training process.
[0136] In non-limiting embodiments or aspects, an AI model may be trained by modifying captured image data to generate adversarial image data or by generating adversarial image data for training. In some non-limiting embodiments or aspects, adversarial image data may be generated based on an independent latent vector identified based on a disassembled data representation and captured image data. The independent latent vector may represent one degree of change in the image data or any other property related to the captured image data. In some non-limiting embodiments, the latent vector representation may be used to modify captured image data into adversarial images. The modified adversarial images can then be used as input for training an AI model so that it can be used in real-world scenarios. In some non-limiting embodiments, this provides the generation of new training image data for use in model training without the need to capture image data in real-world scenarios. The desired training domain may be specified by setting the latent vector representation.
[0137] Non-limiting embodiments or aspects may be implemented as a software application capable of processing medical image data output by a medical imaging device. In other non-limiting embodiments, the system for generalizing ultrasound data may be directly incorporated into the ultrasound device as hardware and / or software, or incorporated into another medical imaging device.
[0138] Referring now to Figure 1, a system 1000 generalized over variations in image data according to a non-limiting embodiment or aspect is shown. System 1000 may include a change engine 100, an imaging device 102, captured image data 104, training image data 105, a machine learning (ML) model 106, and a database 108. In a non-limiting embodiment or aspect, the imaging device 102 may be any type of imaging device configured to capture a medical image of a subject, such as an optical coherence tomography (OCT) scanner or an ultrasound scanner. For example, the imaging device 102 may include an ultrasound device configured to physically capture ultrasound waveform data (e.g., RF waveform data). In a non-limiting embodiment or aspect, the imaging device 102 may store (e.g., store, communicate, etc.) only specific data associated with RF waveform data (e.g., amplitude envelope of the RF waveform) that can be used to create a grayscale ultrasound image. For example, raw element-wise RF waveforms can be combined to form a beamforming RF waveform, and the envelope of the beamforming RF waveform can form the basis of a grayscale ultrasound image (for example, for display on a screen). Additionally or alternatively, the imaging device 102 may use frequency components to calculate the Doppler shift and measure velocity (which may be displayed in color). In non-limiting embodiments or aspects, the original RF waveform data may be discarded after specific data (e.g., envelope, Doppler shift, etc.) has been calculated (e.g., derived, identified, etc.). Additionally or alternatively, the imaging device 102 may retain the RF waveform data for further analysis (e.g., saving, analyzing, etc.). In non-limiting embodiments or aspects, the imaging device 102 may include an ultrasound device that captures and retains the RF waveform data (e.g., beamforming RF waveform data, element-wise RF waveform data, any other suitable representation of the RF waveform (e.g., retaining frequency components), etc.). Additionally or alternatively, the RF waveform data may be used in real time for online analysis, or it may be saved for later analysis. In non-limiting embodiments or aspects, the imaging system 102 may include a portable ultrasonic device such as a crystal linear array scanner.For example, the imaging device 102 may include a Clarius L7 portable ultrasound machine. In some non-limiting embodiments or aspects, the imaging device 102 may include an amorphous microelectromechanical system (MEMS)-based and / or multifunctional ultrasound machine. For example, the imaging device 102 may include a Butterfly iQ+ ultrasound machine. In some non-limiting embodiments or aspects, the imaging device 102 may be used to acquire captured image data 104 (e.g., grayscale image frames) from at least one sample. For example, a clinician may use the imaging device 102 to acquire such images from a subject that is a human. Additionally or alternatively, the imaging device 102 may output captured image data 104 which may include at least one ultrasound image or a plurality of ultrasound image frames. In some non-limiting embodiments or aspects, the imaging device 102 may include one or more devices capable of receiving information from and / or communicating information to the modification engine 100 and / or other devices.
[0139] The captured image data 104 may include sequences such as ultrasound image frames and medical image frames. The captured image data 104 may include image frames that differ due to dramatic additions of image settings, the operation of the imaging device 102, anatomical structures, and / or artifacts in medical injury scenarios, etc. The captured image data 104 can be captured by the imaging device 102 and communicated to the modification engine 100 in real time. Alternatively, the captured image data 104 may be stored for later communication to the modification engine 100. In some non-limiting embodiments, the captured image data 104 may be input to the modification engine 100 for processing. The captured image data 104 may be processed by the modification engine 100 modifying the captured image data 104 so that it generates training image data 105. Alternatively, the captured image data 104 may be processed by the modification engine 100 by copying the captured image data 104 or copying individual frames from the captured image data 104 to generate training image data 105.
[0140] The modification engine 100 may include software and / or hardware configured to, for example, receive information from the imaging device 102 (e.g., captured image data), communicate information to the imaging device 102 (e.g., information for controlling the imaging device 102), and process the image data (e.g., modify the image data). For example, the modification engine 100 may be one or more software applications running on one or more computing devices. In non-limiting embodiments or aspects, the modification engine 100 may implement at least one convolutional neural network (e.g., W-Net, U-Net, AU-Net, SegNet, etc.) as described herein. In non-limiting embodiments or aspects, the modification engine 100 may implement at least one Bayesian neural network (BNN) or other types of probabilistic graphical models as described herein. In non-limiting embodiments or aspects, the modification engine 100 may receive captured image data 104 from the imaging device 102 in real time. As an addition or alternative, the modification engine 100 may receive (e.g., retrieve, query, etc.) captured image data 104 (e.g., past captured image data) which may include at least one ultrasound image frame from previously stored captured image data 104, as described herein.
[0141] In some non-limiting embodiments or aspects, the modification engine 100 may process the captured image data 104 to generate training image data 105. The modification engine 100 may generate training image data 105 based on copying some or all of the frames of the captured image data 104 and processing those copied frames. In some non-limiting embodiments, when the modification engine 100 generates training image data 105, an unmodified copy of the captured image data 104 may remain stored. In some non-limiting embodiments, when the modification engine 100 generates training image data 105, the training image data 105 may include a mixture of both the modified and unmodified image data 104. In some non-limiting embodiments or aspects, the modification engine 100 may instruct the imaging device to capture training data 105 based on the captured image data 105. For example, the modification engine 100 may generate an uncertainty map based on the captured image data 105, and the modification engine 100 may use the uncertainty map as input to at least one CNN or BNN to determine whether to collect additional image data in the form of captured image data 104 or in the form of training data 105. In non-limiting embodiments or aspects, the modification engine 100 may drop or remove frames or frame sequences in the captured image data 104 randomly or based on a predefined pattern (e.g., every other frame). The modification engine 100 may do this by directly modifying the captured image data 104 to generate the training image data 105, or the modification engine 100 may generate the training image data 105 by copying the removed frames or sequences into a newly generated image data sequence. Sequences in the captured image data 104 and / or training image data 105 may include subsequences. Additionally or alternatively, the modification engine 100 may modify the captured image data 104 by changing the amount of time between any two images of the captured image data 104. As an addition or alternative, the modification engine 100 may add a subsequence of frames to the captured image data 104.For example, the modification engine 100 may add at least one copy of an existing frame of the captured image data 104 to the captured image data 104. In other non-limiting embodiments or aspects, the modification engine 100 may generate a composite frame created from one or more existing frames of the captured image data 104. In some non-limiting embodiments, the modification engine 100 may generate interpolated frames that appear between existing frames in the captured image data 104. The modification engine 100 may generate new frames not included in the captured image data 104 based on independent factors such as latent vector representations or existing data.
[0142] In some non-limiting embodiments or aspects, the training image data 105 may include a sequence of ultrasound image frames or other medical imaging frames. The training image data 105 may include image frames that have differences as a result of processing by the modification engine 100. The training image data 105 may be generated by the modification engine 100, or may be a modified version of the captured image data 104 created by the modification engine 100. In some non-limiting embodiments or aspects, the training image data 105 may include newly captured image data captured by the imaging device 102. In some non-limiting embodiments or aspects, the training image data 105 may be input to the modification engine 100 for processing. In some non-limiting embodiments or aspects, the training image data 105 may be processed so that the modification engine 100 modifies the training image data 105 to generate uncertainty maps and / or additional training data. The training image data 105 may be input to the ML model 106 for training the ML model 106.
[0143] In some non-limiting embodiments or aspects, the ML model 106 may be separate from the imaging device 102, such as one or more software applications running on one or more computing devices that communicate with the imaging device 102. Alternatively, the ML model 106 may be incorporated into the imaging device 102 (e.g., entirely, partially, etc.) so that the ML model 106 is implemented by the software and / or hardware of the imaging device 102. In some non-limiting embodiments or aspects, the ML model 106 may be incorporated into the modification engine 100 so that the ML model 106 is implemented by the software and / or hardware of the modification engine 100, or the ML model 106 may be separate from the modification engine 100. The ML model 106 may receive training image data 105 as input. The ML model 106 may be integrated with one or more devices capable of receiving information from and / or communicating information to the database 108. In some non-limiting embodiments or aspects, the ML model 106 may receive data from the database 108. The data received from database 108 may include latent vector representations, uncertainty maps, existing image data (e.g., historical ultrasound image data), etc.
[0144] The database 108 may include one or more devices capable of receiving information from and / or communicating information to the ML model 106, etc. In non-limiting embodiments or aspects, the database 108 may store captured image data 104 (e.g., past ultrasound image data) from previous image acquisitions (e.g., by imaging device 102, other ultrasound and / or RF systems, etc.). For example, the (past) captured image data 104 may include at least one ultrasound subimage frame. In non-limiting embodiments or aspects, previously captured image data may be provided to the database 108 and stored as past image data. Additionally or alternatively, such previously captured image data may be used for training and / or testing the ML model 106 (e.g., to determine how accurately the ML model 106 has learned generalized features such as anatomical landmarks, to determine which generalized features are strengths or weaknesses of the ML model 106, etc.) as described herein.
[0145] In non-limiting embodiments or aspects, the imaging device 102, ML model l06, and database 108 may be implemented by a single computing device, a single system, etc. (e.g., entirely, partially, etc.).
[0146] Referring now to Figure 2, a system 2000 for training a machine learning-based AI model using spatial image deformation is shown according to a non-limiting embodiment or aspect. In a non-limiting embodiment or aspect, the system 2000 may include a deformation engine 200, a training system 202, a deformation agent neural network (NN) 204, a Bayesian time segmentation network (BTSN) 206, a training image 208, an uncertainty map 210, and a segmentation map 212. The deformation engine 200 may include a CNN (e.g., W-Net, U-Net, SegNet, or any combination thereof). In some non-limiting embodiments or aspects, the deformation engine 200 may be the same as or similar to the deformation engine 100 described herein. In a non-limiting embodiment or aspect, the deformation engine 200 may generate a synthetic image for further training using image deformation. For example, the deformation engine 200 may generate a synthetic image for training using image deformation with one deformation of the moving least squares deformation method. In some non-limiting embodiments, the modification engine 200 may receive a set of control points and a set of deformation control points from the modification agent NN204. The modification engine 200 may apply the set of control points and the set of deformation points to the input image received from the training system 202 to generate a composite image that includes image deformation.
[0147] As shown in Figure 2, the training system 202 may include and output a training image 208. The training system 202 may include, for example, a computing device and / or a data storage device for storing image data. The training image 208 may be acquired from real-time image capture, or the training image 208 may be a previously captured and stored image. The training system 202 may communicate the training image 208 to the modification engine 200 to modify the training image 208 using image deformations generated by the modification agent NN 204. In other non-limiting embodiments, the training system 202 may communicate the training image 208 to the BTSN 206 for training. Thus, using image deformations, the training image 208 may be spot-modified so that the BTSN 206 is challenged in areas where it may have already learned certain features of the training image 208.
[0148] Referring to Figure 2, in some non-limiting embodiments or aspects, the modification agent NN204 may be a CNN such as a 3D CNN. In some non-limiting embodiments or aspects, the modification agent NN204 may be the same as or similar to the modification engines 100, 200, and / or ML model 106 described herein. The modification agent NN204 may include a plurality of convolutional blocks. The modification agent NN204 may include a 3D convolutional layer, a batch normalization layer, a normalized linear unit (ReLU) activation, a maximum pooling layer, and so on. In some non-limiting embodiments or aspects, the modification agent NN204 may be the same as or similar to the modification engine 100 or ML model 106 described herein. In some non-limiting embodiments or aspects, the modification agent NN204 may be used to generate control points and deformation control points, which are used to modify the training image 208 using image deformation. In some non-limiting embodiments or aspects, the modification agent NN204 may generate a new training image modified with the deformation by creating a copy of the input training image and applying the deformation to the copy of the input training image. In some non-limiting embodiments or aspects, the modification agent NN204 may directly modify the input training image 208 by applying the deformation. In some non-limiting embodiments or aspects, an unmodified copy of the training image 208 may remain in storage. In some non-limiting embodiments, to generate a composite image, the modification agent NN204 may generate a first set of control points before and after the image boundary from a sequence of captured image data. To generate control points and modify the training image 208, the modification agent NN204 may receive an uncertainty map from BTSN206, which can be used to determine and output a set of directions for shifting the control points applied to the input image. Each control point may receive an individual directional shift. Directional shifts may include linear up / down and left / right directional shifts and additional degrees of freedom. Points representing the new location of the control point following the directional shift are included as deformation control points. The modification agent NN204 can generate a second set of control points before and after each anatomical classification detected in the training image.The modification agent NN204 can also generate a set of deformation control points, which are as follows:
number
number
number
number
number
[0149] The modification agent NN204 can be trained based on how much (e.g., up to a threshold or by a percentage of the threshold) it can challenge BTSN206, which is measured by the loss on the modified training image. In some non-limiting embodiments or aspects, the modification agent NN204 can be trained by generating a random set of points and signaling the deformation direction of the control points. The moving least-squares image deformation can be computed for both agent-generated points (e.g., control points and deformation control points) and randomly generated points. The segmentation loss can be computed for both sets. If the agent-generated points lead to a lower segmentation loss, it can be assumed that the randomly generated points are more difficult. The more difficult points are assigned as labels for training the modification agent NN204. However, if the randomly generated points lead to a lower loss, agent-generated points in the opposite direction are assigned as labels for training.
[0150] Referring to Figure 2, according to non-limiting embodiments or aspects, BTSN206 may be the same as or similar to ML Model 106 described herein. BTSN206 may include a dropout-based Bayesian formulation of a 3D U-net encoder-decoder architecture. BTSN206 may include four convolutional blocks on the encoder side, with corresponding pairs on the decoder side. Each block may include an input layer followed by two pairs: a convolutional layer, batch normalization, and ReLU. Each block may further include a single dropout layer before the output layer. BTSN206 may include a predictive mean that can be used for a recognizability uncertainty map.
number
number
number
number
number
number
[0151] In some non-limiting embodiments or aspects, BTSN206 may receive a training image 208 as training input. The training image 208 may be a modified training image or the original training image before modification. In some non-limiting embodiments or aspects, BTSN206 may receive a modified training image from a modification engine 200 as training input. BTSN206 may generate a segmentation map of the training image 208 or other modified training image as an output. BTSN206 may generate an uncertainty map as an output, which may be communicated to the modification agent NN204 for training the modification agent NN204 and for generating control points for image modification.
[0152] In non-limiting embodiments or aspects, the uncertainty map 210 may include a recognition uncertainty map, a randomness uncertainty map, and the like. The uncertainty map 210 may be communicated from the BTSN 206 to the modification agent NN 204. The uncertainty map 210 may be supplied to the modification agent NN 204 as input for training and to generate control points for image deformation.
[0153] Referring now to Figure 3, a system 3000 for training a machine learning-based AI model using adversarial image data modification (e.g., an image modified to deceive the AI model) is shown in some non-limiting embodiments or aspects. In some non-limiting embodiments or aspects, system 3000 may include a variation autoencoder model (MVAM) 300, a deep learning semantic segmentation model (DLSSM) 302, an input medical image 304, an input noise vector 306, an adversarial generated image 308, and a segmentation output 310. System 3000 may use a disentangled data representation in the form of latent vectors, such that each vector dimension may represent one degree of change in the image data. In some non-limiting embodiments or aspects, system 3000 may be used in real time (e.g., concurrently with image acquisition) to train a machine learning-based AI model. In some non-limiting embodiments or aspects, system 3000 may be used to train a machine learning-based AI model using previously acquired image data.
[0154] In some non-limiting embodiments or aspects, MVAM300 may be the same as or similar to the modification engines 100, 200 described herein. In some non-limiting embodiments or aspects, DLSSM302 may be the same as or similar to the ML model 106 described herein. DLSSM302 may include a variation of a deep learning-based semantic segmentation model. The input medical image 304 may be the same as or similar to the captured image data 104 described herein. In some non-limiting embodiments or aspects, the input medical image 304 may include a single image, a sequence of image frames, or raw image data. In some non-limiting embodiments or aspects, the input noise vector 306 may include multiple dimensions. The input noise vector 306 may include dimensions independent of other dimensions. Each dimension of the input noise vector 306 may represent one degree of change in the image data. In some non-limiting embodiments or aspects, the dimensions can be pre-selected to represent a target domain. In some non-limiting embodiments or aspects, the dimensions of the input noise vector 306 can be derived from a distribution. The adversarial generated image 308 may include image data modified using the input noise vector 306. In some non-limiting embodiments, the adversarial generated image 308 may be identical to the input medical image 304 in that no modification has been applied to the adversarial generated image 308, since the dimensions of the input noise vector 306 can be derived from the distribution (and therefore the dimensions may represent a state where no noise is added). The adversarial generated image 308 may include perceptible differences in the image compared to the input medical image 304 modified by MVAM 300.
[0155] According to a non-limiting embodiment, the MVAM 300 may receive an input medical image 304 and an input noise vector 306 as inputs. The MVAM 300 may decode the inputs into an adversarial generated image 308. The input noise vector 306 may independently transform the spatial features of the input medical image 304, such as the shape, size, and imaging settings of anatomical structures. The adversarial generated image may be supplied to the DLSSM 302 as input for training. The DLSSM 302 may output a segmentation output 310, which may become more interpretable as the DLSSM 302 is trained.
[0156] Referring now to Figure 4, a system 4000 for training a machine learning-based AI model using active data modification in conjunction with robot control is shown according to a non-limiting embodiment or aspect. In some non-limiting embodiments or aspects, the system 4000 may include a computing device 400, a robot 402, an imaging device 404, training image data 406, and a database 408. In some non-limiting embodiments or aspects, the computing device 400 may be the same as or similar to the modification engine 100 described herein. In some non-limiting embodiments or aspects, the computing device 400 may be the same as or similar to the ML model 106 described herein. The computing device 400 may include or communicate with an ML model that can be trained. The robot 402 may include a robotic arm, a robotic manipulator, and / or a robotic tool capable of manipulating tissue 410 (e.g., pressing, deforming, pushing, probing, expanding, contracting, etc.). In some non-limiting embodiments or aspects, the robot 402 may include a computing device that controls the movement of the robot 402. In other non-limiting embodiments or aspects, the robot 402 may include a computing device such as a controller that communicates with the computing device 400 so that the computing device 400 can control the robot 402. The imaging device 404 may be the same as or similar to the imaging device 102 described herein. The training image data 406 may be the same as or similar to the captured image data 104 and / or training image data 105 described herein. The database 408 may be the same as or similar to the database 108 described herein. In some non-limiting embodiments or aspects, the tissue 410 may include living or dead tissue of the object being scanned, such as human tissue, animal tissue, or synthetic tissue.
[0157] Continuing to refer to Figure 4, according to non-limiting embodiments or aspects, the computing device 400 may apply a Gaussian process to process and analyze input data such as training image data 406. In some non-limiting embodiments, the computing device 400 may apply another process that may represent the mean, best estimate, uncertainty, and / or covariance of current beliefs about the collected data or training image data 406 being collected. For example, the computing device 400 may apply a best mean, confidence limit, and / or expected improvement measure to the collected data. Beliefs about the training image data 406 or other collected training data (e.g., location data collected by the robot 402 and imaging device 404) may include estimates of desired information, which can be used to further train the computing device 400 about locations to look for new image data to capture and to instruct the robot 402. In some non-limiting embodiments or aspects, the best estimate and the uncertainty of the estimate can be used by the computing device 400 to assess the quality of the training image data. In some non-limiting embodiments or aspects, the best estimate and estimation uncertainty may be represented by a recognition uncertainty map and a chance uncertainty map output from a Bayesian segmentation model.
[0158] In some non-limiting embodiments or aspects, the robot 402 may be configured to autonomously acquire training image data of the tissue 410. In some non-limiting embodiments or aspects, the computing device 400 may be configured to control the robot 402 so that it autonomously acquires training image data.
[0159] In some non-limiting embodiments or aspects, using best estimates and estimation uncertainty, the robot 402 may be trained to manipulate tissue 410 to represent the current training domain of the first ML model of the computing device 400. The manipulation of tissue 410 may be applied so that the tissue 410 resembles what the first ML model of the computing device 400 has previously learned. For example, if the first ML model has learned an artery of a trained size, the robot 402 may be used to shrink or enlarge the artery to the trained size for further data acquisition using the imaging device 404. In some non-limiting embodiments or aspects, the robot 402 may be used to move the imaging device 404 to an area of tissue 410 where the first ML model has not been trained or where additional training is required. In some non-limiting embodiments, the system 4000 or computing device 400 may include a teacher ML model. The teacher ML model may be able to correct the first ML model if the first ML model propagates undesirable errors and causes the system to exceed an error threshold. The errors may include errors detected by the first ML model or the teacher ML model in training image data 406 or other collected training data.
[0160] In non-limiting embodiments or aspects, the computing device 400 may create a map of the collected training image data based on the image quality of the training image data. Image quality may include the overall image quality of the training image data 406. Image quality may include the interpretability of the training image data 406. In some non-limiting embodiments or aspects, the computing device 400 may coordinate the acquisition of new training image data 406. Coordination of the acquisition of training image data 406 may include any of the described methods, such as instructing the robot 402 to manipulate the tissue 410 based on what the first ML model has previously learned. Coordination of the acquisition of training image data 406 may be based on whether the first ML model is in a weak training domain or where the first ML model has not yet been trained. Coordination of acquisition may be based on the best estimate and uncertainty of the estimate of the desired information to be collected for the commands and / or training of the teacher ML model.
[0161] Referring hereto to Figure 5, an exemplary artificial neural network 500 (e.g., a U-net CNN architecture) according to a non-limiting embodiment or aspect is shown. The artificial neural network 500 may include a recurrent neural network, a feedforward neural network, a long short-term memory neural network, a Bayesian network, a transformer network, an autoencoder network, a deep Boltzmann machine, a deep belief network, a random forest, a Bayesian random forest, a gradient-boosted regression tree, a segmentation network, a semantic segmentation network, a Bayesian segmentation network, or any combination thereof. The artificial neural network 500 may include a 3D U-net architecture, batch normalization, and ReLU activation, and may include a downsampling layer as the last layer of the encoder block. Each decoder block may include two pairs of convolutional layers, a batch normalization layer, a convolutional LSTM layer, and an upsampling layer followed by ReLU activation in the encoder block. Each convolutional layer may include a 3×3×3 kernel dimension. The artificial neural network 500 may be implemented (e.g., entirely, partially, etc.) by the ML model 106, the modification engine 200, the modification agent NN204, BTSN206, DLSSM302, and / or the computing device 400. Additionally or alternatively, the artificial neural network 500 may be implemented (e.g., entirely, partially, etc.) by at least one other computing device and / or by direct implementation in a circuit (e.g., digital and / or analog) separate from or including the ML model 106, the modification engine 200, the modification agent NN204, BTSN206, DLSSM302, the computing device 400, and / or the robot 402.
[0162] In non-limiting embodiments or aspects, the artificial neural network 500 may be similar to the CNN described in Ronneberger et al., "U-net: Convolutional networks for biomedical image segmentation," International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 234-241 (2015).
[0163] Referring now to Figure 6, a flowchart of a method for training a machine learning-based AI model to handle various types of motion that occur during image acquisition, according to a non-limiting embodiment or aspect, is shown. In some non-limiting embodiments or aspects, one or more steps of the method may be performed (e.g., entirely, partially, etc.) by the change engine 100, change engine 200, change agent NN204, BTSN206, MVAM300, DLSSM302, and computing device 400. When referring to the change engine or AI model, it can be understood that the steps described in the non-limiting embodiments herein may be performed by any one of the change engine 100, change engine 200, change agent NN204, BTSN206, MVAM300, DLSSM302, and / or computing device 400. In some non-limiting embodiments or aspects, one or more steps of the method may be performed (e.g., entirely, partially, etc.) separately from the modification engine 100, computing device 400, etc., such as the ML model 106, or by another system, another device, another group of systems, or another group of devices that include them.
[0164] As shown in Figure 6, in step 600, the method may include capturing image data as described herein. For example, imaging devices 102, 404 may capture image data to generate captured image data for multiple frames. The captured image data may include image frames relative to other image frames based on the movement between the imaging devices 102, 404 and the tissue 410.
[0165] As shown in Figure 6, in step 602, the method may include modifying image data as described herein. For example, the modification engine 100 may modify captured image data (e.g., image data 104, 406, etc.). By modifying the captured image data, modified image data (e.g., training image data 105, 208, 308, etc.) may be generated. In non-limiting embodiments or aspects, modifying captured image data may include changing the amount of time between any two frames of the captured image data. Changing the amount of time between any two frames of the captured image data may include adding one or more frames between any two images of the captured image data. The one or more frames may be duplicate frames copied or deleted from the captured image data, or they may be new frames. Alternatively, the one or more frames may include frames generated by any method described herein. In non-limiting embodiments or aspects, modifying captured image data may include removing a subsequence of frames from the captured image data to create a generated image sequence that can be used for training. In some non-limiting embodiments or aspects, subsequences of image data may be generated by probabilistically dropping every other frame or randomly selected frames. For example, a training sequence may be generated by randomly selecting frames from the capture image data sequence and combining the selected frames into a new sequence. In some non-limiting embodiments, modifying the capture image data may involve adding one or more new subsequences of frames to the capture image data sequence. In some non-limiting embodiments or aspects, the new frames may include copies of at least one frame of the capture image data, composite frames created from other frames in the capture image data, interpolated frames generated to appear as if they were between other frames, frames not included in the original capture image data, or any combination of such frames.
[0166] As shown in Figure 6, in step 604, the method may include training a model based on modified image data, as described herein. For example, the ML model 106 (or any other AI model described herein) may be trained using the modification engine 100 by receiving modified image data (e.g., training image data 105) from the modification engine 100 as input to the ML model 106.
[0167] Referring now to Figure 7, a flowchart of a method for training a machine learning-based AI model to handle various types of tissue appearances during medical imaging, according to a non-limiting embodiment or aspect, is shown. In some non-limiting embodiments or aspects, one or more steps of the method may be performed (e.g., entirely, partially, etc.) by the change engine 100, imaging device 102, change engine 200, change agent NN204, BTSN206, MVAM300, DLSSM302, and computing device 400. When referring to the change engine or AI model, it can be understood that the steps described in the non-limiting embodiments herein may be performed by any one of the change engine 100, change engine 200, change agent NN204, BTSN206, MVAM300, DLSSM302, and / or computing device 400. In some non-limiting embodiments or aspects, one or more steps of the method may be performed (e.g., entirely, partially, etc.) separately from the modification engine 100, computing device 400, etc., such as the ML model 106, or by another system, another device, another group of systems, or another group of devices that include them.
[0168] As shown in Figure 7, in step 700, the method may include capturing image data as described herein. For example, imaging devices 102, 404 may capture image data as a series of training medical images. The series of images may include multiple image frames in sequence. The captured image data may include image frames relative to other image frames based on the movement between the imaging devices 102, 404 and the tissue 410.
[0169] As shown in Figure 7, in step 702, the method may include generating a set of control points before and after each frame of the captured image data and before and after each classified object in at least a subset of the frames in the multiple frames captured as described herein. For example, the modification engine 100 may generate a set of control points before and after the boundary of each frame in a subsequence of frames. The number of control points in a set of control points at both the boundary and the classified object may each be set by hyperparameters of the model that generates the set of control points. The set of control points may be used to deform the captured image data to generate training image data. The set of control points may be used to generate an uncertainty map by calculating the segmentation loss for a set of randomly generated points.
[0170] As shown in Figure 7, in step 704, the method may include generating a set of deformation control points before and after each frame of the captured image data and before and after each classified object in at least a subset of the frames in the multiple frames captured as described herein. For example, the modification engine 100 may generate a set of deformation control points that define the deformation of the image frame in each frame in a subsequence of frames. The deformation control points may be generated based on an affine transformation.
[0171] As shown in Figure 7, in step 706, the method may include providing geometric deformations to the frame to generate a new training image. For example, the modification engine 100 may generate a new subsequence of frames, each frame containing a deformed portion of the frame based on a set of control points and a set of deformation control points. In some non-limiting embodiments, the modification engine may generate the training image data 105 by directly applying the distortion to the captured image data 104 so that the captured image data 104 is modified. Applying the distortion to the frame may use a moving least-squares image deformation method. In some non-limiting embodiments or aspects, the distortion may be partially defined by translational values.
[0172] As shown in Figure 7, in step 708, the method may include training the model based on the new training images. For example, the modification engine 100 may send a new subsequence of the deformed frames (e.g., training image data 105) to the ML model 106 as input for training.
[0173] Referring now to Figure 8, a flowchart shows a method for training a machine learning-based AI model to handle diverse types and / or tissue appearances during medical imaging by acquiring new training image data during training, according to a non-limiting embodiment or aspect. In some non-limiting embodiments or aspects, one or more steps of the method may be performed (e.g., entirely, partially, etc.) by the change engine 100, imaging device 102, change engine 200, change agent NN204, BTSN206, MVAM300, DLSSM302, and computing device 400. When referring to the change engine or AI model, it can be understood that the steps described in the non-limiting embodiments herein may be performed by any one of the change engine 100, change engine 200, change agent NN204, BTSN206, MVAM300, DLSSM302, and / or computing device 400. In some non-limiting embodiments or aspects, one or more steps of the method may be performed (e.g., entirely, partially, etc.) separately from the modification engine 100, computing device 400, etc., such as the ML model 106, or by another system, another device, another group of systems, or another group of devices including them. In some non-limiting embodiments or aspects, the method shown in Figure 8 can be repeated iteratively in real time to train the AI model.
[0174] As shown in Figure 8, in step 800, the method may include capturing image data using the robot 402 described herein. For example, the robot 402 may be configured to autonomously capture image data of a target. The robot 402 may communicate with one or more computing devices and one or more AI models. The robot 402 may capture image data by moving the imaging device relative to the target. In some non-limiting embodiments or aspects, the robot 402 may move to capture image data at a location on the target based on the coordination of a computing device communicating with the robot 402. In some non-limiting embodiments or aspects, the robot 402 may capture image data by autonomously and physically manipulating the target to change the appearance of the captured image data and obtain desired training image data.
[0175] As shown in Figure 8, in step 802, the method may include assessing the image quality of the training image data. For example, the image quality may be assessed in real time as the training image data is acquired by the robot 402 and the imaging device. In some non-limiting embodiments or aspects, the image quality may be assessed asynchronously by a computing device (e.g., computing device 400) communicating with the robot 402 after the image data has been captured and / or stored. Assessing the image quality may include assessing the interpretability of the captured image data.
[0176] As shown in Figure 8, in step 804, the method may include creating a map of the training image data. For example, the map may be created based on the image quality of the training image data. In some non-limiting embodiments or aspects, the map may include a mean map or an uncertainty map.
[0177] As shown in Figure 8, in step 806, the method may include coordinating the acquisition of new training image data by the robot. In some non-limiting embodiments or aspects, coordinating the acquisition of new training image data by the robot 402 may include the computing device 400 receiving a map as input. The computing device 400 may use measurements from the map to determine where the robot 402 should drive to acquire new training image data. In some non-limiting embodiments or aspects, the computing device 400 may coordinate the acquisition of new training image data based on what the trained model already knows or has learned previously. In some non-limiting embodiments or aspects, the computing device 400 may coordinate the acquisition of new training image data based on a domain outside the current training domain. A domain outside the current training domain may include any image features that the model has not yet trained on.
[0178] Referring now to Figure 9, a flowchart of a method for training a machine learning-based AI model to handle various types of changes that occur during medical image acquisition, according to a non-limiting embodiment or aspect, is shown. In some non-limiting embodiments or aspects, one or more steps of the method may be performed (e.g., entirely, partially, etc.) by the change engine 100, imaging device 102, change engine 200, change agent NN204, BTSN206, MVAM300, DLSSM302, and computing device 400. When referring to the change engine or AI model, it can be understood that the steps described in the non-limiting embodiments herein may be performed by any one of the change engine 100, change engine 200, change agent NN204, BTSN206, MVAM300, DLSSM302, and / or computing device 400. In some non-limiting embodiments or aspects, one or more steps of the method may be performed (e.g., entirely, partially, etc.) separately from the modification engine 100, computing device 400, etc., such as the ML model 106, or by another system, another device, another group of systems, or another group of devices that include them.
[0179] As shown in Figure 9, in step 920, the method may include capturing image data as described herein. In step 922, the method may include determining at least one latent vector representation. Determining a latent vector representation may include determining a latent vector representation based on a target domain. The target domain can be considered more difficult than what the trained model has already learned. The target domain may include domains outside the current training domain of the model being trained. In some non-limiting embodiments or aspects, the latent vector may be derived from image data using an AI model such as a CNN, which includes an encoder-decoder architecture with a bottleneck layer. The bottleneck layer may generate a feature map when processing the image data, and the feature map may be used as a latent space for deriving the latent vector representation.
[0180] As shown in Figure 9, in step 924, the method may include inputting the latent vector representation and captured image data into a variation autoencoder model (e.g., MVAM300). In some non-limiting embodiments or aspects, the latent vector representation and captured image data may be input as a single image to which the vector representation is associated. In some non-limiting embodiments or aspects, the latent vector representation and captured image data may be input as a sequence of frames of captured image data, along with a sequence of vector representations corresponding to each frame. In some non-limiting embodiments or aspects, the latent vector representation includes individually distinct dimensions that can be independent of variations in other individual dimensions of the latent vector representation.
[0181] As shown in Figure 9, in step 926, the method may include decoding the latent vector representation and the captured input data into adversarial image data using a variation autoencoder model (e.g., MVAM300). For example, MVAM300 may encode the captured image data and apply the dimensions of the latent vector representation to the captured image data. MVAM300 may then decode the captured image data modified by the latent vector representation into new training image data, which can be used as adversarial image data to train an AI model. In some non-limiting embodiments, when the latent vector representation is applied to the captured image data, the various dimensions and / or representations of the latent vector may independently transform different spatial features of the input captured image data.
[0182] As shown in Figure 9, in step 928, the method may include training an AI model (e.g., DLSSM302) based on adversarial image data. For example, a frame sequence of adversarial image data may be input to the AI model to train on areas of captured image data that may have been modified in latent vector representations.
[0183] Referring now to Figure 10, a diagram of example components of device 900 according to a non-limiting embodiment is shown. Device 900 may, for example, correspond to change engines 100, 200, ML model 106, change agent NN204, MVAM300, DLSSM302, computing device 400, and / or robot 402. In some non-limiting embodiments, such a system or device may include at least one device 900 and / or at least one component of device 900. The number and arrangement of components are provided as an example. In some non-limiting embodiments, device 900 may include additional components, fewer components, different components, or components arranged differently compared to those shown in Figure 10. As an addition or alternative, one set of components of device 900 (e.g., one or more components) may perform one or more functions described as being performed by another set of components of device 900.
[0184] As shown in Figure 10, device 900 may include a bus 902, a processor 904, a memory 906, a storage component 908, an input component 910, an output component 912, and a communication interface 914. The bus 902 may include components that enable communication between components of device 900. In some non-limiting embodiments, the processor 904 may be implemented in hardware, firmware, or a combination of hardware and software. For example, the processor 904 may include a processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), etc.), a microprocessor, a digital signal processor (DSP), and / or any processing component that can be programmed to perform a function (e.g., a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), etc.). The memory 906 may include random access memory (RAM), read-only memory (ROM), and / or other types of dynamic or static storage devices (e.g., flash memory, magnetic memory, optical memory, etc.) that store information and / or instructions for use by the processor 904.
[0185] Continuing to refer to Figure 10, the storage component 908 may store information and / or software related to the operation and use of device 900. For example, the storage component 908 may include a hard disk (e.g., magnetic disk, optical disk, magneto-optical disk, solid-state disk, etc.) and / or another type of computer-readable medium. The input component 910 may include components that enable device 900 to receive information via user input (e.g., touchscreen display, keyboard, keypad, mouse, buttons, switches, microphone, etc.). Additionally or alternatively, the input component 910 may include sensors that detect information (e.g., Global Positioning System (GPS) components, accelerometers, gyroscopes, actuators, etc.). The output component 912 may include components that provide output information from device 900 (e.g., display, speaker, one or more light-emitting diodes (LEDs), etc.). The communication interface 914 may include components such as a transceiver (e.g., a transceiver, separate receivers and transmitters, etc.) that enable device 900 to communicate with other devices via a wired connection, a wireless connection, or a combination of wired and wireless connections. The communication interface 914 may enable device 900 to provide information to and / or provide information to other devices. For example, the communication interface 914 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a Universal Serial Bus (USB) interface, a Wi-Fi® interface, a cellular network interface, etc.
[0186] Device 900 may execute one or more processes described herein. Device 900 may execute these processes based on a processor 904 that executes software instructions stored in a computer-readable medium such as memory 906 and / or storage component 908. The computer-readable medium may include any non-temporary memory device. The memory device includes a memory space located inside a single physical storage device or a memory space spread across multiple physical storage devices. Software instructions may be read into memory 906 and / or storage component 908 from another computer-readable medium or another device via a communication interface 914. When executed, the software instructions stored in memory 906 and / or storage component 908 can cause the processor 904 to execute one or more processes described herein. In addition or alternatively, hardwired circuits may be used instead of or in combination with software instructions to execute one or more processes described herein. Thus, the embodiments described herein are not limited to any particular combination of hardware circuits and software. The term “programmed or configured” as used herein refers to the configuration of software, hardware circuits, or any combination thereof in one or more devices.
[0187] While embodiments have been described in detail for illustrative purposes, it should be understood that such details are merely illustrative and that this disclosure is not limited to the embodiments disclosed. On the contrary, it is intended that this disclosure covers modifications and equivalent configurations that fall within the spirit and scope of the appended claims. For example, it should be understood that this disclosure intends, wherever possible, to allow one or more features of any embodiment to be combined with one or more features of any other embodiment.
Claims
1. 1. A method for training a machine learning based artificial intelligence (AI) model to handle various types of motion occurring during image acquisition, comprising: capturing image data using an imaging device, whereby a plurality of frames of captured image data are generated, the captured image data including motion between the imaging device and tissue; Modifying the captured image data using a computing device, modifying the amount of time between any two frames of said captured image data; removing a subsequence of frames from the captured image data; and adding a subsequence of one or more new frames to the captured image data, each of the one or more new frames comprising at least one of a copy of at least one frame of the captured image data, a composite frame created from other frames in the captured image data, an interpolated frame generated to appear between other frames in the captured image data, a frame not included in the captured image data, or any combination thereof; modifying the captured image data by at least one of the steps to generate modified image data; training a machine learning based AI model based on the modified image data using a computing device; The method includes:
2. The method of claim 1 , wherein the image data is captured in real time or previously captured.
3. Modifying the captured image data includes removing a subsequence of frames from the captured image data, the method comprising: determining a probability value for each frame of the captured image data; identifying the subsequence of frames of the captured image data by comparing the probability value for each frame to a predetermined probability threshold; The method of claim 1 further comprising:
4. Modifying the captured image data includes removing a subsequence of frames from the captured image data, the method comprising: selecting a frame range from the captured image data; generating a random integer having a value greater than or equal to the number of a first frame in the range of frames and less than or equal to the number of a last frame in the range of frames plus one; determining frames to be selected for the subsequence of frames based on the random integer; The method of claim 1 further comprising:
5. Modifying the captured image data includes adding a subsequence of one or more new frames to the captured image data, the method comprising: Randomly generating a time shift of a predetermined length; Dividing the captured image data into a plurality of subsequences equal to the predetermined length; generating a random integer for each subsequence of the plurality of subsequences; spatially shifting at least one of a width and a height of at least one frame of each of the plurality of subsequences based on the random integers corresponding to the subsequences; The method of claim 1 further comprising:
6. The captured image data may include: ultrasound images, optical coherence tomography (OCT) images, CT images, MRI images, PET images, SPECT images, fluoroscopy images, X-ray images, mammography images, tomosynthesis images, photoacoustic images, acousto-optical images, endoscopic images, microscopy images, fundus images, scanning laser ophthalmoscopy (SLO) images, smartphone images, 3D (depth) images, focal stack images, bright field images, visible light images, infrared images, ultraviolet images, thermal images, multispectral images, tomography images, projection images, integrated images, and reconstructed images. , or any combination thereof, and the machine learning based AI model comprises at least one of the following: a convolutional neural network, a recurrent neural network, a long short-term memory neural network, a Bayesian network, a Bayesian neural network, a transformer network, an autoencoder network, a deep Boltzmann machine, a deep belief network, a random forest, a Bayesian random forest, a gradient boosted regression tree, or any combination thereof.
7. 1. A system for training a machine learning based artificial intelligence (AI) to handle various types of motions occurring during video capture, the system comprising at least one computing device, the at least one computing device comprising: capturing image data using an imaging device, whereby a plurality of frames of captured image data are generated, the captured image data including motion between the imaging device and tissue; Modifying the captured image data, modifying the amount of time between any two images of said captured image data; removing a subsequence of frames from the captured image data; and adding a subsequence of one or more new frames to the captured image data, each of the one or more new frames comprising at least one of a copy of at least one frame of the captured image data, a composite frame created from other frames in the captured image data, an interpolated frame generated to appear between other frames in the captured image data, a frame not included in the captured image data, or any combination thereof; modifying the captured image data by at least one of the steps to generate modified image data; training a machine learning based AI model based on the modified image data; The system is programmed or configured to:
8. The system of claim 7 , wherein the image data is captured in real time or previously captured.
9. Modifying the captured image data includes removing a subsequence of frames from the captured image data, the at least one computing device being configured to: determining a probability value for each frame of the captured image data; identifying the subsequence of frames of the image data by comparing the probability value for each frame to a predetermined probability threshold; The system of claim 7 , further configured to:
10. Modifying the captured image data includes removing a subsequence of frames from the captured image data, the at least one computing device being configured to: selecting a frame range from the captured image data; The number of frames in the frame range that is greater than or equal to the first frame and the number of frames in the last frame of the frame range that is greater than or equal to 1 generating a random integer having a value less than or equal to determining frames to be selected for the subsequence of frames based on the random integer; The system of claim 7 , further configured to:
11. Modifying the captured image data includes adding a subsequence of one or more new frames to the captured image data, the at least one computing device being: Randomly generating a magnitude of a time shift of a predetermined length; Dividing the captured image data into a plurality of subsequences equal to the predetermined length; generating a random integer for each subsequence of the plurality of subsequences; spatially shifting at least one of a width and a height of at least one frame of each of the plurality of subsequences based on the random integers corresponding to the subsequences; The system of claim 7 , further configured to:
12. The captured image data may include: an ultrasound image, an optical coherence tomography (OCT) image, a CT image, an MRI image, a PET image, a SPECT image, a fluoroscopy image, an X-ray image, a mammography image, a tomosynthesis image, a photoacoustic image, an acousto-optical image, an endoscopic image, a microscopy image, a fundus image, a scanning laser ophthalmoscope (SLO) image, a smartphone image, a 3D (depth) image, a focal stack image, a bright field image, a visible light image, an infrared image, an ultraviolet image, a thermal image, a multispectral image, a tomography image, a projection image, an integrated image, a reconstructed image, 8. The system of claim 7, wherein the machine learning based AI model comprises at least one of the following: a convolutional neural network, a recurrent neural network, a long short-term memory neural network, a Bayesian network, a Bayesian neural network, a transformer network, an autoencoder network, a deep Boltzmann machine, a deep belief network, a random forest, a Bayesian random forest, a gradient boosted regression tree, or any combination thereof,
13. 1. A computer program product for training a machine learning based artificial intelligence (AI) to handle various types of motions occurring during video capture, comprising: at least one non-transitory computer readable medium comprising program instructions, the program instructions, when executed by at least one computing device, causing the at least one computing device to: capturing image data using an imaging device, whereby a plurality of frames of captured image data are generated, the captured image data including motion between the imaging device and tissue; Modifying the captured image data, modifying the amount of time between any two images of said captured image data; removing a subsequence of frames from the captured image data; and adding a subsequence of one or more new frames to the captured image data, each of the one or more new frames comprising at least one of a copy of at least one frame of the captured image data, a composite frame created from other frames in the captured image data, an interpolated frame generated to appear between other frames in the captured image data, a frame not included in the captured image data, or any combination thereof; modifying the captured image data by at least one of the steps to generate modified image data; training a machine learning based AI model based on the modified image data; A computer program product that causes
14. The computer program product of claim 13 , wherein the image data is captured in real time or previously captured.
15. Modifying the captured image data includes removing a subsequence of frames from the captured image data, and the program instructions cause the at least one computing device to: determining a probability value for each frame of the captured image data; identifying the subsequence of frames of the image data by comparing the probability value for each frame to a predetermined probability threshold; 14. The computer program product of claim 13, further comprising:
16. Modifying the captured image data includes removing a subsequence of frames from the captured image data, and the program instructions cause the at least one computing device to: selecting a frame range from the captured image data; generating a random integer having a value greater than or equal to the first frame number of the range of frames and less than or equal to the last frame number of the range of frames plus one; determining frames to be selected for the subsequence of frames based on the random integer; 14. The computer program product of claim 13, further comprising:
17. Modifying the captured image data includes adding a subsequence of one or more new frames to the captured image data, and the program instructions cause the at least one computing device to: randomly generating a time shift magnitude for each subsequence of frames of a predetermined length; Dividing the captured image data into a plurality of subsequences equal to the predetermined length; generating a random integer for each subsequence of the plurality of subsequences; spatially shifting at least one of a width and a height of at least one frame of each of the plurality of subsequences based on the random integers corresponding to the subsequences; 14. The computer program product of claim 13, further comprising:
18. The captured image data may be an ultrasound image, an optical coherence tomography (OCT) image, a CT image, an MRI image, a PET image, a SPECT image, a fluoroscopy image, an X-ray image, a mammography image, a tomosynthesis image, a photoacoustic image, an acousto-optical image, an endoscopic image, a microscopy image, a fundus image, a scanning laser ophthalmoscope (SLO) image, a smartphone image, a 3D (depth) image, a focal stack image, a bright field image, a visible light image, an infrared image, an ultraviolet image, a thermal image, a multispectral image, a tomography image, a projection image, an integrated image, a reconstructed image, or any of the above.
14. The computer program product of claim 13, wherein the machine learning based AI model comprises at least one of the following: a convolutional neural network, a recurrent neural network, a long short-term memory neural network, a Bayesian network, a Bayesian neural network, a transformer network, an autoencoder network, a deep Boltzmann machine, a deep belief network, a random forest, a Bayesian random forest, a gradient boosted regression tree, or any combination thereof.
19. Modifying the captured image data includes generating a copy of the captured image data and modifying the copy of the captured image data to form the modified image data. The method of claim 1 , comprising:
