Systems and methods for enhancing ultrasound images using artificial intelligence

Abstract

The present disclosure provides systems and methods for generating an enhanced image or video of a fetus. An exemplary system includes a computer processing device comprising a processor and a non-volatile storage medium with instructions for the processor to access sonographic data that depicts a fetus, and generate, using a trained machine learning model, as output, at least one of an enhanced image or an enhanced video of the fetus based on the sonographic data. The at least one of the enhanced image or the enhanced video includes a lifelike depiction of the fetus.

Description

SYSTEMS AND METHODS FOR ENHANCING ULTRASOUND IMAGES USINGARTIFICIAL INTELLIGENCECROSS-REFERENCE

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 579,253, filed August 28, 2023, which is hereby incorporated by reference in its entirety for all purposes.BACKGROUND

[0002] Ultrasound imaging is commonly used for aiding disease diagnosis and treatment, and a common use case is the non-invasive observation of fetuses during pregnancy. However, ultrasound images often have low contrast, therefore details in darker regions are often not perceptible. Moreover, ultrasound images and videos often lack clarity and visual appeal.SUMMARY

[0003] Aspects of the present disclosure provides systems and methods for generating an enhanced image or video of a fetus. In one aspect, a method for generating an enhanced image or video of a fetus is disclosed. The method comprises: accessing sonographic data that depicts a fetus; and generating, using a trained machine learning model, as output, at least one of an enhanced image or an enhanced video of the fetus based on the sonographic data, wherein the at least one of the enhanced image or the enhanced video includes a lifelike depiction of the fetus.

[0004] In some embodiments, the method comprises: accessing sonographic data that depicts a fetus, wherein the sonographic data is stored in a user device; transmitting the sonographic data to a server or a computing device implemented with a trained machine learning model; receiving, from the trained machine learning model, at least one of an enhanced image or an enhanced video of the fetus based on the sonographic data, wherein the at least one of the enhanced image or the enhanced video includes a lifelike depiction of the fetus; and displaying the at least one of the enhanced image or the enhanced video on a display on or coupled to the user device.

[0005] Another aspect is a computer-implemented method of constructing a machine learning model for generating an enhanced image or video of a fetus. The method comprises: accessing training data comprising at least one of an enhanced image or an enhanced video of a fetus, wherein the at least one of the enhanced image or the enhanced video includes a lifelike depiction of the fetus; and training a machine learning model that runs on one or more processors coupled to memory on the training data, by progressively matching output of themachine learning model with a ground truth label of the at least one of the enhanced image or the enhanced video.

[0006] Another aspect is a method for enhancing fetal ultrasound imagery to make the image more realistic. The method comprises: receiving an ultrasound image of a fetus and non-fetal tissue; removing ultrasound artifacts associated with the non-fetal tissue from the ultrasound image to generate a first modified image; applying a smoothing algorithm to the ultrasound image to reduce noise and enhance clarity of the fetus to generate a second modified image; applying an enhancement algorithm to enhance the second modified image of the fetus to generate a final image, wherein the final image comprises a fetus without non-fetal tissue; outputting the enhanced image of the fetus.

[0007] In some embodiments, the removing ultrasound artifacts associated with the non-fetal tissue comprises: identifying the non-fetal tissue from the ultrasound image using machine learning; removing the identified non-fetal tissue from the ultrasound image; and generating image data to fill in a gap created by the removal of the identified non-fetal tissue using stable diffusion to generate the ultrasound image without non-fetal tissue.

[0008] In some embodiments, the smoothing algorithm comprises: detecting uneven surfaces in the ultrasound image; and modifying the images to remove the uneven surfaces to output a second modified image.

[0009] In some embodiments, applying the enhancement algorithm comprises: applying a modified stable diffusion algorithm to the second modified image to output the final image.

[0010] In some embodiments, the sonographic data comprises one or more sonographic images of the fetus.

[0011] In some embodiments, the one or more sonographic images comprise one or more of two-dimensional (2D) or three-dimensional (3D) images.

[0012] In some embodiments, the sonographic data comprises sonographic images directly transmitted from an ultrasound imaging device.

[0013] In some embodiments, the sonographic data comprises one or more sonographic videos of the fetus.

[0014] In some embodiments, the method further comprises removing personal information from the sonographic data, wherein the personal information comprises personal health information (PHI) of the fetus and information associated health service to the fetus.

[0015] In some embodiments, the machine learning model comprises a convolutional neural network.

[0016] In some embodiments, the convolutional neural network comprises one or more of convolution layers, pooling layers, or fully connected layers.

[0017] In some embodiments, the neural network is trained on a dataset of fetal images to improve the accuracy of artifact removal and image smoothing.

[0018] In some embodiments, the smoothing algorithm employs machine learning techniques to distinguish between fetal tissue and non-fetal artifacts.

[0019] In some embodiments, the non-fetal artifacts comprises at least one of a placenta or an umbilical cord.

[0020] Another aspect is a non-transitory computer readable medium comprising computerexecutable instructions that, when executed by the processor, causes the processor to perform any one of the described methods of enhancing fetal ultrasound imagery.

[0021] Another aspect is a system for enhancing fetal ultrasound imagery, the system comprising: a one or more processors; and a non-transitory computer readable medium comprising computer-executable instructions that, when executed by the processor, causes the processor to perform any one of the described methods of enhancing fetal ultrasound imagery.

[0022] Another aspect is a system for enhancing fetal ultrasound imagery, the system comprising: an ultrasound imaging device for capturing an ultrasound image of a fetus; one or more memories comprising computer-executable instructions; and one or more processor connected to the one or more memory and configured to execute the computer-executable instructions that comprise any one of the described methods of enhancing fetal ultrasound imagery.

[0023] Additional aspects and advantages of the present disclosure will become readily apparent to those skilled in this art from the following detailed description, wherein only illustrative embodiments of the present disclosure are shown and described. As will be realized, the present disclosure is capable of other and different embodiments, and its several details are capable of modifications in various obvious respects, all without departing from the disclosure.Accordingly, the drawings and description are to be regarded as illustrative in nature, and not as restrictive.BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The novel features of the present disclosure are set forth with particularity in the appended claims. A better understanding of the features and advantages of the present disclosure will be obtained by reference to the following detailed description that sets forth illustrative embodiments, in which the principles of the present disclosure are utilized, and the accompanying drawings (also “Figure” and “FIG.” herein), of which:

[0025] FIG. 1 A is a block diagram of a non-limiting example of machine learning model that processes sonographic data of a fetus as input and generate an enhanced image or video, according to some embodiments of the present disclosure;

[0026] FIGS. IB, 1C, ID, and IE are examples of sonographic data as input to a machine learning model and enhanced images as output, according to some embodiments of the present disclosure;

[0027] FIG. 2 is a block diagram of a convolutional neural network for generating an enhanced sonographic image or video that depicts a fetus, according to some embodiments of the present disclosure;

[0028] FIG. 3 is a block diagram of training a machine learning model, according to some embodiments of the present disclosure;

[0029] FIG. 4 is a workflow diagram of generating an enhanced sonographic image or video that depicts a fetus using a machine learning model, according to some embodiments of the present disclosure;

[0030] FIG. 5 is a dataflow diagram of generating an enhanced sonographic image or video that depicts a fetus using a machine learning model, according to some embodiments of the present disclosure; and

[0031] FIG. 6 shows a computer system that is programmed or otherwise configured to implement methods provided herein.DETAILED DESCRIPTION

[0032] While various embodiments of the present disclosure have been shown and described herein, it will be obvious to those skilled in the art that such embodiments are provided by way of example only. Numerous variations, changes, and substitutions may occur to those skilled in the art without departing from the present disclosure. It should be understood that various alternatives to the embodiments of the present disclosure described herein may be employed.

[0033] Whenever the term “at least,” “greater than,” or “greater than or equal to” precedes the first numerical value in a series of two or more numerical values, the term “at least,” “greater than” or “greater than or equal to” applies to each of the numerical values in that series of numerical values. For example, greater than or equal to 1, 2, or 3 is equivalent to greater than or equal to 1, greater than or equal to 2, or greater than or equal to 3.

[0034] Whenever the term “no more than,” “less than,” or “less than or equal to” precedes the first numerical value in a series of two or more numerical values, the term “no more than,” “less than,” or “less than or equal to” applies to each of the numerical values in that series ofnumerical values. For example, less than or equal to 3, 2, or 1 is equivalent to less than or equal to 3, less than or equal to 2, or less than or equal to 1.

[0035] Certain inventive embodiments herein contemplate numerical ranges. When ranges are present, the ranges include the range endpoints. Additionally, every sub range and value within the range is present as if explicitly written out. The term “about” or “approximately” may mean within an acceptable error range for the particular value, which will depend in part on how the value is measured or determined, e.g., the limitations of the measurement system. For example, “about” may mean within 1 or more than 1 standard deviation, per the practice in the art. Alternatively, “about” may mean a range of up to 20%, up to 10%, up to 5%, or up to 1% of a given value. Where particular values are described in the application and claims, unless otherwise stated the term “about” meaning within an acceptable error range for the particular value may be assumed.Overview

[0036] A method for generating an enhanced image or video of a fetus is described herein. In some embodiments, the method may comprise accessing sonographic data that depicts a fetus, and generating, using a trained machine learning model, as output, at least one of an enhanced image or an enhanced video of the fetus based on the sonographic data, where the at least one of the enhanced image or the enhanced video includes a lifelike depiction of the fetus.

[0037] A method for generating an enhanced image or video of a fetus is described herein. In some embodiments, the method may comprise accessing sonographic data that depicts a fetus, where the sonographic data is stored in a user device, transmitting the sonographic data to a server or a computing device implemented with a trained machine learning model, receiving, from the trained machine learning model, at least one of an enhanced image or an enhanced video of the fetus based on the sonographic data, wherein the at least one of the enhanced image or the enhanced video includes a lifelike depiction of the fetus, and displaying the at least one of the enhanced image or the enhanced video on a display coupled to the user device.

[0038] For example, a parent may have in their smartphone or tablet computer a sonogram image or video of their baby. The sonogram image or video may be blurry or unclear, and the parent may wish to have the image cleaned up. The parent may provide the sonogram to a server via any conventional means of transferring electronic data (e.g., email, text, USB, browser, FTP, etc.) to ultimately provide a server that is trained on the methods described herein. The sonogram image may be modified to remove unnecessary features (e.g., non-fetal tissue, noise, blurring, etc.), smoothed, and enhanced and transferred back to the parent. The parent may then be able to see an enhanced image of the parent’s fetus in the womb.

[0039] A method for constructing a machine learning model for generating an enhanced image or video of a fetus. In some embodiments, the method may comprise accessing training data comprising at least one of an enhanced image or an enhanced video of a fetus, where the at least one of the enhanced image or the enhanced video includes a lifelike depiction of the fetus, and training a machine learning model that runs on one or more processors coupled to memory on the training data, by progressively matching output of the machine learning model with a ground truth label of the at least one of the enhanced image or the enhanced video. Training methods and systems are described in greater detail herein.

[0040] FIG. 1 A is a block diagram of a non-limiting example of machine learning model that processes sonographic data of a fetus as input and generate an enhanced image or video, according to some embodiments of the present disclosure. As illustrated, a trained machine learning model 120 accesses sonographic data 110 of a fetus and generates an enhanced image or video of the fetus 130. The enhanced image or video 130 of the fetus may be a lifelike depiction of the fetus.

[0041] The machine learning model 120 may include one or more machine learning models that are trained and configured to do different things. For example, one machine learning model of machine learning model 120 may be trained to identify and remove non-anatomical features. A second machine learning model of machine learning model 120 may be trained to identify and remove non-fetal tissue such as placenta and umbilical cord. In some embodiments, a third machine learning model of machine learning model 120 may be trained to apply a smoothing algorithm to an ultrasound image or a modified ultrasound image. In some embodiments, a fourth machine learning model of machine learning model 120 may include a generative artificial intelligence (Al) model such as a stable diffusion model or a modified version of a stable diffusion model that is configured to provide enhanced images that better represent anatomical features such as face, eyes, nose, lips, chin, forehead, fingers, ears, and others. In some embodiments, the first through fourth machine learning models may be configured to act sequentially (e.g., output of first model is input to second model, output of second model is input to third model, etc.). The sequence of models may be changed depending on embodiments (e.g., smoothing may be performed before non-anatomical features are removed, etc.). In some embodiments, the first through fourth machine learning models may be combined into one model that is configured to perform each of the functions altogether. Furthermore, additional machine learning models may be added to the machine learning model 120 to provide cleaner and better ultrasound images.

[0042] FIGS. IB, 1C, ID, and IE are examples of sonographic data as input to a machine learning model and enhanced images as output, according to some embodiments of the presentdisclosure. FIGS. IB and ID are input sonographic data generated from an ultrasound imaging device. In some embodiments, non-fetal tissue may be removed, a smoothing algorithm applied, and noise from the original sonogram data may be removed to generate an enhanced image. As illustrated, the face of the fetus depicted in these images can be blurred and / or distorted. The machine learning model 120 may process these images and generate corresponding enhanced images and / or videos, as illustrated in FIGS. 1C and IE. The facial features of the fetuses FIGS. 1C and IE are clear and smooth.

[0043] In some embodiments, the enhancement of sonographic data may comprise one or more of upscaling and / or downscaling of resolution, or adjustment in lighting, tone, contrast, and / or color. In other embodiments, the enhancement of sonographic data may comprise removing imperfections, adding and / or remove details, or smoothing. In other embodiments, the enhancement of sonographic data may comprise outpainting. Outpainting may use visual data provided to expand the scene. For example, outpainting may extend an original image and create a large-scaled image therefrom in any aspect ratio.Methods and Systems for Enhancing Image or Video of a Fetus

[0044] In some embodiments, a method for enhancing fetal ultrasound imagery to make the image more realistic is disclosed. A method for enhancing fetal ultrasound imagery to make the image more realistic, the method comprising: receiving an ultrasound image of a fetus and non- fetal tissue; removing ultrasound artifacts associated with the non-fetal tissue from the ultrasound image to generate a first modified image; applying a smoothing algorithm to the ultrasound image to reduce noise and enhance clarity of the fetus to generate a second modified image; applying an enhancement algorithm to enhance the second modified image of the fetus to generate a final image, wherein the final image comprises a fetus without non-fetal tissue; outputting the enhanced image of the fetus.

[0045] In some embodiments, the enhancement algorithm works by training it using samples of enhanced imagery. The enhanced imagery used to train the model was fetal ultrasound imagery enhanced with an image editing tool such as Adobe® Photoshop®. A Stable Diffusion model may be used to produce certain anatomical features like the face, arms, hands, feet and torso.

[0046] A sonographic image is very different than enhancing a photograph. A sonographic image far blurrier, has one color with shading, and uneven surfaces whereas a typical photograph may not. Sonogram images of a fetus may also include floating placenta and umbilical cord that obscure the core image.

[0047] Various steps of the method for enhancing an ultrasound image or video of a fetus is now described. The steps provided herein is meant to be exemplary and not necessarily applied for allembodiments. One of ordinary skill will recognize that various steps may be included or omitted, depending on the embodiments.

[0048] Non-Anatomical Feature Exclusion

[0049] In some embodiments, the removing ultrasound artifacts associated with the non-fetal tissue comprises: identifying the non-fetal tissue from the ultrasound image using machine learning; removing the identified non-fetal tissue from the ultrasound image; and generating image data to fill in a gap created by the removal of the identified non-fetal tissue using stable diffusion to generate the ultrasound image without non-fetal tissue.

[0050] In some embodiments, imagery not related to human anatomy may be removed. One or more machine learning models may be used to identify aspects of the image that are not related to the human anatomy, such as noise. In some embodiments, a convolutional neural network (CNN) or different architecture may be used to identify and exclude non-anatomical features. Accordingly, an input to the machine learning model may include raw ultrasound images and an output may include a modified image that has the non-atomical features removed. Various parameters may be configured to optimize the machine learning model. For example, the number of layers, filter sizes, stride, padding, and activation functions may be modified, optimized, and fine-tuned via a training process to optimize the algorithm to recognize the non- anatomical features of the image.

[0051] Smoothing Algorithm

[0052] In some embodiments, the smoothing algorithm comprises: detecting uneven surfaces in the first modified image; and modifying the images to remove the uneven surfaces to output a second modified image.

[0053] In some embodiments, a smoothing algorithm may be applied to the modified image. The smoothing algorithm may refine the modified image further by detecting uneven surfaces and smoothing the uneven surfaces in the ultrasound images. In some embodiments, a machine learning model may be used that includes a CNN architecture or any suitable smoothing algorithm that is configured to identify features in a ultrasound image or video and smooth the edges in the image or video. Various parameters may be configured to optimize the machine learning model. For example, the number of layers, filter sizes, stride, padding, activation functions, kernel sizes for smoothing filters, the degree of smoothing applied, and other related hyperparameters may be modified and fine-tuned via a training process to optimize the algorithm to recognize the non-anatomical features of the image.

[0054] Image Enhancement Using Modified Stable Diffusion

[0055] In some embodiments, applying the enhancement algorithm comprises: applying a modified stable diffusion algorithm to the second modified image to output the final image.

[0056] In some embodiments, after the smoothing algorithm is applied, a generative Al model may be used to enhance the cleaned, smoothed images, making them more similar to the training set images, which focus on anatomical features. In embodiments, the generative Al model may include a modified Stable Diffusion model that is modified for medical imaging. The generative Al model may receive the image that has been output by the smoothing algorithm and output an enhanced image that better represent anatomical features. Various parameters may be configured to optimize the machine learning model. For example, the number of layers, filter sizes, stride, padding, activation functions, number of diffusion steps, the noise schedule, and specific modifications made to the diffusion model to enhance anatomical features and other related hyperparameters may be modified and fine-tuned via a training process to optimize the algorithm to recognize the non-anatomical features of the image.Machine Learning Model for Identifying Fetal Anatomy

[0057] In some embodiments, a machine learning model may be used to distinguish between fetal anatomy and non-fetal anatomy features such as the placenta and umbilical cord. After the fetal and non-fetal anatomy is identified, the image may be enhanced by removing the non-fetal features from the image.

[0058] Training the machine learning model to identify the fetal anatomy may be performed as follows. The objective during training is for the algorithm to accurately differentiate between fetal anatomy and surrounding non-fetal features (placenta, umbilical cord). First, the algorithm may be trained on a diverse dataset of ultrasound images that includes labeled examples of fetal anatomy, placenta, and umbilical cord structures. The training data set may include high-quality ultrasound images that have been annotated by experts to clearly identify fetal features and non- fetal artifacts. The model may be trained to maximize the inclusion of fetal anatomy in the final output while minimizing or entirely excluding non-fetal features. In some embodiments, the training process may employ supervised learning techniques, where the algorithm leams from the labeled data through iterative adjustments of its internal parameters to optimize its performance.

[0059] In some embodiments, the machine learning model may deploy a convolutional neural network (CNN) architecture. The CNN may be configured with layers designed to extract features at various levels of abstraction from the input ultrasound images. In some embodiments, some layers may be added to the CNN to focus on anatomical details, ensuring that the model can effectively differentiate between fetal anatomy and other structures. In some embodiments, the model may use feature extraction techniques to identify and categorize different parts of the ultrasound image, focusing on features that are characteristic of fetal anatomy.

[0060] In some embodiments, various parameters may be used. In some embodiments, a dynamic learning rate may be employed to adjust the algorithm's learning pace during training. In some embodiments, a batch size may be selected to ensure that the model generalizes well without overfitting. In some embodiments, a number of training epochs may be optimized to balance between adequate learning and overfitting. In some embodiments, techniques such as dropout and L2 regularization may be applied to prevent overfitting and improve the model's robustness.

[0061] In some embodiments, various hyperparameters may be deployed. For example, various filter sizes may be used in the convolutional layers to capture different features at multiple scales. In some embodiments, the depth of the network (i.e., the number of convolutional layers) may be chosen to provide the necessary complexity for accurate feature detection. In some embodiments, a ReLU (Rectified Linear Unit) may be used as the activation function to introduce non-linearity and improve model performance. In some embodiments, in postprocessing (after the primary ML model generates the output), additional post-processing steps may be employed to further refine the exclusion of non-fetal features.Input to Machine Learning Model

[0062] In some embodiments, sonographic data may comprise sonographic images of a fetus. The images may be two-dimensional (2D) and / or three-dimensional (3D) images obtained from an ultrasound imaging device. The 2D images may be black-and-white images that depict the fetus, where the bones of the fetus are highlighted in white. The 3D images may be yellow / tan color images that depict specific features of the fetus in a well-defined formation. However, embodiments are not limited thereto, and the 2D and 3D images may include any other color schemes for identifying the bones and other features. Non-limiting examples of the size of sonographic images may comprise 1920 x 1080 pixels, 1280 x 872 pixels, 1280 x 720 pixels, 640 x 480 pixels, 852 x 480 pixels, or any other resolutions that the machine learning model may be able to process. Non-limiting examples of the format of sonographic images may comprise “.jpg”, “.jpeg”, “.gif’, “.tiff’, “ psd”, “.raw”, and others. In other embodiments, sonographic images may be printed on ultrasound thermal papers.

[0063] In other embodiments, sonographic data may comprise sonographic videos or cine clips of the fetus obtained from a four-dimensional (4D) ultrasound imaging device. Non-limiting examples of the resolution of videos or cine clips may comprise 1920 x 1080 pixels, 1280 x 872 pixels, 1280 x 720 pixels, 640 x 480 pixels, 852 x 480 pixels, or any other resolutions that the machine learning model may be able to process. Non-limiting examples of the format of sonographic images may comprise “ avi”, “.mp4”, “.mpeg”, “ mkv”, “.dem”, “ mov”, “.wmv”,“ flv”, “ ts”, and others. In some embodiments, videos or cine clips may be converted to a series of static images for the machine learning model to process. In other embodiments, sonographic data may comprise livestream of sonographic videos. The machine learning model may process sonographic data directly transmitted from an ultrasound imaging device and generate a livestream of enhanced videos available for the expectant mother to watch in real time while she is under ultrasound examination.

[0064] The sonographic data, including sonographic images and videos, may be directly transmitted from an ultrasound imaging device. The machine learning model may be implemented in an ultrasound imaging device such that the real-time sonographic images and videos obtain from the device may be used as input to the machine learning model for generating enhanced images and / or videos. The model may generate enhanced images and / or videos during ultrasound examination of an expectant mother. Alternatively, sonographic data may be stored in an imaging archiving system (e.g., picture archiving and communication system, PACS), web-based or cloud-based storage system. When the machine learning model accesses sonographic data, the model may generate enhanced images and / or videos.

[0065] The sonographic data may depict a fetus from an early gestational age to delivery of the newborn. In some embodiments, the sonographic data may depict a fetus from pregnancy weeks 8-41, 9-40, 10-39, 11-38, 12-37, 13-36, and the like.

[0066] The sonographic data may be pre-processed or augmented. In some embodiments, sections of images and videos that include personal information (e.g., personal health information (PHI) of expectant mother and fetus) may be removed or masked. In other embodiments, information associated health service to the fetus may be removed or masked, including information of the hospital and personnel who provided health care, and device information (e.g., device settings, scales, resolutions). The images and videos may be resized (e.g., downscaling, upscaling) preserving the original aspect of ratio or cropped. Other preprocessing steps may comprise perturbations to brightness, contrast, and color. The preprocessing may be performed by the machine learning model or other computing devices.Example Machine Learning Techniques

[0067] In some embodiments, the systems, the methods, the computer-readable media, and the techniques disclosed herein may use various machine learning techniques. In some embodiments, machine learning may generally involve identifying and recognizing patterns in existing data in order to facilitate the transformation of existing data. Machine learning may include a machine learning model (which may include, for example, a machine learning algorithm). Machine learning, whether analytical or statistical in nature, may provide deductiveor abductive inference based on real or simulated data. The machine learning model may be a trained model. Machine learning techniques may comprise one or more supervised, semisupervised, self-supervised, or unsupervised machine learning techniques. For example, a machine learning model may be a trained model that is trained through supervised learning (e.g., various parameters are determined as weights or scaling factors). Machine learning may comprise one or more of regression analysis, regularization, classification, dimensionality reduction, ensemble learning, meta learning, association rule learning, cluster analysis, anomaly detection, deep learning, or ultra-deep learning. Machine learning may comprise, but is not limited to: k-means, k-means clustering, k-nearest neighbors, learning vector quantization, linear regression, non-linear regression, least squares regression, partial least squares regression, logistic regression, stepwise regression, multivariate adaptive regression splines, ridge regression, principal component regression, least absolute shrinkage and selection operation (LASSO), least angle regression, canonical correlation analysis, factor analysis, independent component analysis, linear discriminant analysis, multidimensional scaling, non-negative matrix factorization, principal components analysis, principal coordinates analysis, projection pursuit, Sammon mapping, t-distributed stochastic neighbor embedding, AdaBoosting, boosting, gradient boosting, bootstrap aggregation, ensemble averaging, decision trees, conditional decision trees, boosted decision trees, gradient boosted decision trees, random forests, stacked generalization, Bayesian networks, Bayesian belief networks, naive Bayes, Gaussian naive Bayes, multinomial naive Bayes, hidden Markov models, hierarchical hidden Markov models, support vector machines, encoders, decoders, auto-encoders, stacked auto-encoders, perceptrons, multi-layer perceptrons, artificial neural networks, feedforward neural networks, convolutional neural networks, recurrent neural networks, long short-term memory, deep belief networks, deep Boltzmann machines, deep convolutional neural networks, deep recurrent neural networks, generative adversarial networks, vision transformers, long short-term memory networks (LSTM), masked autoencoders, etc.

[0068] Training the machine learning model may include, in some embodiments, selecting one or more untrained data models to train using a training data set. The selected untrained data models may include any type of untrained machine learning models for supervised, semisupervised, self-supervised, or unsupervised machine learning. The selected untrained data models may be specified based upon input (e.g., user input) specifying relevant parameters to use as predicted variables or other variables to use as potential explanatory variables. For example, the selected untrained data models may be specified to generate an output based upon the input. Conditions for training the machine learning model from the selected untrained data models may likewise be selected, such as limits on the machine learning model complexity orlimits on the machine learning model refinement past a certain point. The machine learning model may be trained (e.g., via a computer system such as a server) using a training data set. In some embodiments, a first subset of the training data set may be selected to train the machine learning model. The selected untrained data models may then be trained on the first subset of training data set using appropriate machine learning techniques, based upon the type of machine learning model selected and any conditions specified for training the machine learning model. In some embodiments, due to the processing power requirements of training the machine learning model, the selected untrained data models may be trained using additional computing resources (e.g., cloud computing resources). Such training may continue, in some embodiments, until at least one aspect of the machine learning model is validated and meets selection criteria to be used as a predictive model.

[0069] In some embodiments, one or more aspects of the machine learning model may be validated using a second subset of the training data set (e.g., distinct from the first subset of the training data set) to determine accuracy and robustness of the machine learning model. Such validation may include applying the machine learning model to the second subset of the training data set to make predictions derived from the second subset of the training data. The machine learning model may then be evaluated to determine whether performance is sufficient based upon the derived predictions. The sufficiency criteria applied to the machine learning model may vary depending upon the size of the training data set available for training, the performance of previous iterations of trained models, or user-specified performance requirements. If the machine learning model does not achieve sufficient performance, additional training may be performed. Additional training may include refinement of the machine learning model or retraining on a different first subset of the training dataset, after which the new machine learning model may again be validated and assessed. When the machine learning model has achieved sufficient performance, in some embodiments, the machine learning model may be stored for present or future use. The machine learning model may be stored as sets of parameter values or weights for analysis of further input (e.g., further relevant parameters to use as further predicted variables, further explanatory variables, further user interaction data, etc.), which may also include analysis logic or indications of model validity in some instances. In some embodiments, a plurality of machine learning models may be stored for generating predictions under different sets of input data conditions. In some embodiments, the machine learning model may be stored in a database (e.g., associated with a server).

[0070] The systems, the methods, the computer-readable media, and the techniques disclosed herein may implement one or more computer vision techniques. Computer vision is a field of artificial intelligence that uses computers to interpret and understand the visual world at least inpart by processing one or more digital images from cameras and videos. In some instances, computer vision may use deep learning models (e.g., convolutional neural networks). Bounding boxes and tracking techniques may be used in object detection techniques within computer vision.

[0071] The systems, the methods, the computer-readable media, and the techniques disclosed herein may implement one or more deep-learning techniques. Deep learning is an example of machine learning that may be based on a set of algorithms that attempt to model high-level abstractions in data by using multiple processing layers, with complex structures or otherwise, composed of multiple non-linear transformations. In some embodiments, a drop out method can be used to reduce overfitting. At each training stage, individual nodes are either “dropped out” of the net (e.g., ignored) with probability 1-p or kept with probability p, so that a reduced network is left; incoming and outgoing edges to a dropped-out node may also be removed. In some embodiments, the reduced network is trained on the data in that stage. The removed nodes may then be reinserted into the network with their original weights.

[0072] The systems, the methods, the computer-readable media, and the techniques disclosed herein may implement one or more vision transformer (ViT) techniques. A ViT is a transformerlike model that handles vision processing tasks. While convolutional neural networks (CNNs) use convolution, a “local” operation bounded to a small neighborhood of an image, ViTs use self-attention, a “global” operation, since the ViT draws information from the whole image. This allows the ViT to capture distant semantic relevance in an image effectively. Advantageously, ViTs may be well-suited catching long-term dependencies. In some embodiments, ViTs may be a competitive alternative to convolutional neural networks as ViTs may outperform the current state-of-the-art CNNs by almost four times in terms of computational efficiency and accuracy. ViTs may be well-suited to object detection, image segmentation, image classification, and action recognition. Moreover, ViTs may be applied in generative modeling and multi-model tasks, including visual grounding, visual-question answering, and visual reasoning. In some embodiments, ViTs may represent images as sequences, and class labels for the image are predicted, which allows models to leam image structure independently. Input images may be treated as a sequence of patches where every patch is flattened into a single vector by concatenating the channels of all pixels in a patch and then linearly projecting it to the desired input dimension. For example, a ViT architecture may include the following operations: (A) split an image into patches; (B) flatten the patches; (C) generate lower-dimensional linear embeddings from the flattened patches; (D) add positional embeddings; (E) provide the sequence as an input to a standard transformer encoder; (F) pretrain a model with image labels (e.g., fully supervised on a huge dataset); and (G) finetune on the downstream dataset for imageclassification. In some embodiments, there may be multiple blocks in a ViT encoder, with each block comprising three major processing elements: (1) Layer Norm; (2) Multi-head Attention Network; and (3) Multi-Layer Perceptrons. The Layer Norm may keep the training process on track and enable the model to adapt to the variations among the training images. The Multi-head Attention Network may be a network responsible for generating attention maps from the given embedded visual tokens. These attention maps may help the network focus on the most critical regions in the image, such as object(s). The Multi-Layer Perceptrons may be a two-layer classification network with a Gaussian Error Linear Unit at the end. The final Multi-Layer Perceptrons block may be used as an output of the transformer. An application of sofimax on this output can provide classification labels (e.g., if the application is image classification).

[0073] The systems, the methods, the computer-readable media, and the techniques disclosed herein may implement one or more masked autoencoder (MAE) techniques. MAEs are scalable self-supervised learners for computer vision. The MAE leverages the success of autoencoders for various imaging and natural language processing tasks. Some computer vision models may be trained using supervised learning, such as using humans to look at images and created labels for the images, so that the model could learn the patterns of those labels (e.g., a human annotator would assign a class label to an image or draw bounding boxes around objects in the image). In contrast, self-supervised learning may not use any human-created labels. One technique for selfsupervised image processing training using an MAE is for before an image is input into an encoder transformer, a certain set of masks are applied to the image. Due to the masks, pixels are removed from the image and therefore the model is provided an incomplete image. At a high level, the model’s task is to now leam what the full, original image looked like before the mask was applied.

[0074] In other words, MAE may include masking random patches of an input image and reconstructing the missing pixels. The MAE may be based on two core designs. First, an asymmetric encoder-decoder architecture, with an encoder that operates on the visible subset of patches (without mask tokens), along with a lightweight decoder that reconstructs the original image from the latent representation and mask tokens. Second, masking a high proportion of the input image, e.g., 75%, may yield a nontrivial and meaningful self-supervisory task. Coupling these two core designs enables training large models efficiently and effectively, thereby accelerating training (e.g., by 3* or more) and improving accuracy. MAE techniques may be scalable, enabling learning of high-capacity models that generalize well, e.g., a vanilla ViT- Huge model. As mentioned, the MAE may be effective in pre-training ViTs for natural image analysis. In some embodiments, the MAE uses the characteristic of redundancy of image information to observe partial images to reconstruct original images as a proxy task, and theencoder of the MAE may have the capability of deducing the content of the masked image area by aggregating context information. This contextual aggregation capability may be important in the field of image processing and analysis.

[0075] The systems, the methods, the computer-readable media, and the techniques disclosed herein may implement one or more decision tree or random forest techniques. A decision tree may be a supervised machine learning algorithm that can be applied to both regression and classification problems. Decision trees may mimic the decision-making process of a human brain. For example, a decision tree may grow from a root (base condition), and when it meets a condition (internal node / feature), it may split into multiple branches. The end of the branch that does not split anymore may be an outcome (leaf). A decision tree can be generated using a training data set according to the following operations: (1) Starting from a root node (the entire dataset), the algorithm may split the dataset in two branches using a decision rule or branching criterion; (2) each of these two branches may generate a new child node; (3) for each new child node, the branching process may be repeated until the dataset cannot be split any further; (4) each branching criterion may be chosen to maximize information gain (e.g., a quantification of how much a branching criterion reduces a quantification of how mixed the labels are in the children nodes). The labels may be the data or the classification that is predicted by the decision tree.

[0076] A random forest regression is an extension of the decision tree model that tends to yield more robust predictions by stretching the use of the training data partition. Whereas a decision tree may make a single pass through the data, a random forest regression may bootstrap 50% of the data (e.g., with replacement) and build many trees. Rather than using all explanatory variables as candidates for splitting, a random subset of candidate variables may be used for splitting, which may enable trees that have completely different data and different variables (hence the term random). The predictions from the trees, collectively referred to as the “forest,” may be then averaged together to produce the final prediction. Many trees (e.g., one hundred trees) may be included in a random forest model, with a number (e.g., 3, 6, 10, etc.) of terms sampled per split, a minimum of number (e.g., 1, 2, 4, 10, etc.) of splits per tree, and a minimum split size (e.g., 16, 32, 64, 128, 256, etc.). Random forests may be trained in a similar way as decision trees. Specifically, training a random forest may include the following operations: (1) select randomly k features from the total number of features; (2) create a decision tree from these k features using the same operations as for generating a decision tree; and (3) repeat the previous two operations until a target number of trees is created.

[0077] The systems, the methods, the computer-readable media, and the techniques disclosed herein may implement one or more long short-term memory (LSTM) techniques. LSTM may bean artificial neural network used in the fields of artificial intelligence and deep learning. Unlike standard feedforward neural networks, LSTM may use feedback connections. The LSTM architecture may provide a short-term memory for a recurrent neural network (RNN). Such RNN can process not only single data points (such as images), but also entire sequences of data (such as speech or video). This characteristic may mean that LSTM networks are well-suited for processing and predicting data. The name of LSTM may refer to the analogy that a standard RNN has both “long-term memory” and “short-term memory.” The connection weights and biases in the RNN may change once per episode of training, analogous to how physiological changes in synaptic strengths store long-term memories; the activation patterns in the network may change once per time-step, analogous to how the moment-to-moment change in electric firing patterns in the brain store short-term memories. The LSTM architecture may provide a short-term memory for an RNN that can last many (e.g., thousands) timesteps.

[0078] In some embodiments, a LSTM unit may comprise a cell, an input gate, an output gate, and a forget gate. The cell may remember values over arbitrary time intervals and the input gate, the output gate, and the forget gate may regulate the flow of information into and out of the cell. Forget gates may be used to decide what information to discard from a previous state by assigning a previous state, compared to a current input, a value between 0 and 1 (e.g., a (rounded) value of 1 may mean to keep the information, and a value of 0 means to discard it). The input gate may decide which pieces of new information to store in the current state, using the same system as the forget gates. The output gate may control which pieces of information in the current state to output (e.g., by assigning a value from 0 to 1 to the information, considering the previous and current states). Selectively outputting relevant information from the current state may allow the LSTM network to maintain useful, long-term dependencies to make predictions, both in current and future time-steps. LSTM networks may be well-suited to classifying, processing and making predictions based on time series data, since there can be lags of unknown duration between important events in a time series. LSTMs may resolve the vanishing gradient problem that can be encountered when training traditional RNNs. Relative insensitivity to gap length may be an advantage of LSTM over RNNs, hidden Markov models and other sequence learning methods in numerous applications.

[0079] In some embodiments, LSTMs may be used with one or more various types of neural networks (e.g., convolutional neural networks (CNNs), deep neural network (DNNs), recurrent neural networks (RNNs), etc.). In some embodiments, CNNs, LSTM, and DNNs are complementary in their modeling capabilities and may be combined a unified architecture. For example, in such unified architecture, CNNs may be well-suited at reducing frequency variations, LSTMs may be well-suited at temporal modeling, and DNNs may be well-suited formapping features to a more separable space. For example, input features to a machine learning model using LSTM techniques in the unified architecture may include segment features for each of a plurality of segments. To process the input features for each of the plurality of segments, the segment features for the segment may be processed using one or more CNN layers to generate first features for the segment; the first features may be processed using one or more LSTM layers to generate second features for the segment; and the second features may be processed using one or more fully connected neural network layers to generate third features for the segments, where the third features may be used for classification operations. In some examples, to process the first features using the one or more LSTM layers to generate the second features, the first features may be processed using a linear layer to generate reduced features having a reduced dimension from a dimension of the first features; and the reduced features may be processed using the one or more LSTM layers to generate the second features. Short-term features having a first number of contextual frames may be generated based on the input features, where features generated using the one or more CNN layers may include long-term features having a second number of contextual frames that are more than the first number of contextual frames of the short-term features. In some embodiments, the one or more CNN layers, the one or more LSTM layers, and the one or more fully connected neural network layers may have been jointly trained to determine trained values of parameters of the one or more CNN layers, the one or more LSTM layers, and the one or more fully connected neural network layers. In some embodiments, the input features may include log-mel features having multiple dimensions. The input features may include one or more contextual frames indicating a temporal context of a signal (e.g., input data). Advantageously, implementations for such unified architecture may leverage complementary advantages associated with each of a CNN, LSTM, and DNN. For example, convolutional layers may reduce spectral variation in input, which may help the modeling of LSTM layers. Having DNN layers after LSTM layers may help reduce variation in the hidden states of the LSTM layers. Training the unified architecture jointly may provide a better overall performance. Training in the unified architecture may also remove the need to have separate CNN, LSTM and DNN architectures, which may be expensive (e.g., in computational resource, in network traffic, in financial resources, in energy consumption, etc.). By adding multi-scale information into the unified architecture, information may be captured at different time scales.

[0080] The systems, the methods, the computer-readable media, and the techniques disclosed herein may implement one or more support vector machine learning techniques. In machine learning, support vector machines (SVMs) may be supervised learning models with associated learning algorithms that analyze data for classification and regression analysis. SVMs may be arobust prediction method, being based on statistical learning. SVMs may be well-suited for domains characterized by the existence of large amounts of data, noisy patterns, or the absence of general theories.

[0081] In general terms, SVMs may map input vectors into high dimensional feature space through non-linear mapping function, chosen a priori. In this high dimensional feature space, an optimal separating hyperplane may be constructed. The optimal hyperplane may then be used to determine things such as class separations, regression fit, or accuracy in density estimation. More formally, a SVM constructs a hyperplane or set of hyperplanes in a high or infinitedimensional space, which can be used for classification, regression, or other tasks like outlier detection.

[0082] Support vectors may be defined as the data points that he closest to the decision surface (or hyperplane). Support vectors may therefore be the data points that are most difficult to classify and may have direct bearing on the optimum location of the decision surface. Given a set of training examples, each marked as belonging to one of two categories, an SVM training algorithm may build a model that assigns new examples to one category or the other, making it a non-probabilistic binary linear classifier (although methods such as Platt scaling exist to use SVM in a probabilistic classification setting). SVM may map training examples to points in space so as to maximize the width of the gap between the two categories. New examples may then be mapped into that same space and predicted to belong to a category based on which side of the gap they fall. In addition to performing linear classification, SVMs can efficiently perform a non-linear classification using what is called the kernel trick, implicitly mapping their inputs into high-dimensional feature spaces.

[0083] Within a support vector machine, the dimensionally of the feature space may be large. For example, a fourth-degree polynomial mapping function may cause a 200-dimensional input space to be mapped into a 1.6 billionth dimensional feature space. The kernel trick and the Vapnik-Chervonenkis dimension may allow the SVM to thwart the “curse of dimensionality” limiting other methods and effectively derive generalizable answers from this very high dimensional feature space. Accordingly, SVMs may assist in discovering knowledge from vast amounts of input data.

[0084] The systems, the methods, the computer-readable media, and the techniques disclosed herein may implement one or more gradient boosting techniques. Gradient boosting is a machine learning technique used in regression and classification tasks, among others. It gives a prediction model in the form of an ensemble of weak prediction models, which are typically decision trees. When a decision tree is the weak learner, the resulting algorithm is called gradient-boosted trees; it usually outperforms random forest. A gradient-boosted trees model is built in a stage-wisefashion as in other boosting methods, but it generalizes the other methods by allowing optimization of an arbitrary differentiable loss function.

[0085] The systems, the methods, the computer-readable media, and the techniques disclosed herein may implement one or more K-nearest neighbors (KNN) techniques. KNN is a nonparametric classification method. In KNN classification, the output is a class membership. An object is classified by a plurality vote of its neighbors, with the object being assigned to the class most common among its k nearest neighbors (k is a positive integer, typically small). If k=l, then the object is assigned to the class of that single nearest neighbor. In KNN regression, the output is the property value for the object. This value is the average of the values of k nearest neighbors. KNN is a type of classification where the function is approximated locally and computation is deferred until function evaluation. Since this algorithm relies on distance for classification, if the features represent different physical units or come in vastly different scales then normalizing the training data can improve its accuracy dramatically.

[0086] The systems, the methods, the computer-readable media, and the techniques disclosed herein may implement one or more Monte Carlo techniques. Monte Carlo is a broad class of computational algorithms that rely on repeated random sampling to obtain numerical results. The underlying concept is to use randomness to solve problems that might be deterministic in principle.

[0087] The systems, the methods, the computer-readable media, and the techniques disclosed herein may implement one or more one-hot encoding techniques. One-hot encoding can be used to deal with categorical data. For example, a machine learning model can use input variables that are numeric. In one example, the categorical variables can be transformed in a pre-processing part. Categorical data can be either nominal or ordinal. Ordinal data can have a ranked order for its values and can therefore be converted to numerical data through ordinal encoding.

[0088] The systems, the methods, the computer-readable media, and the techniques disclosed herein may implement one or more quadratic discriminant analysis (QDA) techniques. In some embodiments, QDA may assume that the measurements from each class are normally distributed. In QDA there is no assumption that the covariance of each of the classes is identical. When the normality assumption is true, the best possible test for the hypothesis that a given measurement is from a given class is the likelihood ratio test.

[0089] The systems, the methods, the computer-readable media, and the techniques disclosed herein may implement one or more response surface methodology (RSM) techniques. RSM may explore the relationships between several explanatory variables and one or more response variables. RSM can use a sequence of designed experiments to obtain an optimal response. RSM can use a factorial experiment or a fractional factorial design. This may be sufficient todetermine which explanatory variables affect the response variable(s) of interest. Once it is suspected that only significant explanatory variables are left, then a more complicated design, such as a central composite design can be implemented to estimate a second-degree polynomial model, which is still only an approximation at best. However, the second-degree model can be used to optimize (e.g., maximize, minimize, or attain a specific target for) the response variable(s) of interest.

[0090] The systems, the methods, the computer-readable media, and the techniques disclosed herein may implement diffusion models (DM), also called diffusion probabilistic models or score-based generative models. These models may leam and operate by utilizing the latent structure of a dataset by modeling the way in which data points diffuse through the latent space. In the context of imaging, as an example, a neural network may be trained to denoise images blurred with Gaussian noise by learning to reverse the diffusion process. It may comprise three components: the forward process, the reverse process, and the sampling procedure. Some nonlimiting examples of generic diffusion modeling frameworks used in computer vision may comprise denoising diffusion probabilistic models, noise conditioned score networks, and stochastic differential equations.

[0091] The systems, the methods, the computer-readable media, and the techniques disclosed herein may implement variational autoencoders (VAEs). VAEs are probabilistic generative models that may require a neural network operating with encoder and / or decoder functions. An encoder neural network may map an input variable to a latent space that corresponds to the parameters of a variational distribution. A decoder may perform the opposite function, which is to map from the latent space to the input space, in order to produce or generate data points. Both networks may be trained together with the usage of the reparameterization trick, although the variance of the noise model may be learned separately.

[0092] The systems, the methods, the computer-readable media, and the techniques disclosed herein may implement latent variable models. This is a statistical model or probabilistic model that aims to explain observed data by introducing hidden or unobservable variables, often referred to as “latent variables.” The observable data could be characterised as either continuous or categorical.

[0093] The systems, the methods, the computer-readable media, and the techniques disclosed herein may implement U-Net models. U-Net models may have convolutional neural network (CNN) architectures commonly used for tasks related to image segmentation, which may involve partitioning an image into different segments and assigning labels to each segment. A U- Net model may have both a contracting path (the left side of the “U”) and an expansive path (the right side of the “U”). The contracting path may comprise convolutional and pooling layers thatprogressively reduce the spatial dimensions of the input image while increasing the number of feature channels. This path may capture the contextual information from the input image. The expansive path may involve a series of up-sampling and convolutional layers that gradually increase the spatial dimensions of the representation, allowing the network to generate a pixelwise segmentation mask that aligns with the original input image's dimensions. Skip connections can be important features of the U-Net architecture. They may connect corresponding layers in the contracting and expansive paths. These connections may allow the U-Net to combine high- resolution features from the contracting path with the up-sampled features from the expansive path, aiding in precise localization.

[0094] Machine learning may utilize written text, known as a “text encoder”. This may transform textual input, such as sentences or paragraphs, into numerical representations that may be understood and processed by machine learning algorithms. These numerical representations may capture the semantic and contextual information present in the text, enabling downstream tasks like sentiment analysis, machine translation, text classification, and more. Some nonlimiting examples of text encoders may comprise Bag-of-Words (BoW) and TF-IDF, Word Embeddings, Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU), and Transformer-based Models.

[0095] Machine learning may also utilize noise to alter images. In some embodiments, denoising may refer to a process of removing unwanted or extraneous noise from a signal, data, image, or any form of information. Noise in this context may refer to any unwanted variation or interference that may obscure the underlying structure or information contained within the data. Some non-limiting examples of noise may comprise Gaussian noise, salt-and-pepper noise, speckle noise, shot noise, quantization noise, color noise, and aliasing noise. Denoising techniques may comprise mean and median filters, Gaussian filters, Non-local Means (NLM), total variation denoising, wavelet denoising, and deep learning.

[0096] FIG. 2 is a block diagram of a convolutional neural network for generating an enhanced sonographic image or video that depicts a fetus, according to some embodiments of the present disclosure. The convolutional neural network 200 comprises a deep learning algorithm that takes in input 210 (e.g., sonographic images and / or videos of a fetus) and generates an output 280 of enhanced sonographic image and / or video.

[0097] The convolutional neural network 200 may have one or more of convolution layers and pooling layers. The convolution layers may perform convolution operations between the input 210 values and convolution filters (matrix of parameters) that are learned over many gradient update iterations during the training. Convolutions operate over 3D tensors, called feature maps, with two spatial axes (height and width) as well as a depth axis (i.e., channel axis). For an RGBimage, the dimension of the depth axis is 3, because the image has three color channels (red, green, and blue). For a black-and-white picture, for example, 2D sonographic images, the depth is 1 (levels of gray). The convolution operation may extract patches from its input feature map and applies the same transformation to all of these patches, producing an output feature map. This output feature map is still a 3D tensor: it has a width and a height. Its depth can be arbitrary, because the output depth is a parameter of the layer, and the different channels in that depth axis no longer stand for specific colors as in RGB input; rather, they stand for filters that encode specific aspects of the input 210.

[0098] A pooling layer may operate a pooling operation that divides input into non-overlapping two-dimensional spaces. For example, the feature maps 220 and 240 are outputs generated from previous convolution operations and used as input to subsequent pooling layers, respectively. A filter with a size of 2 * 2 is slid over the feature maps using a stride of 2. For a receptive field with a size of 2 * 2 (e.g., the part of the feature maps 220 and 240 under the filter), an average pooling operation may produce an average value of the four pixels in the receptive field, whereas a maximum pooling operation may select a maximum value of the four pixels in the receptive field. As such, pooling operations may consolidate the features learned by the convolutional neural network 200 and gradually reduce the spatial dimension of the feature maps to minimize the numbers of parameters and computations in the network.

[0099] As illustrated in FIG. 2, input 210 is processed via a convolution operation to generate feature maps 220, which in turn are processed by a pooling operation to generate pooled features maps 230. As the convolutional neural network 200 may comprise a plurality of convolution layers and pooling layers where output from a previous layer may be input to a next layer, the convolution and pooling operations may repeat, thereby generating feature maps 240 and pooled feature maps 250, respectively. In some embodiments, the pooled feature maps 250 which are 2- dimensional arrays, may be processed via one or more fully connected layer 260 and sofimax function 270, which generates output 280.

[0100] In some embodiments, the convolutional neural network 200 may be scaled to construe machine learning models with better accuracy and efficiency. For example, one or more of depth, width, resolution of the convolutional neural network 200 may be scaled. Depending on the input, the convolutional neural network 200 may be scaled in a single dimension or multiple dimensions. For example, for higher resolution input sonographic data, the width and depth of the convolutional neural network 200 may be scaled, such that larger receptive fields are able to capture similar features that include more pixels in larger images / videos.

[0101] It should be noted that FIG. 2 is a non-limiting example of convolutional neural network for illustrative purposes. The number of convolution layers, pooling layers, flattening layers andfully connected layers may be adjusted without deviating from the scope of the disclosure herein.Training of Machine Learning Model

[0102] The present disclosure further provides systems, methods, and non-transitory computer readable media for constructing a machine learning model for generating enhanced sonographic images and / or videos that are lifelike depiction of a fetus. In some embodiments, a method of constructing a machine learning model for generating an enhanced image or video of a fetus may comprise accessing training data comprising at least one of an enhanced image or an enhanced video of a fetus, where the at least one of the enhanced image or the enhanced video includes a lifelike depiction of the fetus, and training a machine learning model that runs on one or more processors coupled to memory on the training data, by progressively matching output of the machine learning model with a ground truth label of the at least one of the enhanced image or the enhanced video.

[0103] In some embodiments, the machine learning model 120 may be trained by way of supervised learning. A data set may be divided into a training set, a test set, and, in some embodiments, a validation set. In supervised learning, training data and validation data may be annotated with ground truth labels. During the training process, training data is repeatedly presented to the machine learning model 120, and for each sample presented during training, output generated by the machine learning model 120 may be compared with the corresponding ground truth label. The difference between the ground truth and the generated output may be calculated, and the machine learning model 120 may be modified to cause the output to more closely approximate or predict the ground truth. In some embodiments, a backpropagation algorithm may be utilized to cause the output to more closely approximate the ground truth. During many training iterations, the machine learning may generate outputs that progressively match the corresponding ground truth labels. Subsequently, when new and previously unseen input is presented to the machine learning model, it may generate an output classification value indicating which of the categories the new sample is most likely to fall into. In other words, the machine learning model may “generalize” from its training to new, previously unseen input.

[0104] In some embodiments, the machine learning model 120 may be validated using a validation dataset (e.g., distinct from training data set) to determine accuracy and robustness of the model. Such validation may include applying the model to the validation dataset to make predictions derived from the validation dataset. The machine learning model 120 may then be evaluated to determine whether performance is sufficient based upon the derived predictions. The sufficiency criteria applied to the machine learning model 120 may vary depending uponthe size of the training data set available for training, the performance of previous iterations of trained models, or user-specified performance requirements. If the machine learning model 120 does not achieve sufficient performance, additional training may be performed. Additional training may include refinement of the model or retraining on a different training dataset, after which the newly trained model may again be validated and assessed. When the machine learning model 120 has achieved sufficient performance, in some embodiments, the machine learning model 120 may be stored for present or future use. The model may be stored as sets of parameter values or weights for analysis of further input (e.g., further relevant parameters to use as further predicted variables, further explanatory variables, further user interaction data, etc.), which may also include analysis logic or indications of model validity. In some embodiments, a plurality of machine learning models may be stored for generating predictions under different sets of input data conditions. In some embodiments, the plurality of machine learning models may be stored in a database (e.g., associated with a server).

[0105] FIG. 3 is a block diagram of training a machine learning model, in accordance with some embodiments. In the training process 300, weight parameters in each layer of the machine learning model 320 are optimized using backpropagation 350 based on comparison between the estimated / predicted output 330 and the ground truth 340 until the estimated output 330 progressively matches or approaches the ground truth 340. A single cycle of the optimization process is organized as follows. First, given a training dataset as input 310, the forward pass sequentially computes the output in each layer and propagates the function signals forward through the machine learning model 320. In the final output layer, an objective loss function measures an error between the estimated output 330 and given labels (e.g., ground truth 340) of the training data. To minimize the training error, the backward pass 360 uses the chain rule to backpropagate error signals and compute gradients with respect to all weights throughout the neural network. Finally, the weight parameters are updated using optimization algorithms based on stochastic gradient descent (SGD). Several optimization algorithms stem from stochastic gradient descent. For example, the Adagrad and Adam training algorithms perform stochastic gradient descent while adaptively modifying learning rates based on update frequency and moments of the gradients for each parameter, respectively.

[0106] In some embodiments, training data used to train the machine learning model 120 may be annotated with ground truth labels. The training data may comprise sonographic images and videos collected from a variety of ultrasound imaging devices. The sonographic images and videos may be enhanced by a trained machine learning model or other image enhancement or editing tools (e.g., Photoshop).

[0107] In some embodiments, the training data may comprise 2D and 3D sonographic images. In some embodiments, the training data may comprise sonographic videos or cine clips of the fetus. The fetus depicted in the training data may be from an early gestational age to delivery of the newborn. In some embodiments, the fetus may be from pregnancy weeks 8-41, 9-40, 10-39, 11-38, 12-37, 13-36, and the like.

[0108] In some embodiments, the training data may comprise at least 50, at least 100, at least 150, at least 200, at least 500, at least 1000, at least 2,000, at least 3,000, at least 5,000, at least 8,000, at least 10,000, at least 12,000 sonographic images and / or videos. At least some of the sonographic images and / or videos may be enhanced by a trained machine learning model or other image enhancement or editing tools.

[0109] Unsupervised learning may be used, in some embodiments, to train the machine learning model 120 to use input data (e.g., sonographic data) and generate enhanced images and / or videos. Unsupervised learning, in some embodiments, includes feature extraction which is performed by the machine learning model 120 on the input data. Extracted features may be used for visualization, for classification, for subsequent supervised training, and more generally for representing the input for subsequent storage or analysis.

[0110] Machine learning models that are commonly used for unsupervised training include k- means clustering, mixtures of multinomial distributions, affinity propagation, discrete factor analysis, hidden Markov models, Boltzmann machines, restricted Boltzmann machines, autoencoders, convolutional autoencoders, recurrent neural network autoencoders, and long short-term memory autoencoders. While there are many unsupervised learning models, they all have in common that, for training, they require a training set without associated labels.Inference of Machine Learning Model

[0111] Following training, the machine learning model can be used to analyze new data that the model has never encountered. Accordingly, the present disclosure provides systems, methods and non-transitory computer readable media for generating enhanced sonographic images and / or videos using a trained machine learning model. In some embodiments, a method for generating enhanced sonographic images and / or videos is disclosed herein. The method may comprise accessing sonographic data and generating, using a trained machine learning model, as output, enhanced sonographic images and / or videos.

[0112] FIG. 4 is a workflow diagram of generating an enhanced sonographic image or video that depicts a fetus using a machine learning model, according to some embodiments of the present disclosure. At step 410, the trained machine learning model accesses sonographic data that depicts a fetus. The sonographic data may comprise sonographic images and / or videos. At step420, the trained model processes the sonographic data and generates at least one of an enhanced image or an enhanced video as output. In some embodiments, the at least one of an enhanced image or an enhanced video may be lifelike depiction of the fetus.

[0113] FIG. 5 is a dataflow diagram 500 of generating an enhanced sonographic image or video that depicts a fetus using a machine learning model, according to some embodiments of the present disclosure. The machine learning model may be implemented via web-based computing service (e.g., Amazon Web Service). The machine learning model may access, process sonographic images and videos of a fetus uploaded by the and generate an enhanced image and / or video that includes a lifelike depiction of the fetus. The machine learning model may be stored in at least one of Al 1.2 G3 Large 520 or Al 1.3 G3 Large 518. Both of Al 1.2 G3 Large 520 and Al 1.3 G3 Large 518 may comprise servers that house the machine learning model. In some embodiments, the machine learning model may perform a two-stage process of enhancing sonographic data. For example, sonographic data may be processed in a first stage and further refined in a second stage. In other embodiments, the machine learning model may enhance sonographic data in a one-stage process.

[0114] A user uploads sonographic data from a user device (e.g., 502, 504, and 506) to File Handler Web Service 508. The user device (e.g., 502, 504, and 506) may be any suitable computing devices including, by way of non-limiting examples, desktop computers, laptop computers, notebook computers, netbook computers, netpad computers, handheld computers, Internet appliances, mobile smartphones, and tablet computers. The user may upload sonographic data via a mobile application or a web application. The user may be expectant parents, friends and relatives thereof, or health care providers. In some embodiments, the user may be required to pay service charge prior to uploading the sonographic data.

[0115] In order to maintain the simplicity in user operation, users may not need to configure the types of enhancements. For example, the system may pre-configure one or more settings for sonographic data enhancement, including smoothing, lighting adjustments, adding skin details, making hair more pronounced, and emphasizing body features (e.g., ears, nose, lips, fingers, shoulders, chest). In other embodiments, users may be provided with options of configuring one or more settings and generating enhanced images and videos customized to their preference.

[0116] The sonographic data uploaded by the user to File Handler Web Service 508, and transmitted to File Handling Lambda 512, Kaltura EC2 Instances 510 and S3 Input Bucket 514. The sonographic data is securely transferred and stored within the web-based computing service, for example in S3 Input Bucket 514, S3 Transfer Bucket 516, and S3 Output Bucket 524. The enhanced sonographic images and / videos generated by the machine learning model are securely stored within the web-based computing service, for example in S3 Output Bucket 524.

[0117] In some embodiments, the user may retrieve the enhanced sonographic images and / videos. For example, the user may retrieve and display on the user device (e.g., 502, 504, and 506) via a mobile application or a web application. Alternatively, the enhanced sonographic images and / videos may be stored in an imaging archive (e.g., gallery) provided by the webbased computing service. The user can access the imaging archive via the user device.Computer Systems

[0118] The present disclosure provides computer systems that are programmed to implement methods of the disclosure. FIG. 6 shows a computer system 601 that is programmed or otherwise configured to generate enhanced sonographic images and / or videos, in accordance with some embodiments. The computer system 601 can regulate various aspects of the present disclosure, for example, implementing machine learning algorithms. The computer system 601 can be an electronic device of a user or a computer system that is remotely located with respect to the electronic device.

[0119] The computer system 601 includes a central processing unit (CPU, also “processor” and “computer processor” herein) or graphical processing unit (GPU) 605, which can be a single core or multi core processor, or a plurality of processors for parallel processing. The computer system 601 also includes memory or memory location 610 (e.g., random-access memory, readonly memory, flash memory), electronic storage unit 615 (e.g., hard disk), communication interface 620 (e.g., network adapter) for communicating with one or more other systems, and peripheral devices 625, such as cache, other memory, data storage and / or electronic display adapters. The memory 610, storage unit 615, interface 620 and peripheral devices 625 are in communication with the CPU / GPU 605 through a communication bus (solid lines), such as a motherboard. The storage unit 615 can be a data storage unit (or data repository) for storing data. The computer system 601 can be operatively coupled to a computer network (“network”) 630 with the aid of the communication interface 620. The network 630 can be the Internet, an intranet and / or extranet, or an intranet and / or extranet that is in communication with the Internet. The network 630 in some embodiments is a telecommunication and / or data network. The network 630 can include one or more computer servers, which can enable distributed computing, such as cloud computing. The network 630, in some embodiments with the aid of the computer system 601, can implement a peer-to-peer network, which may enable devices coupled to the computer system 601 to behave as a client or a server.

[0120] The CPU / GPU 605 can execute a sequence of machine-readable instructions, which can be embodied in a program or software. The instructions may be stored in a memory location, such as the memory 610. The instructions can be directed to the CPU / GPU 605, which cansubsequently program or otherwise configure the CPU / GPU 605 to implement methods of the present disclosure. Examples of operations performed by the CPU / GPU 605 can include fetch, decode, execute, and writeback.

[0121] The CPU / GPU 605 can be part of a circuit, such as an integrated circuit. One or more other components of the system 601 can be included in the circuit. In some embodiments, the circuit is an application specific integrated circuit (ASIC).

[0122] The storage unit 615 can store files, such as drivers, libraries and saved programs. The storage unit 615 can store user data, e.g., user preferences and user programs. The computer system 601 in some embodiments can include one or more additional data storage units that are external to the computer system 601, such as located on a remote server that is in communication with the computer system 601 through an intranet or the Internet.

[0123] The computer system 601 can communicate with one or more remote computer systems through the network 630. For instance, the computer system 601 can communicate with a remote computer system of a user (e.g., a mobile device). Examples of remote computer systems include personal computers (e.g., portable PC), slate or tablet PC’s (e.g., Apple® iPad, Samsung® Galaxy Tab), telephones, Smart phones (e.g., Apple® iPhone, Android-enabled device, Blackberry®), or personal digital assistants. The user can access the computer system 601 via the network 630.

[0124] Methods as described herein can be implemented by way of machine (e.g., computer processor) executable code stored on an electronic storage location of the computer system 601, such as, for example, on the memory 610 or electronic storage unit 615. The machine executable or machine-readable code can be provided in the form of software. During use, the code can be executed by the processor 605. In some embodiments, the code can be retrieved from the storage unit 615 and stored on the memory 610 for ready access by the processor 605. In some situations, the electronic storage unit 615 can be precluded, and machine-executable instructions are stored on memory 610.

[0125] The code can be pre-compiled and configured for use with a machine having a processor adapted to execute the code, or can be compiled during runtime. The code can be supplied in a programming language that can be selected to enable the code to execute in a pre-compiled or as-compiled fashion.

[0126] Aspects of the systems and methods provided herein, such as the computer system 601, can be embodied in programming. Various aspects of the technology may be thought of as “products” or “articles of manufacture” typically in the form of machine (or processor) executable code and / or associated data that is carried on or embodied in a type of machine readable medium. Machine-executable code can be stored on an electronic storage unit, such asmemory (e.g., read-only memory, random-access memory, flash memory) or a hard disk. “Storage” type media can include any or all of the tangible memory of the computers, processors or the like, or associated modules thereof, such as various semiconductor memories, tape drives, disk drives and the like, which may provide non-transitory storage at any time for the software programming. All or portions of the software may at times be communicated through the Internet or various other telecommunication networks. Such communications, for example, may enable loading of the software from one computer or processor into another, for example, from a management server or host computer into the computer platform of an application server. Thus, another type of media that may bear the software elements includes optical, electrical and electromagnetic waves, such as used across physical interfaces between local devices, through wired and optical landline networks and over various air-links. The physical elements that carry such waves, such as wired or wireless links, optical links or the like, also may be considered as media bearing the software. As used herein, unless restricted to non-transitory, tangible “storage” media, terms such as computer or machine “readable medium” refer to any medium that participates in providing instructions to a processor for execution.

[0127] Hence, a machine readable medium, such as computer-executable code, may take many forms, including but not limited to, a tangible storage medium, a carrier wave medium or physical transmission medium. Non-volatile storage media include, for example, optical or magnetic disks, such as any of the storage devices in any computer(s) or the like, such as may be used to implement the databases, etc. shown in the drawings. Volatile storage media include dynamic memory, such as main memory of such a computer platform. Tangible transmission media include coaxial cables; copper wire and fiber optics, including the wires that comprise a bus within a computer system. Carrier-wave transmission media may take the form of electric or electromagnetic signals, or acoustic or light waves such as those generated during radio frequency (RF) and infrared (IR) data communications. Common forms of computer-readable media therefore include for example: a floppy disk, a flexible disk, hard disk, magnetic tape, any other magnetic medium, a CD-ROM, DVD or DVD-ROM, any other optical medium, punch cards paper tape, any other physical storage medium with patterns of holes, a RAM, a ROM, a PROM and EPROM, a FLASH-EPROM, any other memory chip or cartridge, a carrier wave transporting data or instructions, cables or links transporting such a carrier wave, or any other medium from which a computer may read programming code and / or data. Many of these forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to a processor for execution.

[0128] The computer system 601 can include or be in communication with an electronic display 635 that comprises a user interface (UI) 640 for providing, for example, a dashboard. Examples of UI include, without limitation, a graphical user interface (GUI) and web-based user interface.

[0129] Methods and systems of the present disclosure can be implemented by way of one or more algorithms. An algorithm can be implemented by way of software upon execution by the CPU / GPU 605.

[0130] While preferred embodiments of the present disclosure have been shown and described herein, it will be obvious to those skilled in the art that such embodiments are provided by way of example only. It is not intended that the present disclosure be limited by the specific examples provided within the specification. While the present disclosure has been described with reference to the aforementioned specification, the descriptions and illustrations of the embodiments herein are not meant to be construed in a limiting sense. Numerous variations, changes, and substitutions will now occur to those skilled in the art without departing from the scope of the present disclosure. Furthermore, it shall be understood that all aspects of the present disclosure are not limited to the specific depictions, configurations or relative proportions set forth herein which depend upon a variety of conditions and variables. It should be understood that various alternatives to the embodiments of the present disclosure described herein may be employed in practicing the present disclosure. It is therefore contemplated that the present disclosure shall also cover any such alternatives, modifications, variations or equivalents. It is intended that the following claims define the scope of the invention and that methods and structures within the scope of these claims and their equivalents be covered thereby.

Claims

CLAIMSWHAT IS CLAIMED IS:

1. A method for generating an enhanced image or video of a fetus, the method comprising: accessing sonographic data that depicts a fetus; and generating, using a trained machine learning model, as output, at least one of an enhanced image or an enhanced video of the fetus based on the sonographic data, wherein the at least one of the enhanced image or the enhanced video includes a lifelike depiction of the fetus.

2. A method for generating an enhanced image or video of a fetus, the method comprising: accessing sonographic data that depicts a fetus, wherein the sonographic data is stored in a user device; transmitting the sonographic data to a server or a computing device implemented with a trained machine learning model; receiving, from the trained machine learning model, at least one of an enhanced image or an enhanced video of the fetus based on the sonographic data, wherein the at least one of the enhanced image or the enhanced video includes a lifelike depiction of the fetus; and displaying the at least one of the enhanced image or the enhanced video on a display on or coupled to the user device.

3. A computer-implemented method of constructing a machine learning model for generating an enhanced image or video of a fetus, the method comprising: accessing training data comprising at least one of an enhanced image or an enhanced video of a fetus, wherein the at least one of the enhanced image or the enhanced video includes a lifelike depiction of the fetus; and training a machine learning model that runs on one or more processors coupled to memory on the training data, by progressively matching output of the machine learning model with a ground truth label of the at least one of the enhanced image or the enhanced video.

4. A method for enhancing fetal ultrasound imagery to make the image more realistic, the method comprising: receiving an ultrasound image of a fetus and non-fetal tissue; removing ultrasound artifacts associated with the non-fetal tissue from the ultrasound image to generate a first modified image; applying a smoothing algorithm to the ultrasound image to reduce noise and enhance clarity of the fetus to generate a second modified image;applying an enhancement algorithm to enhance the second modified image of the fetus to generate a final image, wherein the final image comprises a fetus without non-fetal tissue; outputting the enhanced image of the fetus.

5. The method according to any one of the preceding claims, wherein the removing ultrasound artifacts associated with the non-fetal tissue comprises: identifying the non-fetal tissue from the ultrasound image using machine learning; removing the identified non-fetal tissue from the ultrasound image; and generating image data to fill in a gap created by the removal of the identified non-fetal tissue using stable diffusion to generate the ultrasound image without non-fetal tissue.

6. The method according to any one of the preceding claims, wherein the smoothing algorithm comprises: detecting uneven surfaces in the first modified image; and modifying the images to remove the uneven surfaces to output a second modified image.

7. The method according to any one of the preceding claims, wherein applying the enhancement algorithm comprises: applying a modified stable diffusion algorithm to the second modified image to output the final image.

8. The method according to any one of the preceding claims, wherein the sonographic data comprises one or more sonographic images of the fetus.

9. The method according to any one of the preceding claims, wherein the one or more sonographic images comprise one or more of two-dimensional (2D) or three-dimensional (3D) images.

10. The method according to any one of the preceding claims, wherein the sonographic data comprises sonographic images directly transmitted from an ultrasound imaging device.

11. The method according to any one of the preceding claims, wherein the sonographic data comprises one or more sonographic videos of the fetus.

12. The method according to any one of the preceding claims, further comprising: removing personal information from the sonographic data, wherein the personal information comprises personal health information (PHI) of the fetus and information associated health service to the fetus.

13. The method according to any one of the preceding claims, wherein the machine learning model comprises a convolutional neural network.

14. The method according to any one of the preceding claims, wherein the convolutional neural network comprises one or more of convolution layers, pooling layers, or fully connected layers.

15. The method according to any one of the preceding claims, wherein the neural network is trained on a dataset of fetal images to improve the accuracy of artifact removal and image smoothing.

16. The method according to any one of the preceding claims, wherein the smoothing algorithm employs machine learning techniques to distinguish between fetal tissue and non-fetal artifacts.

17. The method according to any one of the preceding claims, wherein the non-fetal artifacts comprises at least one of a placenta or an umbilical cord.

18. A non-transitory computer readable medium comprising computer-executable instructions that, when executed by the processor, causes the processor to perform a method of enhancing fetal ultrasound imagery according to any of the preceding claims.

19. A system for enhancing fetal ultrasound imagery, the system comprising: a one or more processors; and a non-transitory computer readable medium comprising computer-executable instructions that, when executed by the processor, causes the processor to perform a method according to any of the preceding claims.

20. A system for enhancing fetal ultrasound imagery, the system comprising: an ultrasound imaging device for capturing an ultrasound image of a fetus; one or more memories comprising computer-executable instructions; and one or more processor connected to the one or more memory and configured to execute the computer-executable instructions that comprise a method according to any of the preceding claims.

Images