Neural network simulator for ultrasound images and clips

Abstract

The ultrasound image simulator includes a generative neural network that receives a position and orientation of an ultrasound probe and generates at least one simulated ultrasound image or clip of a body part of a subject, the generative neural network being trained with multiple 2D ultrasound images or clips of the body part acquired from multiple ultrasound probe positions and orientations.

Description

[Technical Field]

[0001] The present invention relates generally to simulators, and more particularly to neural network simulators for ultrasound images and clips.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 63 / 376,108, filed September 19, 2022, the contents of which are incorporated herein by reference. [Background technology]

[0003] Generating synthetic ultrasound images is difficult and typically requires complex modeling of the body's 3D structure (shape, density) using 3D MRI or ultrasound data of internal structures.

[0004] 1, existing simulation methods for generating images have three components: an ultrasound beam generator 10 that generates simulated acoustic beams; a modeler 12 of the interaction of the ultrasound beams with a 3D model of the patient's internal organs; and an ultrasound image generator 14 that forms an ultrasound image based on sound waves returning from the internal organs. The 3D model is typically generated by a 3D modeler 16 from 3D type data such as MRI scans and / or 3D ultrasound scans and / or artificial 3D models created using 3D modeling software.

[0005] The simulation may use linear methods, ray tracing, or numerical solutions of the wave equation.

[0006] An exemplary linear method is described in the paper "Calculation of pressure fields from arbitrarily shaped, apodized, and excited ultrasound transducers," by J. Jensen and N. Svendsen, IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control, vol. 39, no. 2, pp. 262-267, 1992. In this method, an acoustic beam profile is simulated by integrating spherical waves from each point on the surface of the ultrasound transducer. The response from the material is calculated as the convolution of the beam profile with the scattering density of the material. This assumes that each point in the material weakly scatters sound waves from the beam independently of other points. Unfortunately, this method does not simulate ultrasound effects such as shadowing and refraction.

[0007] Ray tracing methods are described in the papers "Real-time gpu-based ultrasound simulation using deformable mesh models" by B. Burger, S. Bettinghausen, M. Radle, and J. Hesser, IEEE Transactions on Medical Imaging, vol. 32, no. 3, pp. 609-618, 2012, and "Monte-Carlo ray-tracing for realistic interactive ultrasound simulation" by O. Mattausch and O. Goksel, in Proceedings of the Eurographics Workshop on Visual Computing for Biology and Medicine, 2016, pp. 173-181.

[0008] These methods assume that ultrasound acoustic beams can be modeled as rays. They simulate the refraction and reflection of sound waves as they pass through materials, but ignore the wave-like nature of the acoustic beam. A 3D mesh model is used to describe the anatomy of the organ where the image is being generated, and the various propagation properties of different materials are provided as parameters for the simulator. The image is then post-processed to generate an ultrasound speckle pattern.

[0009] The paper "Representing arbitrary acoustic source and sensor distributions in Fourier collocation methods" by E.S.Wise, B.T.Cox, J. Jaros and B.E.Treeby, The Journal of the Acoustical Society of America, vol. 146, no. 1, p. 278, July 2019, issn: 1520-8524, describes a method that uses numerical solutions of the wave equation.

[0010] This method discretizes space and time and models the propagation of sound waves from a transducer and their interaction with internal organs. The model is generated by solving differential equations on a grid. This method is highly accurate but time-consuming. This method requires a voxel map of the internal organs and parameters to describe their density.

[0011] In another method, a neural network is used to add ultrasound effects to simulated 2D slices as they are generated from the system described above, and these 2D slices are used as input to the neural network that generates the ultrasound image.

[0012] This is described in the paper "Simulating patho-realistic ultrasound images using deep generative networks with adversarial learning" by F. Tom and D. Sheet, 2017, and "A real-time medical ultrasound simulator based on a generative adversarial network model" by B. Peng, X. Huang, S. Wang, and J. Jiang, 2019 IEEE International Conference on Image Processing (ICIP), pp. 4629-4633. The neural network is responsible for simulating the ultrasound effects but not the image geometry.

[0013] All of these methods require some form of three-dimensional modeling, provided either through a 3D (volumetric) imaging system (e.g., 3D ultrasound, MRI, CT scan) or an artificial parametric 3D model. Summary of the Invention

[0014] Thus, in accordance with a preferred embodiment of the present invention, there is provided an ultrasound image simulator comprising a generative neural network that receives an ultrasound probe position and orientation and generates at least one simulated ultrasound image or clip of a body part of a subject, the generative neural network being trained with multiple 2D ultrasound images or clips of the body part acquired from multiple ultrasound probe positions and orientations.

[0015] Furthermore, in accordance with a preferred embodiment of the present invention, the multiple 2D ultrasound images or clips and the positions and orientations of the multiple ultrasound probes are generated by an ultrasound guidance system, or alternatively, by a probe motion sensing system.

[0016] Furthermore, in accordance with a preferred embodiment of the present invention, an image or clip is associated with a number of parameters, which may be, for example, ultrasound probe parameters, patient parameters, operator parameters, or noise parameters.

[0017] Further in accordance with a preferred embodiment of the present invention, at least one simulated ultrasound image or clip includes an artificial or non-human element therein.

[0018] Furthermore, in accordance with a preferred embodiment of the present invention, the generative neural network also includes a latent variable provider.

[0019] Further, in accordance with a preferred embodiment of the present invention, a generative neural network is trained with a latent variable neural network that receives a plurality of 2D ultrasound images or clips and a plurality of ultrasound probe positions and orientations.

[0020] Furthermore, in accordance with a preferred embodiment of the present invention, the generative neural network includes a diffusion model neural network.

[0021] Alternatively, in accordance with a preferred embodiment of the present invention, the generative neural network includes a volumetric neural network, a slicer, and a rendering neural network. The volumetric neural network receives the position and orientation of the ultrasound probe and generates volumetric data of the body-part for the position and orientation of the ultrasound probe. The slicer extracts slices of the volume associated with the position and orientation of the ultrasound probe. The rendering neural network renders the slices as simulated ultrasound images of the body-part.

[0022] Furthermore, in accordance with a preferred embodiment of the present invention, both the volumetric neural network and the rendering neural network are trained with an empirical risk-minimization training procedure and utilize the same loss function.

[0023] Furthermore, in accordance with a preferred embodiment of the present invention, the simulator can be used in a system to create synthetic data for training an ultrasound-based machine learning unit.

[0024] Furthermore, in accordance with a preferred embodiment of the present invention, the ultrasound-based machine learning unit may be a classifier, a segmenter or a regressor.

[0025] Further, in accordance with a preferred embodiment of the present invention, the ultrasound-based machine learning unit includes a navigation neural network under training for generating probe movement instructions for the ultrasound probe on the virtual subject, and a probe position updater for converting the movement instructions into a position and orientation of the ultrasound probe.

[0026] Further, in accordance with a preferred embodiment of the present invention, the simulator can be used in a system to create synthetic data for training a person to perform ultrasound scans of an artificial body with an ultrasound probe, along with sonographer training software that receives at least one simulated ultrasound image or clip and provides instructions to the person.

[0027] Further, in accordance with a preferred embodiment of the present invention, the simulator can be used in a system for improving a partial or corrupted image of a body part. The generative neural network generates a simulated ultrasound image of the body part, and the system includes a comparator that determines a difference between the simulated ultrasound image and the partial or corrupted image, and an optimizer that updates multiple parameters and / or the position and orientation of the ultrasound probe to reduce the difference and thereby generate an improved version of the partial or corrupted image of the body part.

[0028] Furthermore, according to preferred embodiments of the present invention, the system can be used for image completion and / or image correction and / or image noise reduction and / or 3D reconstruction and / or synthesis of new views.

[0029] Also provided in accordance with a preferred embodiment of the present invention is a method for generating ultrasound images, the method including generating, by a generative neural network, at least one simulated ultrasound image or clip of a body part of a subject in response to a position and orientation of an ultrasound probe, the generative neural network being trained with multiple 2D ultrasound images or clips of the body part acquired from multiple ultrasound probe positions and orientations.

[0030] Further in accordance with a preferred embodiment of the present invention, the method includes generating, by an ultrasound guidance system, a plurality of 2D ultrasound images or clips and a plurality of ultrasound probe positions and orientations.

[0031] Further in accordance with a preferred embodiment of the present invention, the method includes generating positions and orientations of the plurality of ultrasound probes with a probe motion sensing system.

[0032] Furthermore, in accordance with a preferred embodiment of the present invention, the method also includes the step of providing latent variables to the generative neural network.

[0033] Further in accordance with a preferred embodiment of the present invention, the method includes training a generative neural network with a latent variable neural network that receives a plurality of 2D ultrasound images or clips and a plurality of ultrasound probe positions and orientations.

[0034] Furthermore, in accordance with a preferred embodiment of the present invention, the generative neural network includes a diffusion model neural network.

[0035] Further in accordance with a preferred embodiment of the present invention, the generating step includes the steps of generating volumetric data of the body part for the position and orientation of the ultrasound probe using a volumetric neural network in response to the position and orientation of the ultrasound probe, extracting slices of the volume associated with the position and orientation of the ultrasound probe, and rendering the slices as simulated ultrasound images of the body part using a rendering neural network.

[0036] Furthermore, in accordance with a preferred embodiment of the present invention, the method includes training both the volumetric neural network and the rendering neural network with an empirical risk-minimization training procedure that utilizes the same loss function.

[0037] Also, according to one aspect of the present invention, there is provided a method for generating synthetic data for training an ultrasound-based machine learning unit, the method including receiving at least one of a plurality of parameters and generating, using a generative neural network, at least one simulated ultrasound image or clip for the at least one parameter.

[0038] Further, in accordance with a preferred embodiment of the present invention, the ultrasound-based machine learning unit is one of a classifier, a segmenter, and a regressor. Alternatively, the ultrasound-based machine learning unit includes a navigation neural network under training for generating probe movement instructions for an ultrasound probe on a virtual subject, and the method includes converting the movement instructions into a position and orientation of the ultrasound probe.

[0039] There is also provided, in accordance with a preferred embodiment of the present invention, a method for generating synthetic data for training a person to perform ultrasound scans of an artificial body with an ultrasound probe, the method including receiving at least one of a plurality of parameters, generating at least one simulated ultrasound image or clip for the at least one parameter using a generative neural network, and providing instructions to the person through sonographer training software.

[0040] Finally, in accordance with a preferred embodiment of the present invention, there is also provided a method for improving a partial or corrupted image of a body part, the method comprising the steps of generating at least one simulated ultrasound image or clip using a generative neural network, determining a difference between the partial or corrupted image and the simulated ultrasound image, and updating a plurality of parameters and / or the position and orientation of the ultrasound probe to reduce the difference, thereby generating an improved version of the partial or corrupted image of the body part. [Brief explanation of the drawings]

[0041] The subject matter which is regarded as the invention is particularly pointed out and distinctly claimed in the concluding portion of the specification, however the invention both as to organization and method of operation, together with its objects, features and advantages, may best be understood by reference to the following detailed description taken in conjunction with the accompanying drawings. [Figure 1]FIG. 1 is a block diagram of a prior art simulation method. [Figure 2] FIG. 2 is a block diagram of a novel simulator for ultrasound images and clips, constructed and operative in accordance with a preferred embodiment of the present invention. [Figure 3] FIG. 3 is a schematic diagram of a sliced ​​heart showing exemplary ultrasound views that are useful for understanding the simulator of FIG. [Figure 4] FIG. 4 is a schematic diagram of the training data set for the simulator of FIG. [Figure 5] 5A and 5B are block diagrams of one embodiment of the simulator of FIG. 2 and its training components, respectively, constructed and operative in accordance with a preferred embodiment of the present invention. [Figure 6] FIG. 6 is a graphical illustration of an exemplary simulated image produced by the simulator of FIG. [Figure 7] 7A and 7B are block diagrams of alternative embodiments of the latent variables, the simulator of FIG. 5A and its training components, respectively, constructed and operative in accordance with a preferred embodiment of the present invention. [Figure 8] Figures 8A and 8B are block diagrams of further alternative embodiments of the simulator of Figure 2 using direct generative neural networks without and with latent variable models, respectively, constructed and operative in accordance with a preferred embodiment of the present invention. Figure 8C is a block diagram of the training portion of the direct generative neural network of Figures 8A and 8B. Figure 8D is a block diagram of the direct generative neural network of Figures 8A and 8B in operation. [Figure 9] FIG. 9 is a block diagram of a system for generating synthetic data for a machine learning system using the simulator of FIG. [Figure 10] FIG. 10 is a block diagram of a training aid for sonographers using the simulator of FIG. [Figure 11]FIG. 11 is a block diagram of an image enhancement system using the simulator of FIG.

[0042] It should be understood that for simplicity and clarity of illustration, elements shown in the figures have not necessarily been drawn to scale. For example, the dimensions of some of the elements may be exaggerated relative to other elements for clarity. Further, where considered appropriate, reference numerals may be repeated among the figures to indicate corresponding or analogous elements. DETAILED DESCRIPTION OF THE INVENTION

[0043] In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be understood by those skilled in the art that the present invention may be practiced without these specific details. In some instances, well-known methods, procedures, and components have not been described in detail so as not to obscure the present invention.

[0044] Applicant has recognized that there are thousands of 2D ultrasound clips available from clinics, hospitals, and the like, which are typically standard views of internal organs. For example, standard views of the heart include two-chamber, four-chamber, and five-chamber views. Some of these ultrasound clips have associated with them patient history and / or diagnostic results. This large amount of data can be easy and relatively inexpensive to collect compared to the cost of collecting 3D data.

[0045] Additionally, applicant has realized that U.S. Patent No. 11,593,638 by New York University and Yeda Research and Development Co. Ltd. (herein incorporated by reference), which provides for probe position and orientation changes to guide the user to a standard ultrasound view, can provide a database of 2D ultrasound images acquired from multiple ultrasound probe positions and orientations, and applicant has realized that the data in U.S. Patent No. 11,593,638 can provide a relatively simple source of probe data.

[0046] Applicant further recognized that easily acquired 2D ultrasound images can be used directly in a generative neural network to assist in the generation of synthetic ultrasound data, particularly for abnormal pathologies, and that synthetic ultrasound clips can be used for training sonographers. Additionally, easily acquired 2D ultrasound images may contain non-tissue elements, such as pacemakers and replacement valves, that are difficult to capture using 3D equipment such as MRI or CT.

[0047] Applicant has realized that generative neural networks can simulate the physics of both ultrasound beamforming and ultrasound propagation in biological tissue, which can provide faster image generation times (suitable for use in real-time systems) and greatly simplify the development process, thereby enabling the simulation of many different types of ultrasound examinations.

[0048] Furthermore, the applicant has realized that prior art methods are limited to simulating ultrasound clips only for specific people included in the 3D data. For example, if the available dataset contains data for person A with a certain BMI and a heart condition specific to person A, there is no way to generate ultrasound clips for a person with that heart condition and a different BMI.

[0049] Applicant has realized that a generative neural network can generate new samples that are similar, but not identical, to samples in the training data. Continuing with the example above, if a dataset has people with a range of BMI values, but only some of them have a particular disease, the network can generate samples with that disease for the entire range of BMIs.

[0050] Furthermore, because the present invention uses 2D ultrasound data rather than 3D data, it allows for a much greater variety of patients and situations than the prior art. Thus, the generative neural network can model the shape space of human organs and generate ultrasound images of various shapes and conditions without collecting data from specific people with specific parameter sets and / or diagnostic outcomes.

[0051] 2, a simulator 20 for ultrasound images and clips is shown that generates ultrasound clips from input probe position and orientation data associated with a particular 2D view or slice of an organ, as well as probe parameters and patient data, such as the patient's BMI, the patient's age, any diagnosed diseases the patient may be suffering from, etc. Other parameters may include operator parameters and noise parameters.

[0052] According to a preferred embodiment of the present invention, simulator 20 may be implemented as a generative neural network-based simulator trained with multiple 2D ultrasound images of at least one body part or organ acquired at associated probe positions and orientations, as well as probe parameters and patient data.

[0053] Applicant has realized that each ultrasound image in the training data set can be mapped to a slice in 3D space at a point in time. This is illustrated in Figure 3, which shows a heart 21 and an ultrasound view 23 through which it has been sliced. View 23 shows a standard four-chamber view of heart 21.

[0054] Thus, the training data represents multiple different views of the same 4D space (three spatial dimensions and one temporal dimension) or organ. Referring to Figure 4, an exemplary training data set of multiple views 27 sliced ​​through the same organ 29 is shown.

[0055] Furthermore, the applicant has realised that it is possible to train a generative neural network on such training data sets and generate 3D models of organs 29 therefrom.

[0056] Figure 5A shows one embodiment of the simulator 20 of Figure 2, and Figure 5B shows the simulator 20 during its training process. The simulator 20 can include an ultrasound-trained volumetric neural network 22 that generates an approximate 3D model of an organ 29, an ultrasound probe slicer 24 that "slices" the 3D model along the ultrasound transducer's line of sight to generate views 27, and an ultrasound-trained rendering neural network 26 that renders an ultrasound image for each of the views 27 along with any effects that occur on the ultrasound image.

[0057] Because generating a 3D model is analytically difficult, the ultrasound-trained volumetric neural network 22 functions as a nonlinear volumetric function φ(p,y,z) to generate the 3D model, as described below, where p is a 4D spatiotemporal location, y is an input parameter, and z is an optional latent variable. The input parameters can be ultrasound transducer parameters, patient data, the organ being observed and its characteristics, relative time / sequence data, pathology (if present), and other physical characteristics. Exemplary transducer parameters can include the probe position and orientation relative to the organ, the shape of the ultrasound field of view, and the type of device / transducer. Exemplary patient data can include gender, age, BMI, and specific diseases. Exemplary operator parameters can include the steadiness of the operator's hands. Additionally, there may be a noise parameter that determines how clear the ultrasound image is.

[0058] The ultrasound-trained volumetric neural network 22 may also be implemented as an MLP (multi-layer perceptron neural network) or learnable parameters on a voxel grid in 4D space-time.

[0059] As is known in the art, neural networks of any architecture contain a number of numerical parameters known as "weights," "learnable parameters," or simply "network parameters," denoted as θ. The output of a given neural network architecture for a particular input depends on the particular value of θ. When a neural network is first initialized, its learnable parameters are random and no desired output is obtained. During training, these learnable parameters are determined.

[0060] All neural networks must go through an optimization process (commonly called the "training process"), in which the learnable parameters are adjusted using a loss function that quantifies the difference between the desired output provided by the training data and the network's (initially random) output. The optimization process uses the loss function to compare the network's actual output with the desired output and quantify the difference between them. Next, the derivative of the loss function with respect to each network learnable parameter is numerically calculated. The derivative shows how the value of the loss function changes when a particular learnable parameter is changed. The learnable parameters are then changed so that the loss function decreases. This step is repeated many times for all examples in the training data, resulting in a neural network that gives the desired output for a given input.

[0061] The ultrasound probe slicer 24 can define a 2D view of the probe pointing at the organ 29, thereby selecting points of a 3D model of the organ 29 that form the ultrasound image field of view. The ultrasound probe slicer 24 can provide the selected points pi to the rendering neural network 26.

[0062] Given the probe position and orientation, the ultrasound probe slicer 24 is tasked with determining which points in 3D space are relevant to forming the ultrasound image. This is performed using a form of ray casting, a computer graphics technique that simulates the propagation of light toward a camera to render a 2D image from a 3D scene. However, the ultrasound probe slicer 24 sends rays from the probe position through the 3D volume in directions determined by the probe orientation in a pattern that matches the transmission of ultrasound waves, and selects points along those rays to represent pixels in the 2D ultrasound image.

[0063] The ultrasound-trained rendering neural network 26 can be implemented as a transformer neural network, using models such as a transformer to perform local (single point p i ) or point clouds at once. This is described in the paper "Attention is All You Need" by A. Vaswani, N. Shazeer, N. Parmar, et al., in Advances in Neural Information Processing Systems, I. Guyon, U.V. Luxburg, S. Bengio, et al., Eds., vol. 30, Curran Associates, Inc., 2017. Models that operate on a set of points can model the directional dependency of image formation due to effects such as ultrasound shadowing and refraction. This is described in the paper "The basics of ultrasound" by M. Demi, in Comprehensive Biomedical Physics, A. Brahme, Ed., Oxford: Elsevier, 2014, pp. 297-322, ISBN: 978-0-444-53633-4.

[0064] For training the simulator 20, the probe data, patient data and other parameters, as well as the ultrasound images or clips associated with that data, may be provided from any source. For example, probe position and orientation data may be obtained from a probe motion sensing system, such as the 3D Guidance trakSTAR available from Northern Digital Inc., Canada, and the ultrasound images or clips may be obtained from an ultrasound machine.

[0065] 5B illustrates an alternative embodiment in which the ultrasound images or clips and the position and orientation of the probe that generated them are provided by an ultrasound guidance system 30 such as that described in U.S. Pat. No. 11,593,638. The neural network-based ultrasound guidance system 30 can guide a user (either a trained sonographer or an untrained user) to move the ultrasound probe to a desired position and orientation to observe standard views of the organ 29. In accordance with a preferred embodiment of the present invention, the ultrasound guidance system 30 can generate probe position and orientation information associated with the ultrasound images or clips in the standard views and / or during movement from the current probe position and orientation to the desired position and orientation.

[0066] For training, the ultrasound guidance system 30 can provide probe position and orientation information as input to the volumetric neural network 22 and provide associated ultrasound images or clips to the loss function evaluator 28. The volumetric neural network 22 can also receive associated patient data and probe parameters.

[0067] 5B, in the case of simulator 20, optimizer 31 can update both neural networks 22, 26 with the derivative of the loss function value generated by loss function evaluator 28. Applicant has realized that because the generated ultrasound image is affected by both rendering neural network 26 and volumetric neural network 22, it is possible to take the derivative of the loss value with respect to the learnable parameters of both of those networks. Using the derivative of the loss function with respect to the learnable parameters, it is possible to modify those learnable parameters so that the generated ultrasound image matches the desired output.

[0068] The training data may consist of pairs of inputs and desired outputs, where the inputs may be input probe information and patient data, and the desired outputs may be associated ultrasound images.

[0069] The loss function evaluator 28 may receive relevant ultrasound images or clips from the ultrasound guidance system 30 and simulated ultrasound images or clips that are the output of the rendering neural network 26 and generate values ​​for the loss function. In one embodiment of the present invention, the volumetric neural network 22 and the rendering neural network 26 may be trained using an empirical risk minimization (ERM) training procedure of the following form: TIFF2025532575000002.tif18170 Also, x i is the input to the simulator 20, Φ(y i , z i ) is the output of the simulator 20, and m is the number of examples used in the training batch.

[0070] The loss function can be written as follows: TIFF2025532575000003.tif9170An example loss function that measures the difference between the related ultrasound image x and the simulated ultrasound image x' is given by: TIFF2025532575000004.tif9170 where, I x and I x’ are the intensities of the I-th pixel in images x and x', respectively.

[0071] After a number of iterations, the simulator 20 is trained, at which point probe data is provided to the simulator 20 from the guidance system 30 or other source of probe data, which generates simulated ultrasound images or clips.

[0072] Cardiac ultrasound images were generated by training the exemplary simulator 20 using training data from one patient. The training data consisted of four clips (each with 32 frames) of standard cardiac views: apical four-chamber view (designated 4C), two-chamber view (designated 2C), five-chamber view (designated 5C), and three-chamber view (designated 3C). The clips were annotated with the probe orientation and position relative to the 4C view, which was defined as the reference coordinate system. The clips were also annotated with a timestamp within the cardiac cycle, normalized to the interval (0,1).

[0073] The volumetric neural network 22, implementing the function φ1, was implemented as an MLP with five layers of 512 neurons. The input to this network was the probe point location in space and time, and its output was a 512-dimensional vector. The rendering neural network 26, implementing the function φ2, was implemented as a convolutional neural network (CNN). After selection by the ultrasound probe slicer 24, the 512-dimensional vector from each selected point was provided to the rendering neural network 26, which generated the final image.

[0074] 6 shows the synthetic sequence generated by simulator 20. Row 54 shows the direction of the ultrasound field of view in each frame. It will be appreciated that the images in the synthetic sequence are highly realistic ultrasound images.

[0075] Although not shown in FIG. 6 , it will be appreciated that simulator 20 can generate ultrasound images of "intermediate" probe positions not found in the original ultrasound clips. This can be particularly useful for observing various diseased body parts where there are only a few original ultrasound clips for training, typically only standard views. In such situations, simulator 20 can generate non-standard views of the affected body part.

[0076] Similarly, simulator 20 can generate various body parts for people with various BMIs, even if the training data includes only a few ultrasound clips from people with extreme BMIs. Thus, simulator 20 can generate synthetic ultrasound clips for various body sizes, thereby modeling the shape space of human organs. As a result, simulator 20 can generate synthetic ultrasound images for a variety of shapes and diseases.

[0077] Additionally, applicant has realized that if simulator 20 is provided with a small number of training images or clips of a body part having an artificial or non-human element (e.g., a pacemaker in a heart, a pin in a bone, etc.), simulator 20 can generate ultrasound images of the body part including the artificial element for other patients, including patients having other parameters.

[0078] As will be explained in more detail below, simulator 20 can greatly simplify the ultrasound method and equipment development process because it can easily simulate many different types of ultrasound examinations on patients with many different parameters, typically fast enough to be used in real-time systems.

[0079] Applicant has realized that simulator 20 can generate clips that provide a blend (sum or average) of input clips, rather than simulated clips that are more specific to a particular patient parameter (such as BMI). For example, a clip of an obese person can be blended with a clip of a thin person.

[0080] 7A and 7B, a simulator (here designated 20′) includes an operational latent variable model provider 32 and a training latent variable model neural network 33, both of which can add distributions for specific parameters. This ensures that the simulated ultrasound image more closely resembles the actual image of the patient with the selected parameters, rather than receiving the average value for all patients with the selected parameters.

[0081] To achieve this, during training, the latent variable model neural network 33 can add a random variable Z having a normal distribution to the input of the neural network 22, 26. The random variable Z has multiple elements z i , each capable of handling uncertainty in one of many parameters, such as the actual shape of the organ 29, the movement of the organ within the patient, uncertainty in the transducer position, uncertainty in time, and variability in patient size.

[0082] During training (FIG. 7B), the latent variable model neural network 33 receives 3D probe data, such as from the guidance system 30 (FIG. 5B), patient data, and associated ultrasound clips, and generates the 3D probe data y i and ultrasound image or clip x i For each pair, a number z between 0 and 1 is assigned. i, which results in a normal distribution of the random variable (e.g., mean 0, standard deviation 1). For example, common cases may be in the center of the distribution, and rare cases may be in the tails of the distribution. The latent variable model neural network 33 can be used to generate similar (x i ,y i ) pair has the same random variable z i The random variable z is assigned i can be assigned.

[0083] A latent variable model neural network 32 can be implemented by characterizing a vector model using standard images, as described in the paper by J. Ho, C. Saharia, W. Chan, D.J. Fleet, M. Norouzi, and T. Salimans, "Cascaded diffusion models for high fidelity image generation," J. Mach. Learn. Res., vol. 23, pp. 47-1, 2022, and by R.T.Q. Chen, Y. Rubanova, J. Bettencourt, and D.K. Duvenaud, "Neural Ordinary Differential Equations," in Advances in Neural Information Processing Systems, vol. 31, Curran Associates, Inc., 2018. Alternatively, the latent variable model neural network 32 can be trained separately as part of a variational autoencoder (VAE), as described in the paper by D.P. Kingma and M. Welling, “An introduction to variational autoencoders,” Foundations and Trends® in Machine Learning, vol. 12, no. 4, pp. 307-392, 2019.

[0084] In FIG. 7B, a latent variable model neural network 33 is trained as part of simulator 20′, and in this embodiment, an additional term is added to the loss function in Equation 2: The additional loss term is added to TIFF2025532575000005.tif9170. i The probability distribution of is simplified / normalized (e.g., normal or another desired distribution) and is guaranteed to maximize the likelihood of the training data given the trained generator and encoder. This is the standard procedure for training latent variable models.

[0085] An example of these additional loss terms is the KL divergence loss. A latent variable model neural network33 assigns a random value z to each entry in the training data using the same procedure as a variational autoencoder. i , each entry in the training dataset has an associated mean μ and standard deviation σ value, which are used by the latent variable model neural network 33 to calculate z i The parameter μ for each random variable is used to generate the value i and σ i is fed into the KL loss term as follows: TIFF2025532575000006.tif9170 where the sum is over the different training samples. This loss term is minimized when the mean (over the training dataset) of the μ values ​​is 0 and the mean of the σ values ​​is 1. This loss term, combined with the reconstruction loss described above, finds a z function such that both samples are correctly reconstructed and each entry has a normal distribution. i Ensures that a value is assigned.

[0086] After training, when simulator 20′ is running (FIG. 7A), latent variable model neural network 33 can be replaced by latent variable provider 32 to provide simulator 20′ with randomly sampled values ​​of z according to its distribution.

[0087] A second embodiment, shown in Figures 8A and 8B, illustrates a simulator 40 / 40' that, like the previous embodiment, can be implemented using a single direct ultrasound-trained generative neural network 42 trained on 2D ultrasound images or clips and associated data. Figure 8B illustrates an embodiment with a latent variable model neural network 32, while Figure 8A illustrates an embodiment without it.

[0088] The simulator 40 / 40' can be implemented as a diffusion model generative neural network that, in response to input data, generates a trained neural network that takes an image of random noise and transforms the image of random noise into a desired image through multiple "de-noising" steps. In accordance with the present invention, the direct ultrasound trained generative neural network 42 can generate an ultrasound image from an image of random noise that is associated with input probe parameters, patient parameters, and probe position and orientation.

[0089] To create neural network 42, it must first be trained. According to a preferred embodiment of the present invention, similar to the previously described embodiments, the training data set includes pairs of ultrasound clips and associated probe parameters and / or position and orientation and / or patient parameters.

[0090] Referring to FIG. 8C, the elements of a trainer 45 for training the direct ultrasound generation neural network 42 are shown. These elements are a noise adder 50, an MSE (mean squared error) loss calculator 52, an optimizer 54, and a noise prediction neural network 56. Each training sample may include an actual 2D ultrasound clip and its associated probe position and orientation, as well as associated patient, noise, or operator parameters, which are generally shown as "parameters" in FIG. 8C. The training process of FIG. 8C can be repeated multiple times to convert input 2D ultrasound clips into random images.

[0091] The noise adder 50 can add a random amount of noise or random distortion to the ultrasound clip for each training sample received. The noise adder 50 can have a predetermined procedure for adding a fixed amount of noise to each clip, and repeat this procedure a random number of times N to generate a random amount of noise to the clip. i In early iterations, the noise adder 50 may add a small fixed amount of noise so that the clip does not change much with the added noise. In subsequent iterations, if the clip image is already very noisy, the fixed amount of noise may be increased. The noise adder 50 may keep track of the amount of noise last added to the clip.

[0092] The noise prediction neural network 56, which may be implemented as a convolutional neural network, receives both the noisy clip and its associated general probe parameters and is able to predict the noise that will be added to the clip in the final iteration.

[0093] MSE loss calculator 52 may use the MSE loss to compare the predictions made by noise prediction neural network 56 with the final noise actually added to the clip as provided by noise adder 50. Optimizer 54 may then use the MSE loss to update the weights of noise prediction neural network 56. MSE loss calculator 52 and optimizer 54 may run multiple times until noise prediction neural network 56 roughly accurately predicts the noise added to the clip by noise adder 50 at each iteration.

[0094] The trainer 42 can repeat the operation many times (e.g., 100-1000 or more times) for different batches of training samples and different amounts of noise addition steps, until a large amount of noise is added to each clip, resulting in completely random clips. At this point, the noise prediction neural network 56 can be trained and used in the direct ultrasound generation neural network 42 to generate synthetic ultrasound clips for a common input set of probe parameters.

[0095] Referring to FIG. 8D, elements of the direct ultrasound generation neural network 42 are shown that use a noise prediction neural network 56 and a noise prediction remover 58. Starting with a clip of completely random noise, the noise prediction neural network 56 predicts the noise to be removed, and the noise prediction remover 58 can remove the predicted noise from the clip. This process can be repeated multiple times, for as many steps as used by the trainer 45. In this way, noise can be gradually removed from the initial random image until a synthetic ultrasound clip is generated for the received typical probe parameters.

[0096] The noise adder 50 may implement the following exemplary noise addition procedure. TIFF2025532575000007.tif9170where x t+1is the clip with noise added, x t is the clip before noise is added, ε is a random noise value generated from a standard normal distribution, β t is a parameter that defines the amount of random noise added to the clip at iteration t.

[0097] The noise adder 50 adjusts β t can be varied, and β t An exemplary schedule for is as follows: TIFF2025532575000008.tif13170Here, β end and β start is a predefined fixed value, and T is the maximum number of iterations that the noise adder 50 can perform.

[0098] Any of the trained versions of simulators 20, 20', 40, 40' can be used in many different applications, as shown in Figures 9, 10, and 11. Although the following description describes simulator 20, it will be understood that any of the simulators described above could be used instead.

[0099] 9 illustrates a system for generating synthetic data for a machine learning system. In this embodiment, one or more parameters in addition to the probe position and orientation are provided to a simulator 20, which generates a synthetic ultrasound image of the body part relative to the position and orientation.

[0100] For example, one type of machine learning system may be a classifier for classifying aspects of a synthetic image or clip generated by simulator 20, a segmenter for segmenting the image into objects therein, and a regressor. For each of these, the user may convert the position and orientation of the probe, and / or any of the other parameters, into annotations to provide the classification, segmentation, or regression.

[0101] In an alternative embodiment, one type of machine learning system may be an ultrasound navigation neural network 100 under training, as shown in FIG. 9. The navigation unit 100 uses the synthetic ultrasound images to determine training movement instructions for the probe to obtain a desired view of the body part. The unit 100 may convert the movement instructions into an updated probe position and provide the position to the simulator 20 to generate the next synthetic ultrasound image. The unit 100 may include a loss function and a training unit to update its neural network based on the results.

[0102] 10, a physical device 102 with position / orientation input sensors can be used in conjunction with simulator 20 to generate ultrasound images in real time that correspond to the sensor position / orientation and patient and / or probe parameters. This system can be used as a training aid for sonographers in conjunction with sonographer training software 104.

[0103] The trainee can move the probe and provide its position to the simulator 20, which in turn can generate an ultrasound image of the body part associated with that position. The ultrasound image is then provided to the sonographer training software 104.

[0104] In accordance with a preferred embodiment of the present invention, as shown in FIG. 11 , simulator 20 can be utilized to correct poor quality ultrasound images. In this embodiment, which includes simulator 20 with external optimizer 110, a partial or corrupted image is provided to comparator 112, which can determine the difference between the partial or corrupted image and a composite image generated by simulator 20. External optimizer 110 can update a set of simulation parameters for simulator 20 based on the difference generated by comparator 112. The simulation parameters can be any of the parameters described above, as well as unknown probe position and orientation.

[0105] Simulator 20 then generates an updated composite image for comparator 112, and this process is repeated multiple times, typically until there is only a slight improvement in the composite image, at which point simulator 20 outputs the current composite image, which is an improved version of the poor image.

[0106] The system of Figure 11 can be utilized to provide image completion and / or image correction and / or image noise reduction and / or 3D reconstruction and / or new view synthesis when there is sparse or corrupted ultrasound image data. The system of Figure 11 can be used with any imaging data, such as TEE (transesophageal echocardiography), TTE (transthoracic echocardiography), ICE (intracardiac echocardiography), CT, MRI, nuclear medicine imaging, X-ray, fluoroscopy, etc.

[0107] The optimizer 110 can implement gradient descent optimization on the simulation parameters, which may initially have random values, and can change those values ​​to make the composite image more similar to the subimages.

[0108] Unless otherwise indicated, and as is apparent from the above description, it should be understood that throughout this specification, descriptions using terms such as "processing," "operation," "calculation," "determination," and the like refer to the operations and / or processes of any type of general-purpose computer, such as a client / server system, mobile computing device, smart appliance, cloud computing unit, or similar electronic computing device, that manipulates and / or transforms data in the registers and / or memory of a computing system into other data in the memory, registers, or other information storage, transmission, or display device of the computing system.

[0109] Embodiments of the present invention may include an apparatus for performing the operations described herein. This apparatus may be specially constructed for a desired purpose, or may comprise a computing device or system, typically having at least one processor and at least one memory, selectively activated or reconfigured by a computer program stored on the computer. The resulting apparatus, when instructed by software, can transform a general-purpose computer into an inventive element as described herein. The instructions can define an inventive device that operates on a desired computer platform. Such a computer program may be stored on a computer-readable storage medium, including, but not limited to, any type of disk, e.g., optical disk, magneto-optical disk, read-only memory (ROM), volatile and non-volatile memory, random access memory (RAM), electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic or optical card, flash memory, disk-on-key, or other type of medium suitable for storing electronic instructions and capable of coupling to a computer system bus. The computer-readable storage medium may also be implemented with cloud storage.

[0110] Some general purpose computers may include at least one communications element to enable communication with data networks and / or mobile communications networks.

[0111] The processes and displays presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems can be used with programs in accordance with the teachings herein. Alternatively, it may be more convenient to construct a more specialized apparatus to implement the desired method. The desired structure for a variety of these systems will appear from the description below. Moreover, embodiments of the present invention are not described with reference to any particular programming language. It will be understood that a variety of programming languages ​​can be used to implement the teachings of the present invention, as described herein.

[0112] While certain features of the invention have been illustrated and described herein, many modifications, substitutions, changes and equivalents will occur to those skilled in the art. It is therefore to be understood that the appended claims are intended to cover all such modifications and changes as fall within the true spirit of the invention.

Claims

1. 1. An ultrasound image simulator, comprising: a generative neural network that receives the position and orientation of the ultrasound probe and generates at least one simulated ultrasound image or clip of the subject's body part; 10. The simulator of claim 9, wherein the generative neural network is trained with multiple 2D ultrasound images or clips of the body part acquired from multiple ultrasound probe positions and orientations.

2. 2. The simulator of claim 1, The simulator, wherein the positions and orientations of the plurality of 2D ultrasound images or clips and the plurality of ultrasound probes are generated by an ultrasound guidance system.

3. 2. The simulator of claim 1, The simulator, wherein the positions and orientations of the plurality of ultrasound probes are generated by a probe motion detection system.

4. 2. The simulator of claim 1, A simulator wherein the image or clip is associated with a plurality of parameters.

5. 5. The simulator according to claim 4, The simulator, wherein the parameters include at least one of an ultrasound probe parameter, a patient parameter, an operator parameter, and a noise parameter.

6. 6. The simulator according to claim 5, A simulator wherein said at least one simulated ultrasound image or clip includes artificial or non-human elements therein.

7. 2. The simulator of claim 1, The simulator, wherein the generative neural network further comprises a latent variable provider.

8. 8. The simulator according to claim 7, The simulator, wherein the generative neural network is trained with a latent variable neural network that receives the plurality of 2D ultrasound images or clips and the positions and orientations of the plurality of ultrasound probes.

9. 2. The simulator of claim 1, The simulator, wherein the generating neural network comprises a diffusion model neural network.

10. 2. The simulator of claim 1, the generative neural network: a volumetric neural network that receives the position and orientation of the ultrasound probe and generates volumetric data of a body part for the position and orientation of the ultrasound probe; a slicer for extracting a slice of a volume related to the position and orientation of the ultrasound probe; a rendering neural network that renders the slices as a simulated ultrasound image of the body-part.

11. The simulator of claim 10, A simulator, wherein the volumetric neural network and the rendering neural network are both trained with an empirical risk minimization training procedure and utilize the same loss function.

12. 1. A system for generating synthetic data for training an ultrasound-based machine learning unit, comprising:

5. A system comprising the simulator of claim 4, wherein the simulator receives at least one of the plurality of parameters and generates at least one simulated ultrasound image or clip for the at least one parameter.

13. 13. The system of claim 12, The system, wherein the ultrasound-based machine learning unit is one of a classifier, a segmenter, and a regressor.

14. 13. The system of claim 12, The system, wherein the ultrasound-based machine learning unit comprises a navigation neural network under training for generating probe movement instructions for an ultrasound probe on a virtual subject, and a probe position updater for converting the movement instructions into a position and orientation of the ultrasound probe.

15. 1. A system for creating synthetic data for training a person to perform ultrasound scans of an artificial body with an ultrasound probe, comprising:

5. The simulator of claim 4, wherein the simulator receives at least one of the plurality of parameters and generates at least one simulated ultrasound image or clip for the at least one parameter; and sonographer training software that receives the at least one simulated ultrasound image or clip and provides instructions to the person.

16. 1. A system for enhancing partial or corrupted images of a body part, comprising:

5. The simulator of claim 4, wherein the simulator generates a simulated ultrasound image of the body part; a comparator that determines the difference between the simulated ultrasound image and the partial or corrupted image; an optimizer that updates the plurality of parameters and / or the position and orientation of the ultrasound probe to reduce the difference, thereby generating an improved version of the partial or corrupted image of the body part.

17. 17. The system of claim 16, A system wherein the output of said system is image completion and / or image correction and / or image noise reduction and / or 3D reconstruction and / or synthesis of new views.

18. 1. A method for producing an ultrasound image, comprising: generating, by a generative neural network, at least one simulated ultrasound image or clip of the subject's body-part in response to a position and orientation of the ultrasound probe; The method, wherein the generative neural network is trained with multiple 2D ultrasound images or clips of the body part acquired from multiple ultrasound probe positions and orientations.

19. 20. The method of claim 18, generating the plurality of 2D ultrasound images or clips and the positions and orientations of the plurality of ultrasound probes with an ultrasound guidance system.

20. 20. The method of claim 18, generating, by a probe motion sensing system, positions and orientations of the plurality of ultrasound probes.

21. 20. The method of claim 18, The method, wherein the image or clip is associated with a plurality of parameters.

22. 22. The method of claim 21, The method, wherein the parameters include at least one of an ultrasound probe parameter, a patient parameter, an operator parameter, and a noise parameter.

23. 23. The method of claim 22, The method, wherein the at least one simulated ultrasound image or clip includes an artificial or non-human element therein.

24. 20. The method of claim 18, The method further comprising providing latent variables to the generative neural network.

25. 25. The method of claim 24, training the generative neural network with a latent variable neural network that receives the plurality of 2D ultrasound images or clips and the positions and orientations of the plurality of ultrasound probes.

26. 20. The method of claim 18, The method, wherein the generative neural network comprises a diffusion model neural network.

27. 20. The method of claim 18, The generating step includes: generating volumetric data of a body part for the position and orientation of the ultrasound probe by a volumetric neural network in response to the position and orientation of the ultrasound probe; extracting a slice of a volume associated with the position and orientation of the ultrasound probe; and rendering the slices as a simulated ultrasound image of the body-part with a rendering neural network.

28. 28. The method of claim 27, The method further comprising training both the volumetric neural network and the rendering neural network with an empirical risk-minimization training procedure that utilizes the same loss function.

29. 1. A method for generating synthetic data for training an ultrasound-based machine learning unit, comprising: receiving at least one of a plurality of parameters; and generating at least one simulated ultrasound image or clip for the at least one parameter using the generative neural network of claim 21.

30. 30. The method of claim 29, The method, wherein the ultrasound-based machine learning unit is one of a classifier, a segmenter, and a regressor.

31. 30. The method of claim 29, The method, wherein the ultrasound-based machine learning unit includes a navigation neural network under training to generate probe movement instructions for an ultrasound probe on a virtual subject, the method including converting the movement instructions into a position and orientation of the ultrasound probe.

32. 1. A method of creating synthetic data for training a person to perform ultrasound scans of an artificial body with an ultrasound probe, comprising: receiving at least one of a plurality of parameters; generating at least one of the simulated ultrasound images or clips for the at least one parameter using a generative neural network as defined in claim 21; and wherein the sonographer training software provides instructions to the person.

33. 1. A method for improving a partial or corrupted image of a body part, comprising: generating at least one simulated ultrasound image or clip using the generative neural network of claim 21; determining a difference between the partial or corrupted image and the simulated ultrasound image; updating a plurality of parameters and / or the position and orientation of the ultrasound probe to reduce the difference, thereby generating an improved version of the partial or corrupted image of the body part.

Images