Motion correction of ultrasound images using machine learning and a reference image selected as a ground truth image

Abstract

Motion correction in ultrasound imaging uses a suitably trained machine learning algorithm or model to correct for tissue motion in ultrasound images. A reference frame is selected from the input image data to use as a ground truth input to the machine learning model.

Description

Mayo 2023-353 630666.01539 MOTION CORRECTION OF ULTRASOUND IMAGES USING MACHINE LEARNING AND A REFERENCE IMAGE SELECTED AS A GROUND TRUTH IMAGE CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Patent Application Serial No. 63 / 553,045, filed on February 13, 2024, and entitled “MOTION CORRECTION OF ULTRASOUND IMAGES USING MACHINE LEARNING AND A REFERENCE IMAGE SELECTED AS A GROUND TRUTH IMAGE,” which is herein incorporated by reference in its entirety. STATEMENT OF FEDERALLY SPONSORED RESEARCH

[0002] This invention was made with government support under CA239548 and CA195527 awarded by the National Institutes of Health. The government has certain rights in the invention. BACKGROUND

[0003] Motion between successive frames in ultrasound imaging can significantly contaminate the acquired data, which causes difficulties in subsequent analysis for different applications, such as extraction of microvasculature. The motion can result from different causes, such as probe motion, respiratory motion, rigid and non-rigid motion, in-plane motion, out-of-plane motion, and so on. When such motion occurs during data acquisition, it causes difficulties in the quantification process and leads to degradation of image quality that can further lead to false diagnostic outcomes in different applications. SUMMARY OF THE DISCLOSURE

[0004] It is an aspect of the present disclosure to provide a method for reducing motion in ultrasound images. The method includes accessing ultrasound image data with a computer system, wherein the ultrasound image data comprise a sequence of ultrasound image frames containing motion; selecting a reference image from the ultrasound image data; accessing a machine learning model with the computer system, wherein the machine learning model has been trained on training data to correct motion in ultrasound images; inputting the ultrasound image data and the reference image to the machine learning model with the computer system, generating motion-corrected ultrasound image data as an output, wherein the reference image 1 QB\630666.01539\94528245.3Mayo 2023-353 630666.01539 is used as a ground truth image by the machine learning model; and outputting the motion- corrected ultrasound image data via the computer system.

[0005] It is another aspect of the present disclosure to provide a method for reducing motion in medical images. The method includes accessing medical image data with a computer system, where the medical image data include a sequence of image frames containing motion. A reference image is selected from the medical image data. A machine learning model is also accessed with the computer system, where the machine learning model has been trained on training data to correct motion in medical images. The medical image data and the reference image are input to the machine learning model with the computer system, generating motion- corrected medical image data as an output. The reference image is used as a ground truth image by the machine learning model. The motion-corrected medical image data are then output via the computer system. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] FIG. 1A is an example motion correction separable convolutional network (MoCoSCNet), showing corresponding inputs and a method for selecting a reference image as a ground truth image for the network.

[0007] FIG. 1B is an example inter-frame motion correction network (IFMoCoNet), showing corresponding inputs.

[0008] FIG. 2 illustrates an example of a general convolution operation for multi- channel input data.

[0009] FIGS. 3A–3B show depth-wise separable convolution operations for multichannel input data: (FIG.3A) Depth-wise convolution, (FIG.3B) Point-wise convolution.

[0010] FIG. 4 shows one example methodology for implementing a MoCoSCNet for performing motion correction using a two-stage motion correction for generating ground truth data.

[0011] FIG.5 shows another example MoCoSCNet architecture. Here, NF denotes the number of filters / kernels, and Fsdenotes the filter size.

[0012] FIG. 6 shows another example methodology for implementing a MoCoSCNet for performing motion correction using a Casorati correlation-based reference frame selection for generating the ground truth data.

[0013] FIG.7 is a flowchart setting forth the steps of an example method for generating motion-corrected ultrasound images using a MoCoSCNet. 2 QB\630666.01539\94528245.3Mayo 2023-353 630666.01539

[0014] FIG.8 is a flowchart of an example method for training a MoCoSCNet.

[0015] FIG. 9 shows MoCoSCNet evaluation metrics with respect to original metrics for all subjects in an example study: (a) PCC, (b) MSE, and (c) SSI.

[0016] FIG.10 shows percentage improvement in MoCoSCNet evaluation metrics with respect to original metrics for all subjects in an example study: (a) PCC, (b) MSE, and (c) SSI.

[0017] FIG. 11 is a block diagram of an example system for generating motion- corrected ultrasound images.

[0018] FIG. 12 is a block diagram of example components that can implement the system of FIG.11. DETAILED DESCRIPTION

[0019] Described here are systems and methods for performing motion correction in ultrasound imaging using a suitably trained machine learning algorithm or model to correct for tissue motion in ultrasound images. The disclosed systems and methods utilize a selected reference frame on which to base motion correction, followed by a motion correction algorithm of low enough complexity to be very fast. As an advantage, the disclosed systems and methods can correct an image on the order of 15 milliseconds, which allows for real-time motion correction. These motion correction techniques can advantageously work without the need for a contrast agent.

[0020] It is an aspect of the present disclosure to use machine learning models, such as those that implement deep learning, to significantly reduce the computational time for optimization while correcting motion in the successive frames. The disclosed systems and methods implement an optimal correlation-based selection of the reference frame and a deep learning-based approach for motion correction in ultrasound image sequences with significantly low computational complexity. As noted above, the approach is applicable in real- time settings and can efficiently correct the motion between successive frames in ultrasound imaging with significantly low computational time. In this context, in some implementations the disclosed systems and methods use a Casorati correlation matrix for the selection of a reference frame based on the maximum correlation. Different methodologies for training a machine learning model for motion correction based on the selection of ground truth data are also described.

[0021] It is an advantage of the disclosed systems and methods to implement an exhaustive selection of the reference frame for registration of other frames, unlike the existing 3 QB\630666.01539\94528245.3Mayo 2023-353 630666.01539 motion correction techniques, which typically choose the first frame as a reference. The selected reference frame can then be used as the ground truth for training the deep learning model, unlike existing motion correction techniques, which use conventional motion correction methods to produce ground truth data. The disclosed machine learning model can advantageously predict a motion corrected frame with significantly lower latency as compared to the existing motion correction techniques.

[0022] Although the systems and methods are described herein with respect to an example of ultrasound imaging, the machine learning model is non-limiting in terms of the imaging modality. As an example, the machine learning model in the present disclosure can be used to correct motion in magnetic resonance imaging (MRI), optical coherence tomography (OCT), and other such imaging modalities.

[0023] In some implementations, the disclosed systems and methods can be adapted for correcting inter-frame motion for contrast-free ultrasound quantitative microvessel imaging and other ultrasound imaging techniques. The inter-frame motion correction scheme uses a spatiotemporal correlation matrix, such as a Casorati correlation matrix, for the selection of a reference frame based on the maximum correlation, which is further used as the ground truth to train the proposed network for this dataset. The approach is applicable in real-time settings and can efficiently correct the motion between successive frames in ultrasound imaging, with significantly low computational time.

[0024] In some cases, the disclosed systems and methods implement a low complex deep learning network including depth-wise separable convolutional layers with hybrid adaptive and squeeze-and-excite attention mechanisms for inter-frame motion correction. As noted above, network training can be implemented using a reference frame obtained from a spatiotemporal correlation matrix as the ground truth. Additionally or alternatively, the ground truth may be obtained from other methods. In some cases, motion corrected frames can be predicted or otherwise generated with significantly low latency, as compared to the existing inter-frame motion correction techniques. Microvasculature Doppler images can be extracted or otherwise constructed from the motion-corrected frames.

[0025] An example machine learning model for generating motion corrected ultrasound images is shown in FIG.1A. The illustrated model architecture implements a deep learning network (e.g., a neural network) that may be referred to as a motion correction separable convolution network (MoCoSCNet). The neural network is built using a sequence of two-dimensional convolutional (conv2D), depth-wise separable convolutional (DSConv2D), 4 QB\630666.01539\94528245.3Mayo 2023-353 630666.01539 and maxpooling (Maxpool2D) layers. In the example illustrated in FIG.1A, the neural network includes two blocks of conv2D and DSConv2D layers, followed by two DSConv2D layers, and a final conv2D layer for prediction. Each DSConv2D layer is followed by a maxpool2D layer, and batch normalization (BN) is performed after each layer, except the final layer.

[0026] The motion containing B-mode images, in-phase and quadrature (IQ) frames, or other ultrasound images obtained from each acquisition are fed along with the selected reference frames as inputs to the first convolutional layer, which as a non-limiting example can include 64 kernels / filters each with a size of 3 × 3. This layer performs a convolution operation between each B-mode image matrix B and a kernel matrix K, to obtain the output C, as shown below: C^ p, q ^^ ^ K ^ B ^^ p , q ^ ^^ B ^ p ^ m , q ^ n ^ K ^ m , n ^ (1);m ^ n

[0027] mand n denote the indexes of rows and columns of the input B-mode matrix B. The kernel is shifted throughout the input matrix to obtain the output map, and its weights are learned during the training process. An example of this convolution operation for multi-channel data is illustrated in FIG.2. As illustrated in FIG.2, each input channel is convolved with a particular kernel, and the final output is obtained as the sum of the convolved outcomes of all channels. In some examples, He uniform initialization is used for the kernel weights, and rectified linear unit (ReLu) activation is applied to the convolved output with zero bias to obtain the outputfeature map of this layer. In general, the output feature map, C o , of layer l is obtained as:C ^ ^ K ^l ^ ^ B ^ l ^1 ^ ^ ^ l ^o ^ bs ^ (2);

[0028] output feature maps obtained from the conv2D layer are given as inputs to the second layer, i.e., depth-wise separable conv2D (DSConv2D). This layer factorizes the convolution operation into independent channel-wise (depth-wise) and spatial (pointwise) operations. The output feature maps obtained from the conv2D layer are fed as different channels to the DSConv2D layer. Firstly, each input channel is convolved with the kernel for depth-wise operation, and the corresponding outputs of all channels are stacked together to obtain the overall output, as illustrated in FIG.3A. Secondly, convolution is performed, such as with a 1 × 1 kernel for point-wise operation, as shown in FIG. 3B. This layer significantly reduces the number of mathematical operations and parameters required in the conv2D layer, making it 5 QB\630666.01539\94528245.3Mayo 2023-353 630666.01539 more efficient. This layer also uses ReLu activation and no bias vector, and is followed by a maxpooling layer that performs a downsampling operation, such as with a pool size of (2, 2). In a similar manner, the subsequent layers perform feature extraction sequentially and the output maps are fed to the output layer (e.g., a conv2D layer) for predicting the motion corrected frames, as shown in FIG.1A.

[0029] As described above, in some implementations the disclosed systems and methods can be adapted or otherwise implemented for inter-frame motion correction. In these cases, the machine learning model may be referred to as an inter-frame motion correction network (IFMoCoNet). An example machine learning model for generating inter-frame motion corrected ultrasound images is shown in FIG. 1B. The illustrated model architecture implements a deep learning network (e.g., a neural network). The neural network is built using a sequence of two-dimensional (conv2D) and depth-wise separable convolutional (DSConv2D) layers with a hybrid adaptive attention block (HAAB) and squeeze-and-excite attention mechanism arranged in an exhaustively selected sequence. Advantageously, the inclusion of hybrid adaptive blocks and squeeze-and-excite attention mechanisms may be used to correct inter-frame motion in ultrasound images. The network receives the input ultrasound image frames (e.g., B-mode images, IQ images) containing motion, along with the ground truth frames derived from either the original motion-free data or the selected reference frames. As noted above, in some cases the reference frames may be selected based on a spatiotemporal correlation matrix, such as a Casorati correlation matrix. After high-level feature extraction by the layers, the motion-corrected frames are obtained at the output layer, as shown in FIG.1B.

[0030] As shown in FIG.1B, in an example configuration the neural network includes two blocks of conv2D and DSConv2D layers, followed by a hybrid adaptive attention block (HAAB), two DSConv2D layers, a squeeze-and-excite attention mechanism, and a final conv2D layer for prediction. Each conv2D and DSConv2D layer is followed by batch normalization (BN) except the final layer. These layers are sequentially arranged for inter- frame motion correction. The motion containing ultrasound image data obtained from each acquisition are fed along with the selected reference frames as inputs to the first convolutional layer, which in the illustrated example includes 64 kernels / filters each with a size of 3 × 3. This layer performs convolution operation between each image frame matrix B and kernel matrix K, to obtain the output C, as indicated in Eqn. (1).

[0031] The kernel is shifted throughout the input matrix to obtain the output map, and its weights are learned during training process. This convolution operation for multi-channel 6 QB\630666.01539\94528245.3Mayo 2023-353 630666.01539 data is illustrated in FIG.2. Each input channel is convolved with a particular kernel, and the final output is obtained as the sum of the convolved outcomes of all channels. In this example, He uniform initialization is used for the kernel weights, and Gaussian error linear unit (GELU) activation is applied to the convolved output for randomly dropping some neurons and capturing more complex patterns, with zero bias to obtain the output feature map of this layer.In general, the output feature map, C o , of layer l is obtained as indicated in Eqn. (2).

[0032] The output feature maps obtained from the conv2D layer are given as inputs to the second layer (i.e., the depth-wise separable conv2D, DSConv2D). This layer factorizes the convolution operation into independent channel-wise (depth-wise) and spatial (pointwise) operations. In the illustrated example, the 64 output feature maps obtained from the conv2D layer are fed as different channels to the DSConv2D layer. First, each input channel is convolved with the kernel for depth-wise operation, and the corresponding outputs of all channels are stacked together to obtain the overall output. Second, standard convolution is performed with a 1 × 1 kernel for point-wise operation. This layer significantly reduces the number of mathematical operations and parameters required in the conv2D layer, making it more efficient. This layer also uses GELU activation and no bias vector.

[0033] In a similar manner, the subsequent conv2D and DSConv2D layers perform feature extraction sequentially and the output maps are then fed to a hybrid adaptive attention block (HAAB). The IFMoCoNet implements the HAAB for inter-frame motion correction in ultrasound images. In general, the HAAB includes conv2D layers with varying filter sizes, channel and spatial attention mechanisms for extracting robust features with different receptive fields and scales from the channel-level and spatial-level aspects.

[0034] The HAAB operates as follows. The input feature map F i is convolved with 3× 3, 5 × 5, and 3-rate dilated 3 × 3 conv2D layers in parallel to achieve H 3 , H 5 , and HD 5feature maps respectively, as shown in FIG.1B. Different receptive fields allow robust feature extractions and adaptations to a variety of images, such as varying levels of inter-frame motion related displacements. To capture the different receptive fields as mentioned above, the featuremaps H 5 and HD 5 are combined and downsized to 1 × 1 to obtain F 1 as:F1 ^ GAP ^ H 5 ^ HD 5 ^ (3);

[0035] addition operation. 7 QB\630666.01539\94528245.3Mayo 2023-353 630666.01539

[0036] The corresponding feature map F 1 is given as an input to a dense layer thatperforms a dot product between its weights and the input. The dense layer output undergoesbatch normalization and GELU activation to obtain a feature map, F 2 , as:F2 ^^g^ B n ^ W D ^ F 1 ^ ^ (4);

[0037] ^ ^ ^ denote GELU and batch normalization operations,respectively, and W D denotes the weight matrix of dense layer.

[0038] The preceding feature map, F 2 , is again fed to a dense layer and sigmoidactivation to generate the channel attention weights a and 1^ a for H 5 and HD 5 ,respectively, where a is obtained as:a^^s ^ W D ^ F2 ^ (5);

[0039]

[0040] attention weights, a and 1^ a , are then used to scale thefeature maps H 5 and HD 5 , respectively, to select the features from different receptive fieldsautomatically, as: FHD 3 ^ a ^ H D 3 (6);FH5 ^ ^1 ^ a ^ ^ H 5 (7).

[0041] This channel-level attention mechanism helps to extract more robust features which can adapt to the variations in the displacements due to inter-frame motion.

[0042] The channel attention features are finally combined to form a feature map, H1C, which is used with the 1 × 1 convolution output of H 3 (i.e., H 1 ) as input to the next stage ofthe HAAB; that is, the spatial-level attention mechanism,in FIG.1B.

[0043] The preceding inputs undergo GELU operation and 1 × 1 convolution to obtaina feature map, H 1S , as:H1S ^ conv2 D 1^ 1^^ g ^ H 1 ^ H 1 C ^ ^ (8).

[0044] level attention weights b and 1^ b for H 3 and H 1C , respectively. These weights are8 QB\630666.01539\94528245.3Mayo 2023-353 630666.01539resampled to the same number of channels as H 3 and H 1C and then multiplied by thesefeatures to obtain the spatial-attention feature maps as: FH3 ^ ^1 ^ b ^ ^ H 3 (9);FH 1 C ^ b ^ H 1 C (10).

[0045] The overall output of the HAAB (i.e., H 1H ) is then obtained as:H1H ^ conv2 D 1^ 1 ^ F H 3 ^ F H 1 C ^ (11).

[0046] feature map, motion. In the illustrated example, the output of the HAAB is again fed to two DSConv2D layers with batch normalization as shown in FIG.1B. The corresponding output may be further refined to obtain channel-level features using a squeeze-and-excite-attention block as shown in FIG. 1B. This block adds an attention mechanism to the output feature maps for considering the inter-dependencies between them to improve the overall network performance. The squeeze-and-excite-attention mechanism is explained as follows.

[0047] First, each output feature map is squeezed by converting the feature map to a single numeric value using GAP as shown in FIG. 1B. This operation returns the spatialaverage, ^ , of the feature matrix. In this way, the squeeze operation returns a vector with thesize as the number of output feature maps, which is 64 in the illustrated example.

[0048] In the excite mechanism, the squeezed output vector is fed into a dense layer with a GELU activation function. The corresponding output is fed to another dense layer with sigmoid activation, which captures complex inter-dependencies and correlations among the feature maps effectively. The corresponding output is the same size as the vector input, containing the learned relevant weights for each feature map.

[0049] The outputs of the excite mechanism are then multiplied with the corresponding original convolutional feature maps to provide weighted adaptations for enhancing the relevant maps and suppressing the less-relevant maps. This adds an effective attention mechanism to further enhance the performance of the illustrated neural network. Finally, the weighted output of the squeeze-and-excite attention mechanism may be given as an input to the final layer of the IFMoCoNet network (i.e., conv2D with a single filter for predicting the motion corrected frames), as shown in FIG.1B.

[0050] As described above, a reference frame is selected to be used as the ground truth in training the machine learning model for motion correction, which may be used for training 9 QB\630666.01539\94528245.3Mayo 2023-353 630666.01539 either the MoCoSCNet network illustrated in FIG. 1A or the IFMoCoNet network illustrated in FIG.1B. FIG.1A shows the different components of one example of such a scheme. In the this illustrated example, the reference frame is selected based on the correlation of each acquired frame with all other frames in the sequence. For this purpose, initially, the acquiredd z^ d x IQ frames ranging from 1 to d t are restructured into a spatiotemporal form to obtaina Casorati matrix, C m . The columns of the Casorati matrix contain the spatial information (i.e.,the frame data) and the rows contain the time series data, as shown in FIG.1A. The covarianceof the Casorati matrix is computed to obtain a matrix, CC, with dimension d t^ d t , whichcontains the correlation values of each frame with respect to all other frames in the sequence. Each diagonal value corresponds to the correlation of each frame with itself, resulting in the maximum value, as observed from the colormap of CC shown in FIG.1A. The frame with the maximum correlation with respect to all other frames can be obtained by computing row- wise / column-wise mean, M, of the CC matrix and extracting the index, I, of the maximum value of M, i.e., MI. The mathematical interpretation of the same is shown in FIG.1A, as:

[0051] Mean computation: ^dt^ CC^^;

[0052]

[0053] MI ^max ^ M ^ (13).

[0054] In one non-limiting example, the disclosed systems and methods can implement motion correction in ultrasound image sequences using a MoCoSCNet framework with ground truth sequences derived from a benchmark two-stage motion correction algorithm. This example implementation can be split into two stages, as shown in FIG.4.

[0055] Stage-I includes at least two components: selection of the reference image frame using a Casorati correlation matrix and application of two-stage motion correction for registering all frames with respect to the selected reference frame. The images are captured using an ultrasound system, or previously acquired ultrasound images are otherwise accessed from a memory or other data storage medium or device at step 402. Initially, the Casorati correlation is computed between the first frame and the remaining frames, and this process is repeated for all frames in the sequence, to form the Casorati correlation matrix, as indicated at 10 QB\630666.01539\94528245.3Mayo 2023-353 630666.01539 steps 404 and 406. Analysis of the Casorati correlation matrix is performed to obtain the frame that possesses maximum correlation with all other frames, as indicated at step 408. This corresponding frame is selected as the reference frame. A benchmark two-stage motion correction method is then applied to register all frames in the imaging sequence with the selected reference frame. It predicts the transformation matrices for non-rigid and affine motion, which are further used to compute the motion corrected image frames.

[0056] Stage-II includes feeding the motion corrected image frames obtained from the previous stage as ground truth along with the original image frames to train the deep learning network (e.g., MoCoSCNet), as illustrated at steps 410 and 412. Another example of a MoCoSCNet model is shown in FIG.5, which includes two convolutional layers, followed by four separable convolution layers and an output convolution layer for prediction. Rectified linear unit (ReLu) is used as the activation function in these layers, and batch normalization is used for reducing overfitting. The corresponding hyperparameters including the filters and size are illustrated in FIG. 5. As an example, stochastic gradient descent (SGD) optimizer with a learning rate of 0.01 can be used for minimizing the loss function (e.g., mean square error). Mean absolute error can be used as the regression metric.

[0057] In another non-limiting example, the disclosed systems and methods can implement motion correction in ultrasound image sequences using a MoCoSCNet framework with ground truth sequences derived from a selected reference frame using the Casorati correlation matrix as described above. This example implementation can be split into two stages, as shown in FIG.6.

[0058] The methodology is split into two stages, as shown in FIG. 6. The two stages are similar to those shown in the first methodology, except that the ground truth is extracted as the reference frame itself, rather than using the two-stage motion correction.

[0059] Referring now to FIG. 7, a flowchart is illustrated as setting forth the steps of an example method for generating classified feature data using a suitably trained neural network or other machine learning algorithm. As will be described, the neural network or other machine learning algorithm takes ultrasound image data and a reference image as input data and generates motion corrected image data as output data.

[0060] The method includes accessing ultrasound image data with a computer system, as indicated at step 702. Accessing the ultrasound image data may include retrieving such data from a memory or other suitable data storage device or medium. Additionally or alternatively, accessing the ultrasound image data may include acquiring such data with an ultrasound 11 QB\630666.01539\94528245.3Mayo 2023-353 630666.01539 imaging system and transferring or otherwise communicating the data to the computer system, which may be a part of the ultrasound imaging system. In some implementations, the ultrasound image data may include contrast-enhanced ultrasound images. In some other implementations, the ultrasound image data may include super-resolution ultrasound images. In still other implementations, as noted above, the ultrasound image data may instead be image data acquired with an imaging modality other than ultrasound. For example, the image data may instead be MRI image data, OCT image data, or the like.

[0061] A reference image is then selected to be used in the motion correction network, as indicated at step 704. As described above, the reference image can be selected based on a Casorati correlation matrix computed amongst the image frames in the ultrasound image data. For instance, the Casorati correlation can be computed between the first frame and the remaining frames in the ultrasound image data, and this process can be repeated for all frames in the ultrasound image data to form the Casorati correlation matrix. The Casorati correlation matrix can then be analyzed to determine the frame in the ultrasound image data having the maximum correlation with all other frames in the ultrasound image data. This frame can then be selected as the reference frame.

[0062] A trained neural network (or other suitable machine learning algorithm) is then accessed with the computer system, as indicated at step 706. In general, the neural network is trained, or has been trained, on training data in order to generate motion-corrected ultrasound images from ultrasound image data that includes images containing motion. In some cases, the neural network may be a MoCoSCNet network for generating motion-corrected ultrasound images, as described above with respect to FIG. 1A. In some other cases, the neural network may be an IFMoCoNet network for generating inter-frame motion-corrected ultrasound images, as described above with respect to FIG.1B.

[0063] Accessing the trained neural network may include accessing network parameters (e.g., weights, biases, or both) that have been optimized or otherwise estimated by training the neural network on training data. In some instances, retrieving the neural network can also include retrieving, constructing, or otherwise accessing the particular neural network architecture to be implemented. For instance, data pertaining to the layers in the neural network architecture (e.g., number of layers, type of layers, ordering of layers, connections between layers, hyperparameters for layers) may be retrieved, selected, constructed, or otherwise accessed. 12 QB\630666.01539\94528245.3Mayo 2023-353 630666.01539

[0064] An artificial neural network generally includes an input layer, one or more hidden layers (or nodes), and an output layer. Typically, the input layer includes as many nodes as inputs provided to the artificial neural network. The number (and the type) of inputs provided to the artificial neural network may vary based on the particular task for the artificial neural network.

[0065] The input layer connects to one or more hidden layers. The number of hidden layers varies and may depend on the particular task for the artificial neural network. Additionally, each hidden layer may have a different number of nodes and may be connected to the next layer differently. For example, each node of the input layer may be connected to each node of the first hidden layer. The connection between each node of the input layer and each node of the first hidden layer may be assigned a weight parameter. Additionally, each node of the neural network may also be assigned a bias value. In some configurations, each node of the first hidden layer may not be connected to each node of the second hidden layer. That is, there may be some nodes of the first hidden layer that are not connected to all of the nodes of the second hidden layer. The connections between the nodes of the first hidden layers and the second hidden layers are each assigned different weight parameters. Each node of the hidden layer is generally associated with an activation function. The activation function defines how the hidden layer is to process the input received from the input layer or from a previous input or hidden layer. These activation functions may vary and be based on the type of task associated with the artificial neural network and also on the specific type of hidden layer implemented.

[0066] Each hidden layer may perform a different function. For example, some hidden layers can be convolutional hidden layers which can, in some instances, reduce the dimensionality of the inputs. Other hidden layers can perform statistical functions such as max pooling, which may reduce a group of inputs to the maximum value; an averaging layer; batch normalization; and other such functions. In some of the hidden layers each node is connected to each node of the next hidden layer, which may be referred to then as dense layers. Some neural networks including more than, for example, three hidden layers may be considered deep neural networks.

[0067] The last hidden layer in the artificial neural network is connected to the output layer. Similar to the input layer, the output layer typically has the same number of nodes as the possible outputs. 13 QB\630666.01539\94528245.3Mayo 2023-353 630666.01539

[0068] The ultrasound image data and reference image are then input to the trained neural network, generating output as motion-corrected image data, as indicated at step 708. In some embodiments, the reference image may be provided as a ground truth image to the trained neural network, as described above with respect to FIG.1A and / or FIG.1B and methodology 2. Additionally or alternatively, a ground truth image may be supplied by using the reference image in a benchmark two-stage motion correction method to register all frames in the ultrasound image data with the selected reference frame, as in methodology 1 described above.

[0069] The motion-corrected ultrasound image data generated by inputting the ultrasound image data and reference image to the trained neural network(s) can then be displayed to a user, stored for later use or further processing, or both, as indicated at step 710.

[0070] Referring now to FIG. 8, a flowchart is illustrated as setting forth the steps of an example method for training one or more neural networks (or other suitable machine learning algorithms) on training data, such that the one or more neural networks are trained to receive ultrasound image data and a ground truth image (e.g., a reference image, as described above) as input data in order to generate motion-corrected ultrasound image data as output data.

[0071] In general, the neural network(s) can implement any number of different neural network architectures. For instance, the neural network(s) could implement a convolutional neural network, a residual neural network, or the like. Alternatively, the neural network(s) could be replaced with other suitable machine learning or artificial intelligence algorithms, such as those based on supervised learning, unsupervised learning, deep learning, ensemble learning, dimensionality reduction, and so on.

[0072] The method includes accessing training data with a computer system, as indicated at step 802. Accessing the training data may include retrieving such data from a memory or other suitable data storage device or medium. Alternatively, accessing the training data may include acquiring such data with an ultrasound system and transferring or otherwise communicating the data to the computer system.

[0073] In general, the training data can include ultrasound images with and without motion. The method can include assembling training data from ultrasound images using a computer system. This step may include assembling the ultrasound images into an appropriate data structure on which the neural network or other machine learning algorithm can be trained. Assembling the training data may include assembling ultrasound images and other relevant data. For training the IFMoCoNet described above, the training data may include inter-frame 14 QB\630666.01539\94528245.3Mayo 2023-353 630666.01539 motion-containing input ultrasound image frames acquired from a plurality of subjects, along with the selected reference ultrasound image frame(s) as the ground truth.

[0074] One or more neural networks (or other suitable machine learning algorithms) are trained on the training data, as indicated at step 804. In general, the neural network can be trained by optimizing network parameters (e.g., weights, biases, or both) based on minimizing a loss function. As one non-limiting example, the loss function may be a mean squared error loss function.

[0075] Training a neural network may include initializing the neural network, such as by computing, estimating, or otherwise selecting initial network parameters (e.g., weights, biases, or both). During training, an artificial neural network receives the inputs for a training example and generates an output using the bias for each node, and the connections between each node and the corresponding weights. For instance, training data can be input to the initialized neural network, generating output as motion-corrected ultrasound image data. The artificial neural network then compares the generated output with the actual output of the training example in order to evaluate the quality of the output data. For instance, the output data can be passed to a loss function to compute an error. The current neural network can then be updated based on the calculated error (e.g., using backpropagation methods based on the calculated error). For instance, the current neural network can be updated by updating the network parameters (e.g., weights, biases, or both) in order to minimize the loss according to the loss function. The training continues until a training condition is met. The training condition may correspond to, for example, a predetermined number of training examples being used, a minimum accuracy threshold being reached during training and validation, a predetermined number of validation iterations being completed, and the like. When the training condition has been met (e.g., by determining whether an error threshold or other stopping criterion has been satisfied), the current neural network and its associated network parameters represent the trained neural network. Different types of training processes can be used to adjust the bias values and the weights of the node connections based on the training examples. The training processes may include, for example, gradient descent, Newton's method, conjugate gradient, quasi-Newton, Levenberg-Marquardt, among others.

[0076] The artificial neural network can be constructed or otherwise trained based on training data using one or more different learning techniques, such as supervised learning, unsupervised learning, reinforcement learning, ensemble learning, active learning, transfer learning, or other suitable learning techniques for neural networks. As an example, supervised 15 QB\630666.01539\94528245.3Mayo 2023-353 630666.01539 learning involves presenting a computer system with example inputs and their actual outputs (e.g., categorizations). In these instances, the artificial neural network is configured to learn a general rule or model that maps the inputs to the outputs based on the provided example input– output pairs.

[0077] By way of non-limiting example, the disclosed motion correction networks may employ a set of learning parameters that are selected based on optimal hyper-parameter tuning. For example, the number of filters / kernels for the convolutional layers may be selected as 1, 64, 128, 256, each with a size of 3 × 3 and a stride of 1, along with batch sizes between 2 and 8 and epochs between 40 and 100. For the IFMoCoNet, these parameters assist the network to learn the mapping between inter-frame motion-containing frames and the reference frame(s). Further, the motion correction networks may be trained using an Adam optimizer with a learning rate of 0.0005 for minimizing the loss function, i.e., mean square error, and maximizing the similarity metric, i.e., cosine similarity in a non-limiting example.

[0078] The one or more trained neural networks are then stored for later use, as indicated at step 806. Storing the neural network(s) may include storing network parameters (e.g., weights, biases, or both), which have been computed or otherwise estimated by training the neural network(s) on the training data. Storing the trained neural network(s) may also include storing the particular neural network architecture to be implemented. For instance, data pertaining to the layers in the neural network architecture (e.g., number of layers, type of layers, ordering of layers, connections between layers, hyperparameters for layers) may be stored.

[0079] In an example study, a MoCoSCNet model was implemented for motion correction of the ultrasound image frames. Initially, the training procedure and hyper- parameters are presented, and subsequently, the evaluation metrics used to evaluate the motion correction results are described.

[0080] The motion correction network was trained using the motion-containing input B-mode image frames acquired from each subject available in the study, along with the selected reference frame as the ground truth. The number of acquired image frames were different for all subjects, ranging between 1000 − 2500. For each subject, 80% of the frames were used for training, 10% for validating, and the remaining 10% for testing the proposed network. The proposed network employed a set of learning parameters that were selected after optimal hyper- parameter tuning using Keras tuner and rigorous experimental assessment, as listed in Table 1. 16 QB\630666.01539\94528245.3Mayo 2023-353 630666.01539 Table 1: Training Hyperparameters Dr k d s L BS e c32 64 128 256 3^ 3 1 001 2 40s.

[0081] These parameters assist the network to learn the mapping between motion- containing frames and the reference frame. Further, the proposed network used stochastic gradient descent (SGD) optimizer for minimizing the loss function, similar to the implementation described above.

[0082] The efficiency of the proposed network was demonstrated in terms of different evaluation metrics, such as, Pearson correlation coefficient (PCC), structural similarity index (SSI), and mean square error (MSE). PCC is mathematically computed for the predicted test output image t with respect to the ground truth, i.e., reference frame r as: ^^ t j ^ t m ^^ r j ^ r m ^^ 2

[0083] t, rjis the intensity of pixel j in image r, tm is the mean intensity of image t, and rm is the mean intensity of image r.

[0084] SSI was utilized to assess the motion corrected ultrasound images. It estimates the structural resemblance between two images in terms of the brightness and contrast of local image pixel intensity patterns. SSI can be estimated between the local windows of test image, i.e., tl and reference frame rl, as: ^2 ^ t l ^ r l ^ c 1 ^^ 2 ^ t l r l ^ c 2 ^^ ^^ ^

[0085] respectively, ^tl and ^rl denotethe variances of tl^t l r l denotes the covariance of tl and rl, c1 and c2 denotearbitrary constants used to prevent instability in the case of division with small denominator. SSIM between the full images is estimated as the mean of all such local SSIM values: 17 QB\630666.01539\94528245.3Mayo 2023-353 630666.01539 W SSI ^ t , r ^ ^1 W^ SSI ^ t j , r j ^^1

[0086] at window j, and W denotes the total number of local

[0087] MSE is another widely adopted quantitative measure used to analyze two images for the extent of fidelity or the amount of distortion. It is calculated as the mean of the sum of squared difference between the pixel intensities of the y × x images t and r as: y^ 1 x ^ 1MSE1 2 ^t , r ^ ^ ^^^^ t ^ i , j ^ ^ r ^ i , j ^ ^ ^

[0088] of the motion corrected imagesby the are now analysis was performed for all subjects to demonstrate the capability of MoCoSCNet. The metrics were estimated for the 10% test images with respect to the reference image, for each subject. Further, to diminish the statistical ambiguity of the proposed network, 10-fold cross-validation was used for each subject. The proposed network achieved an overall PCC of 0.95, MSE of 0.0003, and SSI of 0.92 averaged over all frames of 356 subjects, as shown in Table 2. The table also lists the original values of these metrics, i.e., before applying motion correction using MoCoSCNet. It can be observed from Table 2 that the proposed network achieved improved values of all metrics, demonstrating its effectiveness in registering the frames with respect to reference frame. Table 2: MoCoSCNet Evaluation Metrics (Mean (Std) over subjects) Case PCC MSE SSI

[0089] FIG. 9 represents the density distribution of evaluation metrics with respect to all subjects. It gives an insight of the probabilities of the different metric values for original and the motion corrected data using MoCoSCNet. It can be observed from FIG. 9(a) that the original PCC values are unevenly distributed among the subjects, which is due to the varying motion content. It indicates that there is a considerable motion in a narrow range of subjects, while a wide variety of subjects contain less motion. MoCoSCNet achieves more uniformity 18 QB\630666.01539\94528245.3Mayo 2023-353 630666.01539 in the probability densities above and below the mean value by improving the originally low PCC values in the narrow range significantly. Similarly, it can be observed for other metric distributions that the proposed network achieves considerable improvement for all subjects and is able to maintain evenness among the metric values of all subjects. This further indicates that the proposed network is able to achieve a superior distribution of densities demonstrating its robustness with respect to data.

[0090] FIG. 10 provides additional insight of the above mentioned results by illustrating the percentage improvement in the evaluation metrics obtained by the proposed MoCoSCNet with reference to the original metrics, calculated as: Mt MoCoSCNet^M% ^ ^ ^ t ^ Original ^ ^ 100

[0091] obtained after motioncorrection using MoCoSCNet and that obtained before motion correction, respectively. It can be observed from FIG. 10 that approximately up to 5% improvement is achieved in PCC and SSI, and up to 40% reduction is observed in MSE after motion correction, for 75% of the subjects in this study. Further, for a few subjects containing significant motion, the improvement is substantial, as depicted from the outliers in FIG. 10. This demonstrates the adeptness of the proposed network in suppressing motion.

[0092] This example study presents a comprehensive analysis of the proposed MoCoSCNet for motion correction in ultrasound imaging, along with the selection of ground truth. The architecture of MoCoSCNet highlights its low complexity and straightforwardness. The evaluation metrics presented in FIG. 9 demonstrate that the proposed network is able to capture the variations in motion with respect to different subject data and effectively correct motion. The densities represent the probability distributions of the different metrics, which further demonstrate the efficiency of MoCoSCNet in suppressing motion for diverse dataset. Subsequently, the percentage improvement in evaluation metrics in FIG. 10 indicates that the proposed network is able to suppress motion in all subjects. It is noticeable that the outliers are due to the few subjects with exceptionally high values of PCC and SSI. This is apparent since these few subjects had more motion initially, as observed from the corresponding narrow distribution of the original low metric values in FIG. 9. These results demonstrate that the proposed MoCoSCNet is capable of recognizing frames with significant motion by automatically learning their feature representations and effectively correcting them. 19 QB\630666.01539\94528245.3Mayo 2023-353 630666.01539

[0093] FIG. 11 shows an example of a system 1100 for generating motion-corrected ultrasound image in accordance with some embodiments of the systems and methods described in the present disclosure. As shown in FIG. 11, a computing device 1150 can receive one or more types of data (e.g., ultrasound image data, a reference image) from data source 1102. In some embodiments, computing device 1150 can execute at least a portion of an ultrasound image motion correction system 1104 to correct motion in ultrasound image data received from the data source 1102.

[0094] Additionally or alternatively, in some embodiments, the computing device 1150 can communicate information about data received from the data source 1102 to a server 1152 over a communication network 1154, which can execute at least a portion of the ultrasound image motion correction system 1104. In such embodiments, the server 1152 can return information to the computing device 1150 (and / or any other suitable computing device) indicative of an output of the ultrasound image motion correction system 1104.

[0095] In some embodiments, computing device 1150 and / or server 1152 can be any suitable computing device or combination of devices, such as a desktop computer, a laptop computer, a smartphone, a tablet computer, a wearable computer, a server computer, a virtual machine being executed by a physical computing device, and so on. The computing device 1150 and / or server 1152 can also reconstruct images from the data.

[0096] In some embodiments, data source 1102 can be any suitable source of data (e.g., measurement data, images reconstructed from measurement data, processed image data), such as an ultrasound imaging system, another computing device (e.g., a server storing measurement data, images reconstructed from measurement data, processed image data), and so on. In some embodiments, data source 1102 can be local to computing device 1150. For example, data source 1102 can be incorporated with computing device 1150 (e.g., computing device 1150 can be configured as part of a device for measuring, recording, estimating, acquiring, or otherwise collecting or storing data). As another example, data source 1102 can be connected to computing device 1150 by a cable, a direct wireless link, and so on. Additionally or alternatively, in some embodiments, data source 1102 can be located locally and / or remotely from computing device 1150, and can communicate data to computing device 1150 (and / or server 1152) via a communication network (e.g., communication network 1154).

[0097] In some embodiments, communication network 1154 can be any suitable communication network or combination of communication networks. For example, communication network 1154 can include a Wi-Fi network (which can include one or more 20 QB\630666.01539\94528245.3Mayo 2023-353 630666.01539 wireless routers, one or more switches, etc.), a peer-to-peer network (e.g., a Bluetooth network), a cellular network (e.g., a 3G network, a 4G network, etc., complying with any suitable standard, such as CDMA, GSM, LTE, LTE Advanced, WiMAX, etc.), other types of wireless network, a wired network, and so on. In some embodiments, communication network 1154 can be a local area network, a wide area network, a public network (e.g., the Internet), a private or semi-private network (e.g., a corporate or university intranet), any other suitable type of network, or any suitable combination of networks. Communications links shown in FIG.11 can each be any suitable communications link or combination of communications links, such as wired links, fiber optic links, Wi-Fi links, Bluetooth links, cellular links, and so on.

[0098] Referring now to FIG. 12, an example of hardware 1200 that can be used to implement data source 1102, computing device 1150, and server 1152 in accordance with some embodiments of the systems and methods described in the present disclosure is shown.

[0099] As shown in FIG.12, in some embodiments, computing device 1150 can include a processor 1202, a display 1204, one or more inputs 1206, one or more communication systems 1208, and / or memory 1210. In some embodiments, processor 1202 can be any suitable hardware processor or combination of processors, such as a central processing unit (CPU), a graphics processing unit (GPU), and so on. In some embodiments, display 1204 can include any suitable display devices, such as a liquid crystal display (LCD) screen, a light-emitting diode (LED) display, an organic LED (OLED) display, an electrophoretic display (e.g., an “e- ink” display), a computer monitor, a touchscreen, a television, and so on. In some embodiments, inputs 1206 can include any suitable input devices and / or sensors that can be used to receive user input, such as a keyboard, a mouse, a touchscreen, a microphone, and so on.

[0100] In some embodiments, communications systems 1208 can include any suitable hardware, firmware, and / or software for communicating information over communication network 1154 and / or any other suitable communication networks. For example, communications systems 1208 can include one or more transceivers, one or more communication chips and / or chip sets, and so on. In a more particular example, communications systems 1208 can include hardware, firmware, and / or software that can be used to establish a Wi-Fi connection, a Bluetooth connection, a cellular connection, an Ethernet connection, and so on.

[0101] In some embodiments, memory 1210 can include any suitable storage device or devices that can be used to store instructions, values, data, or the like, that can be used, for 21 QB\630666.01539\94528245.3Mayo 2023-353 630666.01539 example, by processor 1202 to present content using display 1204, to communicate with server 1152 via communications system(s) 1208, and so on. Memory 1210 can include any suitable volatile memory, non-volatile memory, storage, or any suitable combination thereof. For example, memory 1210 can include random-access memory (RAM), read-only memory (ROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), other forms of volatile memory, other forms of non-volatile memory, one or more forms of semi-volatile memory, one or more flash drives, one or more hard disks, one or more solid state drives, one or more optical drives, and so on. In some embodiments, memory 1210 can have encoded thereon, or otherwise stored therein, a computer program for controlling operation of computing device 1150. In such embodiments, processor 1202 can execute at least a portion of the computer program to present content (e.g., images, user interfaces, graphics, tables), receive content from server 1152, transmit information to server 1152, and so on. For example, the processor 1202 and the memory 1210 can be configured to perform the methods described herein (e.g., the method of FIG.7, the method of FIG.8).

[0102] In some embodiments, server 1152 can include a processor 1212, a display 1214, one or more inputs 1216, one or more communications systems 1218, and / or memory 1220. In some embodiments, processor 1212 can be any suitable hardware processor or combination of processors, such as a CPU, a GPU, and so on. In some embodiments, display 1214 can include any suitable display devices, such as an LCD screen, LED display, OLED display, electrophoretic display, a computer monitor, a touchscreen, a television, and so on. In some embodiments, inputs 1216 can include any suitable input devices and / or sensors that can be used to receive user input, such as a keyboard, a mouse, a touchscreen, a microphone, and so on.

[0103] In some embodiments, communications systems 1218 can include any suitable hardware, firmware, and / or software for communicating information over communication network 1154 and / or any other suitable communication networks. For example, communications systems 1218 can include one or more transceivers, one or more communication chips and / or chip sets, and so on. In a more particular example, communications systems 1218 can include hardware, firmware, and / or software that can be used to establish a Wi-Fi connection, a Bluetooth connection, a cellular connection, an Ethernet connection, and so on.

[0104] In some embodiments, memory 1220 can include any suitable storage device or devices that can be used to store instructions, values, data, or the like, that can be used, for 22 QB\630666.01539\94528245.3Mayo 2023-353 630666.01539 example, by processor 1212 to present content using display 1214, to communicate with one or more computing devices 1150, and so on. Memory 1220 can include any suitable volatile memory, non-volatile memory, storage, or any suitable combination thereof. For example, memory 1220 can include RAM, ROM, EPROM, EEPROM, other types of volatile memory, other types of non-volatile memory, one or more types of semi-volatile memory, one or more flash drives, one or more hard disks, one or more solid state drives, one or more optical drives, and so on. In some embodiments, memory 1220 can have encoded thereon a server program for controlling operation of server 1152. In such embodiments, processor 1212 can execute at least a portion of the server program to transmit information and / or content (e.g., data, images, a user interface) to one or more computing devices 1150, receive information and / or content from one or more computing devices 1150, receive instructions from one or more devices (e.g., a personal computer, a laptop computer, a tablet computer, a smartphone), and so on.

[0105] In some embodiments, the server 1152 is configured to perform the methods described in the present disclosure. For example, the processor 1212 and memory 1220 can be configured to perform the methods described herein (e.g., the method of FIG.7, the method of FIG.8).

[0106] In some embodiments, data source 1102 can include a processor 1222, one or more data acquisition systems 1224, one or more communications systems 1226, and / or memory 1228. In some embodiments, processor 1222 can be any suitable hardware processor or combination of processors, such as a CPU, a GPU, and so on. In some embodiments, the one or more data acquisition systems 1224 are generally configured to acquire data, images, or both, and can include an ultrasound imaging system. Additionally or alternatively, in some embodiments, the one or more data acquisition systems 1224 can include any suitable hardware, firmware, and / or software for coupling to and / or controlling operations of an ultrasound imaging system. In some embodiments, one or more portions of the data acquisition system(s) 1224 can be removable and / or replaceable.

[0107] Note that, although not shown, data source 1102 can include any suitable inputs and / or outputs. For example, data source 1102 can include input devices and / or sensors that can be used to receive user input, such as a keyboard, a mouse, a touchscreen, a microphone, a trackpad, a trackball, and so on. As another example, data source 1102 can include any suitable display devices, such as an LCD screen, an LED display, an OLED display, an electrophoretic display, a computer monitor, a touchscreen, a television, etc., one or more speakers, and so on. 23 QB\630666.01539\94528245.3Mayo 2023-353 630666.01539

[0108] In some embodiments, communications systems 1226 can include any suitable hardware, firmware, and / or software for communicating information to computing device 1150 (and, in some embodiments, over communication network 1154 and / or any other suitable communication networks). For example, communications systems 1226 can include one or more transceivers, one or more communication chips and / or chip sets, and so on. In a more particular example, communications systems 1226 can include hardware, firmware, and / or software that can be used to establish a wired connection using any suitable port and / or communication standard (e.g., VGA, DVI video, USB, RS-232, etc.), Wi-Fi connection, a Bluetooth connection, a cellular connection, an Ethernet connection, and so on.

[0109] In some embodiments, memory 1228 can include any suitable storage device or devices that can be used to store instructions, values, data, or the like, that can be used, for example, by processor 1222 to control the one or more data acquisition systems 1224, and / or receive data from the one or more data acquisition systems 1224; to generate images from data; present content (e.g., data, images, a user interface) using a display; communicate with one or more computing devices 1150; and so on. Memory 1228 can include any suitable volatile memory, non-volatile memory, storage, or any suitable combination thereof. For example, memory 1228 can include RAM, ROM, EPROM, EEPROM, other types of volatile memory, other types of non-volatile memory, one or more types of semi-volatile memory, one or more flash drives, one or more hard disks, one or more solid state drives, one or more optical drives, and so on. In some embodiments, memory 1228 can have encoded thereon, or otherwise stored therein, a program for controlling operation of data source 1102. In such embodiments, processor 1222 can execute at least a portion of the program to generate images, transmit information and / or content (e.g., data, images, a user interface) to one or more computing devices 1150, receive information and / or content from one or more computing devices 1150, receive instructions from one or more devices (e.g., a personal computer, a laptop computer, a tablet computer, a smartphone, etc.), and so on.

[0110] In some embodiments, any suitable computer-readable media can be used for storing instructions for performing the functions and / or processes described herein. For example, in some embodiments, computer-readable media can be transitory or non-transitory. For example, non-transitory computer-readable media can include media such as magnetic media (e.g., hard disks, floppy disks), optical media (e.g., compact discs, digital video discs, Blu-ray discs), semiconductor media (e.g., RAM, flash memory, EPROM, EEPROM), any suitable media that is not fleeting or devoid of any semblance of permanence during 24 QB\630666.01539\94528245.3Mayo 2023-353 630666.01539 transmission, and / or any suitable tangible media. As another example, transitory computer- readable media can include signals on networks, in wires, conductors, optical fibers, circuits, or any suitable media that is fleeting and devoid of any semblance of permanence during transmission, and / or any suitable intangible media.

[0111] As used herein in the context of computer implementation, unless otherwise specified or limited, the terms “component,” “system,” “module,” “framework,” and the like are intended to encompass part or all of computer-related systems that include hardware, software, a combination of hardware and software, or software in execution. For example, a component may be, but is not limited to being, a processor device, a process being executed (or executable) by a processor device, an object, an executable, a thread of execution, a computer program, or a computer. By way of illustration, both an application running on a computer and the computer can be a component. One or more components (or system, module, and so on) may reside within a process or thread of execution, may be localized on one computer, may be distributed between two or more computers or other processor devices, or may be included within another component (or system, module, and so on).

[0112] In some implementations, devices or systems disclosed herein can be utilized or installed using methods embodying aspects of the disclosure. Correspondingly, description herein of particular features, capabilities, or intended purposes of a device or system is generally intended to inherently include disclosure of a method of using such features for the intended purposes, a method of implementing such capabilities, and a method of installing disclosed (or otherwise known) components to support these purposes or capabilities. Similarly, unless otherwise indicated or limited, discussion herein of any method of manufacturing or using a particular device or system, including installing the device or system, is intended to inherently include disclosure, as embodiments of the disclosure, of the utilized features and implemented capabilities of such device or system.

[0113] FIG.13 illustrates an example of an ultrasound system 1300 that can implement the methods described in the present disclosure. The ultrasound system 1300 includes a transducer array 1302 that includes a plurality of separately driven transducer elements 1304. The transducer array 1302 can include any suitable ultrasound transducer array, including linear arrays, curved arrays, phased arrays, and so on. Similarly, the transducer array 1302 can include a 1D transducer, a 1.5D transducer, a 1.75D transducer, a 2D transducer, a 3D transducer, and so on. 25 QB\630666.01539\94528245.3Mayo 2023-353 630666.01539

[0114] When energized by a transmitter 1306, a given transducer element 1304 produces a burst of ultrasonic energy. The ultrasonic energy reflected back to the transducer array 1302 (e.g., an echo) from the object or subject under study is converted to an electrical signal (e.g., an echo signal) by each transducer element 1304 and can be applied separately to a receiver 1308 through a set of switches 1310. The transmitter 1306, receiver 1308, and switches 1310 are operated under the control of a controller 1312, which may include one or more processors. As one example, the controller 1312 can include a computer system.

[0115] The transmitter 1306 can be programmed to transmit unfocused or focused ultrasound waves. In some configurations, the transmitter 1306 can also be programmed to transmit diverged waves, spherical waves, cylindrical waves, plane waves, or combinations thereof. Furthermore, the transmitter 1306 can be programmed to transmit spatially or temporally encoded pulses.

[0116] The receiver 1308 can be programmed to implement a suitable detection sequence for the imaging task at hand. In some embodiments, the detection sequence can include one or more of line-by-line scanning, compounding plane wave imaging, synthetic aperture imaging, and compounding diverging beam imaging.

[0117] In some configurations, the transmitter 1306 and the receiver 1308 can be programmed to implement a high frame rate. For instance, a frame rate associated with an acquisition pulse repetition frequency (PRF) of at least 100 Hz can be implemented. In some configurations, the ultrasound system 1300 can sample and store at least one hundred ensembles of echo signals in the temporal direction.

[0118] The controller 1312 can be programmed to design or otherwise select an imaging sequence to acquire ultrasound image data. In some embodiments, the controller 1312 receives user inputs defining various factors used in the design of the imaging sequence.

[0119] A scan can be performed by setting the switches 1310 to their transmit position, thereby directing the transmitter 1306 to be turned on momentarily to energize transducer elements 1304 during a single transmission event according to the imaging sequence. The switches 1310 can then be set to their receive position and the subsequent echo signals produced by the transducer elements 1304 in response to one or more detected echoes are measured and applied to the receiver 1308. The separate echo signals from the transducer elements 1304 can be combined in the receiver 1308 to produce a single echo signal.

[0120] The echo signals are communicated to a processing unit 1314, which may be implemented by a hardware processor and memory, to process echo signals or images 26 QB\630666.01539\94528245.3Mayo 2023-353 630666.01539 generated from echo signals. As an example, the processing unit 1314 can implement motion correction of ultrasound image data using the methods described in the present disclosure. Images produced from the echo signals by the processing unit 1314 can be displayed on a display system 1316.

[0121] The present disclosure has described one or more preferred embodiments, and it should be appreciated that many equivalents, alternatives, variations, and modifications, aside from those expressly stated, are possible and within the scope of the invention. 27 QB\630666.01539\94528245.3

Claims

Mayo 2023-353 630666.01539 CLAIMS 1. A method for reducing motion in ultrasound images, the method comprising: accessing ultrasound image data with a computer system, wherein the ultrasound image data comprise a sequence of ultrasound image frames containing motion; selecting a reference image from the ultrasound image data; accessing a machine learning model with the computer system, wherein the machine learning model has been trained on training data to correct motion in ultrasound images; inputting the ultrasound image data and the reference image to the machine learning model with the computer system, generating motion-corrected ultrasound image data as an output, wherein the reference image is used as a ground truth image by the machine learning model; and outputting the motion-corrected ultrasound image data via the computer system.

2. The method of claim 1, further comprising generating a spatiotemporal correlation matrix from the ultrasound image data using the computer system and wherein the reference image is selected from the ultrasound image data based on the spatiotemporal correlation matrix.

3. The method of claim 2, wherein the spatiotemporal correlation matrix comprises a Casorati correlation matrix.

4. The method of claim 3, wherein generating the Casorati correlation matrix comprises reformatting the ultrasound image data as a Casorati matrix and computing a correlation between image frames in the Casorati matrix.

5. The method of claim 4, wherein computing the correlation between image frames in the Casorati matrix comprises computing a correlation of each image frame in the Casorati matrix with all other image frames in the Casorati matrix.

6. The method of claim 5, wherein computing the correlation comprises computing a covariance of the Casorati matrix. 28 QB\630666.01539\94528245.3Mayo 2023-353 630666.01539 7. The method of claim 3, wherein selecting the reference image comprises selecting the image frame in the ultrasound image data having a maximum correlation in the Casorati correlation matrix.

8. The method of claim 7, wherein the maximum correlation in the Casorati correlation matrix is determined by computing row-wise / column-wise mean values of the Casorati correlation matrix and determining a maximum value of the row-wise / column-wise mean values.

9. The method of claim 1, wherein the reference image is used to benchmark a two-stage motion correction process to obtain motion-free image frames, and the motion-free image frames are used as the ground truth for the machine learning model.

10. The method of claim 1, wherein the machine learning model comprises a neural network.

11. The method of claim 10, wherein the neural network comprises a convolutional neural network.

12. The method of claim 11, wherein the convolutional neural network comprises depth-wise separable convolutional layers.

13. The method of claim 12, wherein the convolutional neural network comprises both convolutional layers and separable convolutional layers.

14. The method of claim 11, wherein the neural network includes a hybrid adaptive attention block that receives a feature map output from a convolutional layer of the convolutional neural network and generates an output as a combined channel and spatial- level attention feature map. 29 QB\630666.01539\94528245.3Mayo 2023-353 630666.01539 15. The method of claim 14, wherein the combined channel and spatial-level attention feature map indicates variable shifts due to different levels of inter-frame motion between ultrasound image frames of the sequence of ultrasound image frames.

16. The method of claim 14, wherein the hybrid adaptive attention block comprises: convolutional layers with varying filter sizes; a channel attention mechanism that extracts features with different receptive fields and scales from channel-level aspects of the feature map input to the hybrid adaptive attention block; and a spatial attention mechanism that extracts features with different receptive fields and scales from spatial-level aspects of the feature map input to the hybrid adaptive attention block.

17. The method of claim 11, wherein the neural network includes a squeeze-and- excite attention mechanism that receives a feature map from a convolutional layer of the convolutional neural network as an input and generates an output as a weighted feature map, wherein the weighted feature map comprises weighted adaptations for enhancing relevant features and suppressing less relevant features.

18. The method of claim 17, wherein the squeeze-and-excite attention mechanism comprises: a squeeze mechanism that converts each input feature map to a numeric value, thereby generating a squeezed vector as an output; and an excite mechanism that captures interdependencies and correlations between each input feature map, thereby generating a weight vector as an output, wherein the weight vector comprises learned weights for each feature map.

19. The method of claim 18, wherein the weighted feature map is generated by weighting the input feature maps by the corresponding weights in the weight vector.

20. The method of claim 18, wherein the numeric value comprises a spatial average of the input feature map. 30 QB\630666.01539\94528245.3Mayo 2023-353 630666.01539 21. The method of claim 20, wherein the squeeze mechanism converts each input feature map to the numeric value using a global average pooling operation.

22. The method of claim 18, wherein the excite mechanism comprises a first dense layer with a first activation function and a second dense layer with a second activation function, wherein the first dense layer receives the squeeze vector as an input and the second dense layer receives an output from the first dense layer as an input.

23. The method of claim 22, wherein the first activation function comprises a Gaussian error linear unit (GELU) activation function and the second activation function comprises a sigmoid activation function.

24. The method of claim 1, wherein the ultrasound image data comprise B-modes images.

25. The method of claim 1, wherein the ultrasound image data comprise in-phase and quadrature (IQ) images.

26. The method of claim 1, further comprising constructing Doppler ultrasound images from the motion-corrected ultrasound image data.

27. The method of claim 1, wherein the ultrasound image data comprise contrast- enhanced ultrasound images.

28. The method of claim 1, further comprising constructing super-resolution ultrasound images from the motion-corrected ultrasound image data.

29. The method of claim 1, wherein the motion-corrected ultrasound image data are generated in real-time while the ultrasound image data are acquired with an ultrasound imaging system.

30. A method for reducing motion in medical images, the method comprising: 31 QB\630666.01539\94528245.3Mayo 2023-353 630666.01539 accessing medical image data with a computer system, wherein the medical image data comprise a sequence of image frames containing motion; selecting a reference image from the medical image data; accessing a machine learning model with the computer system, wherein the machine learning model has been trained on training data to correct motion in medical images; inputting the medical image data and the reference image to the machine learning model with the computer system, generating motion-corrected medical image data as an output, wherein the reference image is used as a ground truth image by the machine learning model; and outputting the motion-corrected medical image data via the computer system.

31. The method of claim 30, further comprising generating a spatiotemporal correlation matrix from the medical image data using the computer system and wherein the reference image is selected from the medical image data based on the spatiotemporal correlation matrix 32. The method of claim 31, wherein the spatiotemporal correlation matrix comprises a Casorati correlation matrix.

33. The method of claim 30, wherein the medical image data comprise magnetic resonance image data acquired with a magnetic resonance imaging (MRI) system.

34. The method of claim 30, wherein the medical image data comprise optical coherence tomography (OCT) image data acquired with an OCT system. 32 QB\630666.01539\94528245.3

Images