Systems and methods for ultrasound blind sweep interpretation and fidelity assessment

Abstract

Ultrasound methods and systems that evaluate a plurality ultrasound sweeps in a blind sweep protocol. The methods and systems include: i) an ultrasound probe configured to acquire a plurality of two-dimensional (2D) images; ii) a signal processor configured to generate a three-dimensional (3D) image using the plurality of 2D images, and further configured to determine a validation metric for the 3D image; and iv) a user interface that communicates evaluation feedback at least partially based upon the 3D image.

Description

SYSTEMS AND METHODS FOR ULTRASOUND BLIND SWEEP INTERPRETATION AND FIDELITY ASSESSMENTField of the Disclosure

[0001] The present disclosure generally relates to medical diagnostic ultrasound systems. More specifically, the present disclosure is directed to systems and methods for guiding ultrasound imaging procedures using information detected from the ultrasound sweeps themselves.Background

[0002] Ultrasound diagnostic systems have become increasingly portable in recent years. In the early 2000s, system miniaturization started with desktop unit, then laptop-based configurations appeared. In recent years, there has been an increasing need for medical ultrasound imaging equipment that is portable to allow medical personnel to easily transport and deploy the equipment to and from or within healthcare facility and / or field locations, and more user-friendly to accommodate medical personnel who may possess a range of skill levels.

[0003] A blind sweep is known protocol for helping a novice operator acquire ultrasound images. In a blind sweep protocol, the operator sweeps the ultrasound over a pre-determined pattern, as opposed to moving the probe freely. This pre-determined pattern greatly simplifies the procedure because the operator is not required to interpret the ultrasound images in realtime and reposition the device according to these interpretations.

[0004] However, even when sweeping in a pre-determined pattern, there is still a significant chance of user error. For example, deviating from the pattern and / or concatenating multiple sweeps or misclassifying sweeps may lead to the erroneous localization of anatomical structures of interest or generation of low-fidelity diagnostic images.

[0005] Accordingly, there still exists a need in the art for systems and methods that automatically evaluate the fidelity of ultrasound sweeps and reconstruct the images into a display that is readily interpretable even by an inexperienced or novice operator.Summary of the Disclosure

[0006] The present disclosure is generally directed to systems and methods for evaluating an ultrasound imaging procedure. This includes acquiring a series of 2D ultrasound images of an anatomical area of interest by sweeping an ultrasound probe over a predetermined pattern, and then reconstructing a 3D image from the sequence of 2D images. The system or method furtherincludes determining a validation metric for the reconstructed 3D image and communicating evaluation feedback based on the validation metric for the 3D image. One manner in which the systems and methods disclosed herein improve upon conventional approaches is through the realization that acceleration and angular velocity acquired during an ultrasound imaging procedure can be used to generate a 3D image from the series of 2D images. Moreover, some ultrasound probes, such as Philips Lum if yTal ready include an inertial measurement unit (“IMU”) that could provide one approach for determining the acceleration and angular velocity. Accordingly, the disclosed systems and methods can automatically generate the reconstructed 3D image as part of the blind sweep protocol, and communicate evaluation feedback based on the 3D image to assist the operator in correcting any errors in executing the blind sweep protocol and obtaining the imaging information of the anatomical area of interest with sufficient fidelity to provide reliable diagnostic information.

[0007] Generally, in one aspect, the disclosure relates to an ultrasound system configured to evaluate a plurality of ultrasounds sweeps in a blind sweep protocol. The ultrasound system includes an ultrasound probe configured to acquire a plurality of two-dimensional (2D) ultrasound images of an anatomical area of interest. The system further includes a signal processor configured to generate at least one three-dimensional (3D) image of at least a portion of the anatomical area of interest based at least in part on the plurality of 2D images (48), and further configured to determine a validation metric for the 3D image. The system further includes a user interface for communicating evaluation feedback at least partially based on the validation metric for the 3D image.

[0008] In some embodiments, the system further includes an inertial measurement unit integrated with, or releasably attached to, the ultrasound probe. The inertial measurement unit including one or more probe estimation sensors configured to detect an acceleration and an angular velocity of the ultrasound probe when moved through the plurality of sweeps in a predetermined pattern, along the anatomical area of interest, in accordance with the blind sweep protocol. The signal processor may further generate the 3D image at least partially based on the detected acceleration and angular velocity.

[0009] In some further embodiments, the probe estimation sensors are selected from the group consisting of an accelerometer, a gyroscope, and a magnetometer.

[0010] The system may further include an image sensor. The image sensor is configured to track a movement of the ultrasound probe. The signal processor may be further configured to reconstruct the 3D image, at least partially based on the movement tracked by the image sensor.

[0011] In some embodiments, the validation metric is based, at least partially on, a completeness of the 3D image.

[0012] In some embodiments, the evaluation feedback includes identifying one or more low- fidelity regions. In some additional or alternative embodiments, the evaluation feedback includes displaying an alphanumeric image fidelity score.

[0013] In some embodiments, the evaluation feedback includes an instruction to repeat all or a portion of the blind sweep protocol. The evaluation feedback may further include an instruction to perform an abridged sweep, wherein at least a portion of the abridged sweep is distinct from the blind sweep protocol.

[0014] In some aspects, the signal processor is configured to generate the 3D image by inputting the plurality of 2D images into a neural network.

[0015] In some embodiments, the signal processor is configured to generate the 3D image by inserting a filling image into the plurality of 2D images. The signal processor may produce the filling image at least partially based on calculating an intermediate image value between two or more images of the plurality of 2D images.

[0016] In some embodiments, the 3D image includes a plurality of voxels. The signal processor may be configured to determine the validation metric at least partially based on a comparison between a fidelity of each voxel in the plurality of voxels and a voxel threshold. In some further embodiments, the signal processor may be configured to determine the validation threshold by comparing a total number of voxels with fidelity greater than or equal to the voxel threshold with a total number of voxels in the 3D image.

[0017] A further aspect of the disclosure relates to a method for evaluating a plurality ultrasound sweeps in a blind sweep protocol. The method includes: i) receiving a plurality of 2D images acquired with an ultrasound probe, ii) detecting an acceleration and an angular velocity of the ultrasound probe as the ultrasound probe is moved, via a plurality of sweeps, in a predetermined pattern over an anatomical area of interest in accordance with the blind sweep protocol, iii) generating a 3D image, with reference to the predetermined pattern, using the plurality of 2D images and based, at least partially, on the detected acceleration and angular velocity, iv)determining a validation metric for the 3D image, and v) communicating evaluation feedback, at least partially, based on the validation metric.

[0018] In various implementations, a processor or controller can be associated with one or more storage media (generically referred to herein as “memory,” e.g., volatile and non-volatile computer memory such as RDM, RAM, PROM, EPROM, and EEPROM, floppy disks, compact disks, optical disks, magnetic tape, Flash, OTP-ROM, SSD, HDD, etc.). In some implementations, the storage media can be encoded with one or more programs that, when executed on one or more processors and / or controllers, perform at least some of the functions discussed herein. Various storage media can be fixed within a processor or controller or can be transportable, such that the one or more programs stored thereon can be loaded into a processor or controller so as to implement various aspects as discussed herein. The terms “program” or “computer program” are used herein in a generic sense to refer to any type of computer code (e.g., software, firmware, or microcode) that can be employed to program one or more processors or controllers.

[0019] It should be appreciated that all combinations of the foregoing concepts and additional concepts discussed in greater detail below (provided such concepts are not mutually inconsistent) are contemplated as being part of the inventive subject matter disclosed herein. In particular, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the inventive subject matter disclosed herein. It should also be appreciated that terminology explicitly employed herein that also may appear in any disclosure incorporated by reference should be accorded a meaning most consistent with the particular concepts disclosed herein.

[0020] These and other aspects of the various embodiments will be apparent from and elucidated with reference to the embodiment(s) described hereinafter.Brief Description of the Drawings

[0021] In the drawings, like reference characters generally refer to the same parts throughout the different views. Also, the drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the various embodiments.

[0022] FIG. 1 is an illustration of an ultrasound system according to some aspects of the present disclosure.

[0023] FIG. 2 is a schematic of one embodiment of an inertial measurement unit according to some aspects of the present disclosure.

[0024] FIG. 3 is a schematic of an electromagnetic tracking system according to some aspects of the present disclosure.

[0025] FIG. 4 is a schematic of an ultrasound probe and a computer according to some aspects of the present disclosure.

[0026] FIG. 5 is an illustration of a blind sweep protocol according to some aspects of the present disclosure.

[0027] FIG. 6 is a method for evaluating a plurality of ultrasound sweeps in a blind sweep protocol according to some aspects of the present disclosure.

[0028] FIG. 7 is an illustration of a 3D image being generated from a plurality of 2D images according to some aspects of the present disclosure.

[0029] FIG. 8 is an illustration of one method of generating a 3D image from a plurality of 2D images according to some aspects of the present disclosure.

[0030] FIG. 9 is an illustration of a filling image being interpolated into a plurality of 2D images according to some aspects of the present disclosure.

[0031] FIG. 10 is a method for communicating an alphanumeric fidelity score according to some aspects of the present disclosure.

[0032] FIG. 11 is an illustration of an abridged sweep according to some aspects of the present disclosure.Detailed Description of Embodiments

[0033] The present disclosure is generally directed to systems and methods for evaluating an ultrasound imaging procedure. This includes acquiring a series of 2D ultrasound images by sweeping an ultrasound probe over a predetermined pattern over an anatomical area of interest, and then generating a reconstructed 3D image from the sequence of 2D images. The system or method further includes determining a validation metric for the reconstructed 3D image and communicating evaluation feedback based on the validation metric for the 3D image.

[0034] Figure 1 is an illustration of an ultrasound system 10 according to some aspects of the present disclosure. Ref er ri ng to Figure 1 , in various embodiments, the ultrasound system 10 includes an ultrasound probe 12 that acquires ultrasound images of an anatomical area of interest as it is moved along the body of a patient 16.

[0035] In particular embodiments, ultrasound system 10 is a portable app-based ultrasound system designed to perform diagnostic imaging at the point of care. Ultrasound probe 12 is a small handheld probe including an array of transducers and the circuitry necessary to producean ultrasound image and is further connectable to a compatible smart device, such as a smartphone or tablet running the suitable software application. In one such embodiment, the ultrasound system 10 is implemented based on Philips Lumify system available from Royal Philips, N.A. of Cambridge, Massachusetts. However, the handheld ultrasound probe 12, configured in accordance with the principles of this disclosure, could be deployed with other ultrasound systems, including mid-range and high-end non-portable ultrasound systems, without deviating from the scope and spirit of this disclosure.

[0036] Ultrasound probe 12 can be connected to a display device having a user interface 20 where the operator can view the acquired ultrasound images. In some embodiments, ultrasound probe 12 can be further connected to a tablet 18 that an operator uses to adjust various settings of the ultrasound probe 12. In other embodiments, the ultrasound probe 12 can be connected to only the tablet 18, so that the tablet is used both for displaying the acquired ultrasound images via the user interface 20 and controlling the ultrasound probe settings. The ultrasound probe could alternatively be connected to any other device suitable for displaying ultrasound images or controlling ultrasound probe settings, for example, a smartphone. The connection between the probe 12 and the display / control components can take the form of a wired connection, e.g. via USB port. The connection can also take the form of a wireless connection using any known wireless connection protocol and hardware, such as Bluetooth or Wi-Fi.

[0037] The ultrasound system 10 may also include one or more probe estimation sensors. The probe estimation sensors, for example, can be implemented as components of an inertial measurement unit (“IMU”) 14. The IMU 14 can be a separate device releasably attached to the ultrasound probe 12, or it can be integrated into the ultrasound probe 12. The probe estimation sensors may be configured to detect a three-dimensional acceleration and an angular velocity using an accelerometer 22 and a gyroscope 24, as shown in FIG. 2. Such an accelerometer 22 and gyroscope 24 are possible components of an IMU 14.

[0038] In some embodiments, the probe estimation sensors may include a magnetometer 26. Including the magnetometer 26 would have the advantage of leading to even more accurate measurements of angular velocity. For example, if using a sensor fusion methodology, magnetometer 26 may be used to compensate for gyroscope drift error in yaw angle estimation over time. For example, this can be applied to reducing errors when transforming acceleration measurements to a global frame.

[0039] As known in the art, a magnetometer is a sensor that is used to detect and measure magnetic fields. When deployed in conjunction with ultrasound probe 12, magnetometer 26 can be used to improve tracking and localization of the probe during imaging procedures. One way this can be achieved is by embedding a small magnetometer sensor in the ultrasound probe 12 itself. In addition to improving tracking of ultrasound probe 12, magnetometer 26 can also be used to reduce the effects of motion artifacts, such as hand tremors, on image fidelity. By detecting and compensating for small movements of the ultrasound probe 12, magnetometer 26 can help to produce more stable and consistent images, reducing the need for repeated scans and improving diagnostic accuracy.

[0040] Other tracking sensors and systems (e.g., magnetic tracking sensors and systems) could be used to track the probe motion and orientation in line with the scope and spirit of the invention. In some such embodiments, a fixed external magnetic field generator 28 along with three orthogonal coils 30 coupled to the ultrasound probe 12, can be provided as shown in FIG. 3. As the ultrasound probe 12 moves through the generated magnetic field, a voltage is induced in the three coils. This induced voltage can be used to determine the angular orientation and position of the ultrasound probe 12. These electromagnetic tracking methods can be used in addition to, or in place of, tracking with the IMU 14.

[0041] Referring back to FIG. 1 , in some embodiments, image processing of a video feed of one or multiple cameras could be used to track a movement of ultrasound probe 12 and its deviation from the intended path. In such embodiments, the system 10 can include an external image sensor 32. The image sensor 32 can take the form of, for example, an RGB or RGBD camera. The image sensor 32 can generate a video feed of the ultrasound probe’s 12 movements. This video feed can be used to visually track the movement of the ultrasound probe 12 as an additional or alternative technique for determining the location, acceleration, and angular orientation. The intended path or the probe itself can be marked using markers which are picked up by the camera. This visual tracking can be accomplished using any known visual odometry or simultaneous localization and mapping (“SLAM”) approach. For example, by tracking the centroid of the probe across the imaging field of known size at a known frame rate, or by using optical flow-based approaches. In another example, a neural network could be used that is trained on videos taken with the image sensor 32. The neural network could extract and calculate any deviation from the intended path indicated by the markers. The network may be a convolutional neural network which is trained on many images of ultrasound probes withdifferent known orientations moving in known directions on a marked surface. The network learns the association between the visible markers / features, in the consecutive video frames and probe location, orientation, and velocity.

[0042] Additionally or alternatively to image tracking with external image sensor 32, image processing of the ultrasound image frames themselves may be used for motion detection in supplement to or in absence of IMU 14 motion data.

[0043] One approach to using ultrasound image frames for motion detection is to compare consecutive frames and calculate the displacement of each pixel between frames. This displacement can be used to estimate the motion of the probe and to track its position and orientation over time. By analyzing multiple frames, it is possible to detect changes in the motion of the probe and to adjust for any movements or vibrations that may be affecting image fidelity. For instance, seeing a mother’s kidneys in fetal ultrasound images indicates that the probe is either on the right or on the left side of the mother’s abdomen, and the velocity of the probe can be estimating using displacements in these images given a known average size for adult kidneys. This processing of the ultrasound images themselves helps create a map of the probe’s location and movements.

[0044] Another approach is to use feature detection and tracking techniques to detect and track specific features in the ultrasound image frames, such as the edges of structures or the location of blood vessels. By tracking these features over time, it is possible to estimate the motion of the probe and to compensate for any movements or vibrations that may be affecting image quality.

[0045] Thus, ultrasound image frame-based motion detection can be particularly useful in situations where IMU 14 motion data is not available or unreliable, such as when using an ultrasound system that does not include IMU 14. It can also be used to supplement IMU 14 data to improve accuracy and robustness of motion detection. In some embodiments, motion between ultrasound image frames may be assessed by optical flow or by similar visual odometry methods. The start of a new sweep is detectable when the probe transitions from a static to moving state. The conclusion of a sweep is detectable as an end to probe motion. If coupling is maintained during intra-sweep transitions, relative position and orientation of sweeps may also be obtained by tracking probe repositioning via the ultrasound image frames. Sweeps may also be classified as longitudinal or cervical based on distance the probe travels or based on image characteristics, as discussed in more detail below.

[0046] As shown in FIG. 4, the ultrasound 12 transmits the acquired ultrasound images to a computing device 34. The computing device 34 may take several forms including being part of the tablet 18 discussed in conjunction with FIG. 1 . In some embodiments the computing device 34 may be in a location remote from the ultrasound probe 12, and the ultrasound probe 12 transmits the acquired ultrasound images to the remote computing device 34 according to any data transmission method previously known in the art. In particular embodiments, the computing device 34 includes a memory 36 for storing acquired ultrasound images and processor instructions. The computing device 34 may also include a signal processor 38 used to generate a 3D ultrasound image from a series of 2D ultrasound images, and further configured to determine a validation metric for the generated 3D ultrasound image.

[0047] In its various embodiments and implementations, the present disclosure involves image acquisition of an anatomical area of interest of a patient 16 via a blind sweep protocol, a technique used in ultrasound imaging that involves systematically scanning an area of interest without any prior knowledge of the location or orientation of the target anatomical structure. This technique is often used in applications where the target structure is difficult to visualize, such as in deep tissue imaging or when the target is obscured by surrounding structures, or as described herein, when assisting an inexperienced operator to conduct antenatal screening of gravid abdomen.

[0048] The parameters of a blind sweep protocol can vary depending on the specific imaging application and the type of ultrasound system being used. In general, the protocol will involve systematically scanning the area of interest using a variety of imaging modes and parameters to optimize image fidelity and increase the chances of detecting the target structure. Advantages of a blind sweep protocol include increased detection rate, because, by systematically scanning the entire area of interest, a blind sweep protocol can increase the chances of detecting the target structure, even if it is small or obscured by surrounding tissues. Further advantage is a reduced operator bias by removing the need for the operator to make assumptions about the location or orientation of the target structure. Further, by increasing the chances of detecting the target structure and reducing operator bias, a blind sweep protocol can help to improve diagnostic accuracy and reduce the likelihood of false negatives or false positives. Finally, a blind sweep protocol can help to ensure that imaging procedures are consistent and repeatable, reducing the likelihood of variability between scans and improving overall image quality.

[0049] FIG. 5 is an illustration of a blind sweep protocol according to some aspects of the present disclosure. The blind sweep protocol utilizes a predetermined pattern 40. The predetermined pattern 40 may be a grid shape, for example, an orthogonal grid, such as the one shown in FIG. 5. In the embodiment shown in FIG. 5, the predetermined pattern 40 is located along the abdomen of patient 16. However, other pattern locations or geometries may be appropriate depending on the particular ultrasound procedure being performed. The predetermined pattern 40 may be used to guide a novice operator in making a series of longitudinal sweeps 42 and lateral sweeps 44. The example embodiment of FIG. 5 shows predetermined pattern 40 including three longitudinal sweeps 42 and three lateral sweeps 44. As part of the ultrasound imaging process, the user will need to execute repositioning movements 46 between sweeps. For example, FIG. 5 shows repositioning movements 46 being made between each of the longitudinal sweeps 42. The repositioning movements 46 introduce a chance for error if the ultrasound operator fails to correctly pause the acquisition of ultrasound images during the repositioning movements 46.

[0050] FIG. 6 is a method 100 of evaluating a plurality of ultrasound sweeps in a blind sweep protocol according to some aspects of the present disclosure. Step 102 of the method 100, includes acquiring ultrasound images by sweeping an ultrasound probe 12 over a predetermined pattern. This can be achieved, for example, by sweeping the ultrasound probe 12 previously discussed over a predetermined pattern 40 such as the one shown in FIG. 5. Step 104 of method 100, includes detecting an acceleration and an angular velocity of the ultrasound probe 12 as the ultrasound probe 12 is swept over the predetermined pattern 40. Acceleration and angular velocity can be detected using any of the methods or devices previously discussed including: an IMU 14, electromagnetic tracking, image tracking with a camera, or any additional detection method previously known in the art.

[0051] In some embodiments, for example when using IMU 14, step 106 of method 100 includes fusing the acceleration and angular velocity data. For example, if the acceleration and the angular velocity are detected at different sampling rates, the sampling must be unified. This may be accomplished by any synchronization algorithm in the art, for example, by resampling one data stream to match the other or by only using samples taken at the same timepoint.

[0052] Once sampling frequencies are unified, there are several approaches to perform sensor fusion. For example, we can define a reference frame for IMU 14 where the origin is the IMU’s 14 location within the ultrasound probe 12. We can then define a three-dimensional coordinatesystem where the first two axes are formed along the long and short lengths of ultrasound probe 12, and the third axis is defined to be perpendicular to these first two axes. The global reference frame would then have as its origin the position of the ultrasound probe 12 at the image acquisition start time. The first axis would align with gravity and the other two axes would be defined by the orientation of the ultrasound probe 12 at the acquisition start time. These additional axes would nominally be aligned with the patient’s longitudinal and transverse axes. The probe estimation sensor may detect an angular velocity as opposed to directly detecting an angular orientation. This would be the case, for example if probe estimation sensor was implemented as a gyroscope. The angular orientation detected by probe estimation sensor would then be integrated to provide the orientation of the IMU 14 from both reference frames. Alternatively, angular velocity and acceleration data may be fused with an Extended Kalman Filter, Kalman Filter, Complementary Filter, Madgwick Filter, or similarly known sensor fusion methodology. The output in this case would be a more accurate estimate of orientation than from simple integration.

[0053] Once orientation is obtained, it is then used to transform the detected acceleration to the global reference frame. In the global reference frame, the gravitational acceleration is subtracted from the vertical axis acceleration. To obtain velocity and position, acceleration is single and double integrated, respectively.

[0054] Step 108 of method 100 includes generating a reconstructed 3D image, with reference to the predetermined pattern 40, using the plurality of 2D images 48. As seen in FIG. 7, the signal processor 38 receives a plurality of 2D images 48 and outputs a 3D image 50. This step can be implemented in several different ways.

[0055] In one example, the ultrasound probe 12 acquires the plurality of 2D images 48 using an array of one-dimensional transducers. The signal processor 38 applies a Fourier transform to the signals acquired by the one-dimensional transducers and combines these into the 3D image 50 in the frequency domain. This can be achieved in accordance with known methods, for example, as disclosed by Han et al. “3D ultrasound imaging in frequency domain with 1 D array transducer”, Ultrasonics, volume 76, pages 28-34 (December 2016), incorporated herein by reference. In this way, the 3D image 50 can be generated using raw ultrasound signals from the transducers. These raw ultrasound signals can either replace or supplement the acceleration and angular velocity data detected by the IMU 14. In the case where the acceleration and angular velocity are also used, this data can be used to map each of the plurality of 2D images48 to a particular location in 3D space. This would significantly improve the accuracy and fluidity of previously known 3D models built using only raw ultrasound data.

[0056] In another example, the signal processor 38 receives the plurality of 2D images 48 as the ultrasound probe 12 moves along the predetermined sweeping pattern 40. The signal processor may further receive the acceleration and angular velocity data detected by the IMU 14. As shown in FIG. 8, the signal processor 38 may use this acceleration and angular velocity data to determine areas of the plurality of 2D images 48 acquired either along the sweep paths 52, or along a space perpendicular to the sweep path 54. The signal processor 38 would then use only overlapping areas 56 along the space perpendicular to the sweep path 54 in reconstructing the 3D image 50. The overlapping areas 56 can be stitched together in accordance with known methods, for example, as disclosed by Deng et al. “Generating panorama photos”, Internet Multimedia Management Systems IV, Proceedings volume 5242 (2003), incorporated herein by reference. This method would have the advantage of reducing the number of visible misalignments in the generated 3D image 50.

[0057] In yet another example, the 3D image 50 could be generated from a neural network trained to estimate the underlying 3D anatomy of the patient 16 from which the plurality of 2D images 48 were acquired. The neural network may be a convolutional neural network (CNN), temporal convolutional network (TCN), recurrent neural network (RNN), transformer or any other variation known in the art. An RNN-based implementation may use unidirectional or bidirectional long short-term memory (LSTM) architecture. The network may include several fully connected or convolutional layers with pooling, normalization, dropout, or non-linear layers between them. The detected acceleration and angular orientation data can be used to train the neural network by projecting the 3D estimate back into the 2D space the ultrasound images were acquired from. The neural network would then reassess the projected 3D estimate in the 2D space to evaluate the loss of the 3D estimate. Additionally or alternatively, the neural network could use raw ultrasound data to estimate the underlying 3D anatomy of patient 16.

[0058] As shown in FIG. 9, the signal processor 38 can further improve the reconstructed 3D image 50 by generating and interpolating a filler image 58 into the plurality of 2D images 48. For example, if the ultrasound operator moves through a portion of the blind sweep protocol too quickly the ultrasound probe 12 may be unable to acquire enough 2D images 48 to produce a complete and continuous 3D image. In this case, the signal processor 38 could fill in areas lacking in image data with one or more filling image 58. This would create a more uniformdistribution of 2D images 48 along a given sweep. The filling image 58 could be produced using any interpolation method previously known in the art. For example, the signal processor 38 could produce filling image 58 by calculating a series of intermediate pixel values between each pixel of any two of the acquired 2D ultrasound images 48. Additionally or alternatively, the filling image 58 could be generated using a deep neural network architecture such as a generative adversarial network (“GAN”) or any of the other neural network configurations previously discussed. The neural network that generates the filling image 58 could be the same or a different neural network from the one that generates the 3D image 50 from the plurality of 2D images 48.

[0059] In some examples, the generated 3D image 50 may be a 3D model of an anatomical area of interest of patient 16. For example, the 3D image could be a 3D model of a heart.

[0060] Returning to FIG. 6, step 110 includes determining a validation metric for the 3D image 50. This validation metric can be determined a variety of ways.

[0061] In one example, the validation metric could be based on the completeness of the 3D image 50. The signal processor 38 may detect portions of 3D image 50 where there is an insufficient number of 2D images 48 to create an accurate and / or continuous 3D reconstruction. This would indicate a low level of completeness. Additionally or alternatively, the number of filling images 58 the signal processor 38 was required to generate and interpolate could be an indication of the completeness of the 3D image. More filling images 58 would indicate a less complete 3D model 50.

[0062] In one example, the validation metric could be based on parameters relating to the sweep fidelity. For example, sweeping with a uniform velocity throughout the blind sweep protocol would be an indication of higher sweep fidelity, as the plurality of 2D images 48 would be uniformly distributed among each area of the patient’s 16 anatomy of interest. Meanwhile, sweeps performed at a high velocity would indicate a lower sweep fidelity as they might result in an insufficient number of image frames for a reliable 3D reconstruction. Moreover, detected deviations from the predetermined pattern 40 could also be an indication of lower sweep fidelity.

[0063] In another example, the validation metric could be based on parameters relating to the ultrasound images themselves. For example, the signal processor 38 could evaluate the plurality of 2D images 48 for shadows obscuring the images or for overall image contrast.

[0064] In embodiments where the 3D image 50 is generated by a neural network, approaches such as Monte Carlo dropout could be used to determine the validation metric based on network confidence. In this approach, the network, trained with drop-out layers, may be run multiple times on the same input during inference to generate slightly different outputs (since dropout drops the outputs from a specified number of nodes at random). These different outputs can be used to compute the mean and variance in the reconstructed 3D image 50. The variance can be used to indicate a level of confidence in the output (i.e., high variance indicates that network output is not consistent and, therefore, confidence is low, while low variance indicates consistent output and high confidence).

[0065] In some embodiment, the validation metric could be an alphanumeric fidelity score that is communicated as evaluation feedback. For example, the 3D image 50 could be made up of a plurality of voxels. An example embodiment of a method 200 for calculating an alphanumeric quality score based the voxels of 3D image 50 is shown in FIG. 10. Step 202 of method 200, includes comparing a fidelity of each voxel in the 3D image 50 with a voxel threshold. This threshold can be determined using any of the previously discussed methods for determining the validation metric. Step 204 of method 200 includes determining a total number of voxels in the 3D image 50. This could, for example, be performed by the signal processor 38. Step 206 of method 200, includes comparing the number of voxels with fidelity greater than the voxel threshold to the total number of voxels. In some examples, this comparison includes calculating a ratio or percentage of the number of voxels with fidelity greater than or equal to the voxel threshold over the total number of voxels. Step 208 of method 200 includes communicating an alphanumeric fidelity score based on the comparison. This alphanumeric fidelity score could again be the calculated ratio or percentage. The comparison may take many forms depending on the particular application and / or imaging ultrasound procedure. Alternatively, this alphanumeric score could include a message, such as a message indicating that the model is “reliable”, “unreliable”, or “moderately reliable”.

[0066] Returning again to FIG. 6, step 112 of method 100 includes communicating evaluation feedback at least partially based on the validation metric. This feedback could be based on the fidelity of individual sweeps. This could, for example, take the form of visual feedback where the sweeps that were performed correctly are shown on the predetermined pattern 40 as green and the sweeps that need to be repeated are shown as red. Additionally or alternatively, the feedback could take the form of auditory feedback that outputs an auditory message to theultrasound probe 12 operator. Additionally or alternatively, the feedback could take the form of haptic feedback. In which case, the ultrasound probe 12 or tablet 18 could vibrate when the user deviates from the predetermined pattern 40. In some further examples, the system may communicate: a fidelity score for individual sweeps, the maximum and minimum ultrasound probe 12 acceleration / velocity, a maximum deviation from the predetermined pattern 40, or guidance to the operator for improving their next sweep.

[0067] In some examples, the evaluation feedback could include visually displaying the 3D image 50 and indicating to the operator less complete areas of the 3D image 50. The system may further communicate instructions to repeat all or a portion of the blind sweep protocol. Referring to FIG. 11 , the system can identify low-fidelity regions 60 on the 3D image 50. The sweeps on predetermined pattern 40 corresponding to the low-fidelity region 60 can be filled in or re-taken with a different sweep than in the original sweeping protocol. One way this can be achieved is by using the detected acceleration and angular velocity to determine locations in the blind sweep protocol that correspond to coordinates on the 3D image 50. The system 10 feedback may further include user interface 20 that including two separate displays, where one display shows the 3D image 50 and the other display shows both the path and orientation of the ultrasound probe 12, as detected by the IMU 14, and the areas of the predetermined pattern 40 corresponding to the low-fidelity regions 60.

[0068] This 3D image 50, along with the motion tracking information from the IMU 14 allows for further guidance of the ultrasound probe 12 operator. In some examples, the system 10 guides the ultrasound probe 12 operator to fill in low-fidelity regions 60 with an abridged sweep 62, as opposed to the complete longitudinal 42 or lateral 44 sweep shown in FIG. 5. Such an abridged sweep 62 can be a subset of the original sweeps, but could also be a shorter sweep in a completely different direction, so long as the low-fidelity region 60 is covered appropriately. The start position will be conveyed to the operator via the user interface 20, and the sweep will be monitored via the IMU 14 for adherence to the intended pattern. The benefit to the operator is shorter additional sweeps that they can complete faster than if they were to repeat the entire blind sweep protocol.

[0069] All definitions, as defined and used herein, should be understood to control over dictionary definitions, definitions in documents incorporated by reference, and / or ordinary meanings of the defined terms.

[0070] The indefinite articles “a” and “an,” as used herein in the specification and in the claims, unless clearly indicated to the contrary, should be understood to mean “at least one.”

[0071] The phrase “and / or,” as used herein in the specification and in the claims, should be understood to mean “either or both” of the elements so conjoined, i.e., elements that are conjunctively present in some cases and disjunctively present in other cases. Multiple elements listed with “and / or” should be construed in the same fashion, i.e., “one or more” of the elements so conjoined. Other elements can optionally be present other than the elements specifically identified by the “and / or” clause, whether related or unrelated to those elements specifically identified.

[0072] As used herein in the specification and in the claims, “or” should be understood to have the same meaning as “and / or” as defined above. For example, when separating items in a list, “or” or “and / or” shall be interpreted as being inclusive, i.e., the inclusion of at least one, but also including more than one, of a number or list of elements, and, optionally, additional unlisted items. Only terms clearly indicated to the contrary, such as “only one of” or “exactly one of,” or, when used in the claims, “consisting of,” will refer to the inclusion of exactly one element of a number or list of elements. In general, the term “or” as used herein shall only be interpreted as indicating exclusive alternatives (i.e. “one or the other but not both”) when preceded by terms of exclusivity, such as “either,” “one of,” “only one of,” or “exactly one of.”

[0073] As used herein in the specification and in the claims, the phrase “at least one,” in reference to a list of one or more elements, should be understood to mean at least one element selected from any one or more of the elements in the list of elements, but not necessarily including at least one of each and every element specifically listed within the list of elements and not excluding any combinations of elements in the list of elements. This definition also allows that elements can optionally be present other than the elements specifically identified within the list of elements to which the phrase “at least one” refers, whether related or unrelated to those elements specifically identified.

[0074] It should also be understood that, unless clearly indicated to the contrary, in any methods claimed herein that include more than one step or act, the order of the steps or acts of the method is not necessarily limited to the order in which the steps or acts of the method are recited.

[0075] In the claims, as well as in the specification above, all transitional phrases such as “comprising,” “including,” “carrying,” “having,” “containing,” “involving,” “holding,” “composed of,”and the like are to be understood to be open-ended, i.e., to mean including but not limited to. Only the transitional phrases “consisting of” and “consisting essentially of” shall be closed or semi-closed transitional phrases, respectively.

[0076] The above-described examples of the described subject matter can be implemented in any of numerous ways. For example, some aspects can be implemented using hardware, software, or a combination thereof. When any aspect is implemented at least in part in software, the software code can be executed on any suitable processor or collection of processors, whether provided in a single device or computer or distributed among multiple devices / computers.

[0077] The present disclosure can be implemented as a system, a method, and / or a computer program product at any possible technical detail level of integration. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present disclosure.

[0078] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM) , a digital versatile disk (DVD) , a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0079] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local areanetwork, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0080] Computer readable program instructions for carrying out operations of the present disclosure can be assembler instructions, instruction-set -architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, configuration data for integrated circuitry, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++, or the like, and procedural programming languages, such as the “C” programming language or similar programming languages. The computer readable program instructions can execute entirely on the user’s computer, partly on the user's computer, as a stand-alone software package, partly on the user’s computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some examples, electronic circuitry including, for example, programmable logic circuitry, field- programmable gate arrays (FPGA), or programmable logic arrays (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

[0081] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to examples of the disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0082] The computer readable program instructions can be provided to a processor of a, special purpose computer, or other programmable data processing apparatus to produce a machine,such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function / act specified in the flowchart and / or block diagram or blocks.

[0083] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0084] The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various examples of the present disclosure. In this regard, each block in the flowchart or block diagrams can represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the Figures. For example, two blocks shown in succession can, in fact, be executed substantially concurrently, or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustration, and combinations of blocks in the block diagrams and / or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.

[0085] Other implementations are within the scope of the following claims and other claims to which the applicant can be entitled.

[0086] While various examples have been described and illustrated herein, those of ordinary skill in the art will readily envision a variety of other means and / or structures for performing the function and / or obtaining the results and / or one or more of the advantages described herein, andeach of such variations and / or modifications is deemed to be within the scope of the examples described herein. More generally, those skilled in the art will readily appreciate that all parameters, dimensions, materials, and configurations described herein are meant to be exemplary and that the actual parameters, dimensions, materials, and / or configurations will depend upon the specific application or applications for which the teachings is / are used. Those skilled in the art will recognize or be able to ascertain using no more than routine experimentation, many equivalents to the specific examples described herein. It is, therefore, to be understood that the foregoing examples are presented by way of example only and that, within the scope of the appended claims and equivalents thereto, examples can be practiced otherwise than as specifically described and claimed. Examples of the present disclosure are directed to each individual feature, system, article, material, kit, and / or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and / or methods, if such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent, is included within the scope of the present disclosure.

Claims

ClaimsWhat is claimed is:1 . An ultrasound system (10), configured to evaluate a plurality ultrasound sweeps in a blind sweep protocol, comprising: an ultrasound probe (12) configured to acquire a plurality of two-dimensional (2D) images (48) of an anatomical area of interest; a signal processor (38) configured to generate a three-dimensional (3D) image (50) of at least a portion of the anatomical area of interest based at least in part on the plurality of 2D images (48), determine a validation metric for the 3D image (50) ; and a user interface (20) configured to communicate evaluation feedback at least partially based upon the validation metric for the 3D image (50).

2. The system (10) of claim 1 , further comprising an inertial measurement unit (14) integrated with, or releasably attached to, the ultrasound probe (12), the inertial measurement unit (14) comprising one or more probe estimation sensors configured to detect an acceleration and an angular velocity of the ultrasound probe (12) when moved through the plurality of sweeps in a predetermined pattern (40), along the anatomical area of interest, in accordance with the blind sweep protocol, wherein the signal processor (38) generates the 3D image (50) at least partially based on the detected acceleration and angular velocity.

3. The system (10) of claim 2, wherein the probe estimation sensors are selected from the group consisting of an accelerometer (22), a gyroscope (24), and a magnetometer (26).

4. The system (10) of claim 1 , further comprising an image sensor (32) configured to track a movement of the ultrasound probe (12), and wherein the signal processor (38) is configured to generate the 3D image (50), at least partially based on the movement tracked by the image sensor (32).

5. The system (10) of claim 1 , wherein the validation metric is based, at least partially on, a completeness of the 3D image (50) and wherein the evaluation feedback comprises identifying one or more low-fidelity regions (60).

6. The system (10) of claim 1 , wherein the evaluation feedback comprises an alphanumeric fidelity score.

7. The ultrasound system (10) of claim 1 , wherein the evaluation feedback comprises an instruction to repeat at least a portion of the blind sweep protocol.

8. The ultrasound system (10) of claim 7, wherein the evaluation feedback further comprises an instruction to perform an abridged sweep (62) , wherein at least a portion the abridged sweep (62) is distinct from the blind sweep protocol.

9. The ultrasound system (10) of claim 1 , wherein the signal processor (38) is configured to generate the 3D image (50) by inputting the plurality of 2D images (48) into a neural network.

10. The ultrasound system (10) of claim 1 , wherein the signal processor (38) is configured to generate the 3D image (50) by inserting a filling image (58) into the plurality of 2D images (48).11 . The system (10) of claim 10, wherein the signal processor (38) is configured to generate the filling image (58) at least partially based on calculating an intermediate pixel image value between two or more images of the plurality of 2D images (48).

12. The system (10) of claim 1 , wherein the 3D image (50) comprises a plurality of voxels, and the signal processor (38) is configured to determine the validation metric at least partially based on comparing a fidelity of each voxel of the plurality of voxels with a voxel threshold.

13. The system (10) of claim 12, wherein the signal processor (38) is further configured to determine the validation metric by comparing a number of voxels with fidelity greater than or equal to the voxel threshold to a total number of voxels in the 3D image (50) .

14. A method (100) for evaluating a plurality of ultrasound sweeps in a blind sweep protocol, comprising: receiving (102) a plurality of 2D images (48) acquired with an ultrasound probe (12) ;detecting (104) an acceleration and an angular velocity of the ultrasound probe (12) as the ultrasound probe (12) is moved, via a plurality of sweeps, in a predetermined pattern (40) over an anatomical area of interest in accordance with the blind sweep protocol; generating (106) a 3D image (50), with reference to the predetermined pattern (40), using the plurality of 2D images (48) and based, at least partially, on the detected acceleration and the angular velocity; determining (110) a validation metric for the 3D image (50); and communicating (112) evaluation feedback at least partially based upon the validation metric.

15. The method (10) of claim 14, wherein the validation metric is based, at least partially on, a completeness of the 3D image (50) and wherein the evaluation feedback comprises identifying one or more low-fidelity regions (60).

Images