System and methods for automatic labelling of ultrasound blind sweeps

Abstract

An ultrasound system, configured for image acquisition of an anatomical area of interest of a patient via a blind sweep protocol comprising: i) an ultrasound probe configured to acquire ultrasound images; ii) a signal processor configured to differentiate between sweeps and to classify the sweeps with reference to the predetermined pattern; and iii) a user interface for communicating the blind sweep protocol and an outcome of the sweep classification to a user.

Description

SYSTEM AND METHODS FOR AUTOMATIC LABELLING OF ULTRASOUND BLIND SWEEPSField of the Disclosure

[0001] The present disclosure generally relates to medical diagnostic ultrasound systems. More specifically, the present disclosure is directed to systems and methods for guiding an ultrasound imaging procedure by classifying ultrasound sweeps.Background

[0002] Ultrasound diagnostic systems have become increasingly portable in recent years. In the early 2000s, system miniaturization started with desktop unit, then laptop-based configurations appeared. In recent years, there has been an increasing need for medical ultrasound imaging equipment that is portable to allow medical personnel to easily transport and deploy the equipment to and from or within healthcare facility and / or field locations, and more user-friendly to accommodate medical personnel who may possess a range of skill levels.

[0003] A blind sweep is a known protocol for helping a novice operator acquire ultrasound images. In a blind sweep protocol, the novice operator sweeps the ultrasound over a predetermined pattern, as opposed to moving the probe freely. This pre-determined pattern greatly simplifies the procedure because the operator is not required to interpret the ultrasound images in real-time and reposition the device according to these interpretations.

[0004] However, even when sweeping in a pre-determined pattern, there is still a significant chance of user error. For example, the operator still must manually start and stop the individual sweeps. This creates the possibility of the novice operator failing to stop one sweep before starting the next one. Alternatively, one or more sweeps may be misclassified. Concatenating multiple sweeps or misclassifying sweeps may lead to the erroneous localization of anatomical structures of interest.

[0005] Accordingly, there still exists a need in the art for systems and methods that automatically differentiate and classify ultrasound sweeps based upon information detected from the ultrasound sweeps themselves so that the ultrasound probe can acquire accurate images even with a novice operator.Summary of the Disclosure

[0006] The present disclosure is generally directed to guiding an ultrasound imaging procedure by providing feedback based at least partially on differentiating and classifying ultrasound sweeps. This includes receiving ultrasound images acquired by an ultrasound probe as the ultrasound probe is moved through a plurality of sweeps in a predetermined pattern, along an anatomical area of interest, in accordance with a blind sweep protocol, differentiating between the sweeps and classifying the sweeps with reference to the predetermined pattern, and communicating the outcome of the sweeps classification to a user. The systems and methods disclosed herein improve upon conventional approaches partially through the realization that information determined from the sweeps themselves, such as acceleration and angular orientation information of the probe can be used to differentiate and classify ultrasound sweeps. Moreover, some ultrasound probes, such as Philips Lumify™ probe include a built-in inertial measurement unit ("IMU”) that could provide one approach for determining the information needed to differentiate and classify the sweeps. Accordingly, the disclosed systems and methods can automatically differentiate and classify ultrasound sweeps and use these classified sweeps to provide feedback that guides a less experienced operator through an ultrasound procedure.

[0007] Generally, in one aspect, the disclosure relates to an ultrasound system configured for image acquisition of an anatomical area of interest of a patient via a blind sweep protocol. The ultrasound system includes an ultrasound probe configured to acquire ultrasound images as the ultrasound probe is moved through a plurality of sweeps in a predetermined pattern (40), along the anatomical area of interest, in accordance with the blind sweep protocol. The system further includes a signal processor configured to differentiate between sweeps and to classify the sweeps, with reference to the predetermined pattern. The system further includes a user interface for communicating the blind sweep protocol and an outcome of the sweep classification to a user.

[0008] In some embodiments, the predetermined pattern includes a grid. In some other aspects, the signal processor is configured to classify at least two of the sweeps from the plurality of sweeps by heading the at least two sweeps as a lateral sweep or a longitudinal sweep relative to the anatomical area of interest and, for any two sweeps having the same heading, to determine positioning of the two sweeps with the same heading relative to each other.

[0009] The system may further include an inertial measurement unit integrated with, or releasably attached to, the ultrasound probe. The inertial measurement unit includes one or more probe estimation sensors, such as an accelerometer and a gyroscope. The accelerometer and gyroscope are configured to detect an acceleration and an angular velocity of the ultrasound probe when moved through the plurality of sweeps in a predetermined pattern, along the anatomical area of interest, in accordance with the blind sweep protocol. In some embodiments, the inertial measurement unit further includes a magnetometer.

[0010] In some embodiments, the signal processor is configured to calibrate the inertial measurement unit by i) placing the ultrasound probe in a fixed orientation, ii) detecting an acceleration and angular velocity using the inertial measurement unit for a predetermined time period, iii) calculating a bias for the inertial measurement unit) and iv) correcting for the calculated bias.

[0011] In some further aspects, the signal processor is configured to classify sweeps relative to an imaging boundary based on a reference sweep, the reference sweep being between the pubis and sternum. The signal processor may be further configured to classify the sweeps at least partially based on a relative location of the sweeps to the reference sweep.

[0012] In yet another embodiment, the signal processor is further configured to classify the sweeps at least partially based on machine learning classification.

[0013] In some aspects, the system may further include an external image sensor configured to detect movement data of the ultrasound probe. The signal processor may be configured to differentiate between sweeps and to classify the sweeps, with reference to the predetermined pattern, using the detected ultrasound probe movement data.

[0014] In some embodiments, the signal processor is further configured to detect coupling and uncoupling of the ultrasound probe, using the acquired ultrasound images, and to differentiate and / or classify the sweeps at least partially based on the detected coupling and uncoupling. The signal processor may be further configured to classify the sweeps based at least partially on an anatomical landmark within the ultrasound images.

[0015] In a further embodiment, the system further includes an electromagnetic field generator and a coil, wherein the coil is coupled to the ultrasound probe. The signal processor determines a position, velocity, and / or angular orientation of the ultrasound probe using a voltage generated in the coil. The signal processor further differentiates between sweeps and classifiesthe sweeps based at least partially on the determined position, velocity, and / or angular orientation.

[0016] A further aspect of this disclosure relates to a method for providing feedback of image acquisition of an anatomical area of interest of a patient in a blind sweep protocol. The method comprising: i) receiving ultrasound images acquired by an ultrasound probe as the ultrasound probe is moved through a plurality of sweeps in a predetermined pattern, along the anatomical area of interest, in accordance with the blind sweep protocol; , ii) differentiating between sweeps and classifying the sweeps with reference to the predetermined pattern, and iii) communicating an outcome of the sweep classification to a user.

[0017] In some embodiments, the method further includes detecting an acceleration and an angular velocity of the ultrasound probe by an inertial measurement unit integrated with, or releasably attached to, the ultrasound probe, such that the step of differentiating between sweeps and classifying the sweeps is based, at least in part on, detecting the acceleration and the angular velocity of the ultrasound probe.

[0018] In various implementations, a processor or controller can be associated with one or more storage media (generically referred to herein as "memory,” e.g., volatile and non-volatile computer memory such as ROM, RAM, PROM, EPROM, and EEPROM, floppy disks, compact disks, optical disks, magnetic tape, Flash, OTP-ROM, SSD, HDD, etc.). In some implementations, the storage media can be encoded with one or more programs that, when executed on one or more processors and / or controllers, perform at least some of the functions discussed herein. Various storage media can be fixed within a processor or controller or can be transportable, such that the one or more programs stored thereon can be loaded into a processor or controller so as to implement various aspects as discussed herein. The terms "program” or "computer program” are used herein in a generic sense to refer to any type of computer code (e.g., software, firmware, or microcode) that can be employed to program one or more processors or controllers.

[0019] It should be appreciated that all combinations of the foregoing concepts and additional concepts discussed in greater detail below (provided such concepts are not mutually inconsistent) are contemplated as being part of the inventive subject matter disclosed herein. In particular, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the inventive subject matter disclosed herein. It should alsobe appreciated that terminology explicitly employed herein that also may appear in any disclosure incorporated by reference should be accorded a meaning most consistent with the particular concepts disclosed herein.

[0020] These and other aspects of the various embodiments will be apparent from and elucidated with reference to the embodiment(s) described hereinafter.Brief Description of the Drawings

[0021] In the drawings, like reference characters generally refer to the same parts throughout the different views. Also, the drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the various embodiments.

[0022] FIG. 1 is an illustration of an ultrasound system according to some aspects of the present disclosure.

[0023] FIG. 2 is a schematic of an inertial measurement unit according to some aspects of the present disclosure.

[0024] FIG. 3 is a schematic of an electromagnetic tracking system, according to some aspects of the present disclosure.

[0025] FIG. 4 is a schematic of an ultrasound probe and a computer according to some aspects of the present disclosure.

[0026] FIG. 5 is an illustration of a blind sweep protocol according to some aspects of the present disclosure.

[0027] FIG. 6 is a schematic of a possible sweep classification according to some aspects of the present disclosure.

[0028] FIG. 7 is a method of communicating sweep feedback in an ultrasound imaging procedure according to some aspects of the present disclosure.

[0029] FIG. 8 is an illustration of an anatomical landmark within a predetermined sweeping pattern according to some aspects of the present disclosure.

[0030] FIG. 9 is a method of calibrating an inertial measurement unit, according to some aspects of the present disclosure.

[0031] FIG. 10 is an illustration of an ultrasound reference sweep according to some aspects of the present disclosure.Detailed Description of Embodiments

[0032] The present disclosure is generally directed to guiding an ultrasound imaging procedure by classifying ultrasound sweeps based on information detected by the sweeps themselves and using this classification to provide feedback to an ultrasound operator in a blind sweep protocol.

[0033] Figure 1 is an illustration of an ultrasound system 10 according to some aspects of the present disclosure. Referring to Figure 1 , in various embodiments, the ultrasound system 10 includes an ultrasound probe 12 that acquires ultrasound images of an anatomical area of interest as it is moved along the body of a patient 16.

[0034] In particular embodiments, the ultrasound system 10 is a portable app-based ultrasound system designed to perform diagnostic imaging at the point of care. Ultrasound probe 12 is a small handheld probe including an array of transducers and the circuitry necessary to produce an ultrasound image and connectable to a compatible smart device, such as a smartphone or tablet running the suitable software application. In one such embodiment, the ultrasound system 10 is implemented based on Philips Lumify system available from Royal Philips, N.A. of Cambridge, Massachusetts. However, the handheld probe 12 configured in accordance the principles of this disclosure could be deployed with other ultrasound systems, including midrange and high-end non-portable systems, without deviating from the scope and spirit of the invention.

[0035] The ultrasound probe 12 can be connected to a display device having a user interface 20 where the operator can view the acquired ultrasound images. In some embodiments, the ultrasound probe 12 can further be connected to a tablet 18 that an operator can use to adjust various settings of the ultrasound probe 12. In other embodiments, the ultrasound probe 12 can be connected to only the tablet 18, so that the tablet is used both for displaying the acquired ultrasound images via the user interface 20 and controlling the ultrasound probe settings. The ultrasound probe could alternatively be connected to any other device suitable for displaying ultrasound images or controlling ultrasound probe settings, for example, a smartphone. The connection between the probe 12 and the display / control components can take the form of a wired connection, e.g. via USB port. The connection can also take the form of a wireless connection using any known wireless connection protocol and hardware, such as Bluetooth or Wi-Fi.

[0036] The ultrasound system 10 may also include one or more probe estimation sensors. The probe estimation sensors, for example, can be implemented as components of an inertial measurement unit ("I MU”) 14. The I MU 14 can be a separate device releasably attached to the ultrasound probe 12, or it can be integrated into the ultrasound probe 12. The probe estimation sensors may be configured to detect a three-dimensional acceleration and angular velocity using an accelerometer 22 and a gyroscope 24, as shown in FIG. 2. Such an accelerometer 22 and gyroscope 24 may both be components of an IMU 14.

[0037] In some embodiments, the probe estimation sensors further include a magnetometer 26 to mitigate the yaw drift of the gyroscope. As known in the art, a magnetometer is a sensor that is used to detect and measure magnetic fields. Including the magnetometer 26 would have the advantage of leading to even more accurate measurements of angular velocity. For example, if using a sensor fusion methodology, a magnetometer may be used to compensate for gyroscope drift error in yaw angle estimation over time. This can be applied to reducing errors when transforming acceleration measurements to a global frame and in predicting probe heading.

[0038] In addition to improving tracking of the ultrasound probe, tracking sensors and systems can also be used to reduce the effects of motion artifacts, such as hand tremors, on image quality. By detecting and compensating for small movements of the probe, they can help to produce more stable and consistent images, reducing the need for repeated scans and improving diagnostic accuracy.

[0039] Other tracking sensors and systems (e.g., magnetic tracking sensors and systems) could be used to track the probe motion and orientation in line with the scope and spirit of the invention. In such embodiments, a fixed external magnetic field generator 28 along with three orthogonal coils 30 coupled to the ultrasound probe 12, can be provided as shown in FIG. 3. As the ultrasound probe 12 moves through the generated magnetic field, a voltage is induced in the three coils. This induced voltage can be used to determine the angular velocity, orientation and / or position of the ultrasound probe 12. These electromagnetic tracking methods can be used in addition to, or in place of, tracking with an IMU.

[0040] Referring back to FIG. 1 , in some embodiments, image processing of a video feed of one or multiple cameras could be used to track the probe motion and its deviation from the intended path. In such embodiments, the system 10 can include an external image sensor 32.The image sensor 32 can take the form of, for example, an RGB or RGBD camera. The image sensor 32 can generate a video feed of the ultrasound probe’s 12 movements. This video feed can be used to visually track the movement of the ultrasound probe 12 as an additional or alternative technique for determining the location, acceleration and angular orientation. The intended path or the probe itself can be marked using markers which are picked up by the camera. This visual tracking can be accomplished using any known visual odometry or simultaneous localization and mapping ("SLAM”) approach. For example, by tracking the centroid of the probe across the imaging field of known size at a known frame rate, or by using optical flow-based approaches. In another example, a neural network could be used that is trained on videos taken with the image sensor 32. The neural network could extract and calculate any deviation from the intended path indicated by the markers. The network may be a convolutional neural network which is trained on many images of ultrasound probes with different known orientations moving in known directions on a marked surface. The network learns the association between the visible markers / features in the consecutive video frames and probe location, orientation and velocity.

[0041] Furthermore, ultrasound image frames may be used for motion detection in supplement to or in absence of IMU 14 motion data. The motion of the probe can be detected by analyzing changes in the ultrasound image frames over time.

[0042] One approach to using ultrasound image frames for motion detection is to compare consecutive frames and calculate the displacement of each pixel between frames. This displacement can be used to estimate the motion of the probe and to track its position and orientation over time. By analyzing multiple frames, it is possible to detect changes in the motion of the probe and to adjust for any movements or vibrations that may be affecting image quality.

[0043] Another approach is to use feature detection and tracking techniques to detect and track specific features in the ultrasound image frames, such as the edges of structures or the location of blood vessels. By tracking these features over time, it is possible to estimate the motion of the probe and to compensate for any movements or vibrations that may be affecting image quality.

[0044] Thus, ultrasound image frame-based motion detection can be particularly useful in situations where IMU 14 motion data is not available or unreliable, such as when using aultrasound system that does not include an IMU 14. It can also be used to supplement IMU 14 data to improve accuracy and robustness of motion detection. In some embodiments, motion between ultrasound image frames may be assessed by optical flow or by similar visual odometry methods. The start of a new sweep is detectable when the probe transitions from a static to moving state. The conclusion of a sweep is detectable as an end to probe motion. If coupling is maintained during intra-sweep transitions, relative position and orientation of sweeps may also be obtained by tracking probe repositioning via the ultrasound image frames. Sweeps may also be classified as longitudinal or cervical based on distance the probe travels or based on image characteristics, as discussed in more detail below.

[0045] As shown in FIG. 4, the ultrasound probe 12 transmits the acquired ultrasound images to a computing device 34. The computing device 34 may take several forms including being part of the tablet 18 discussed in conjunction with FIG. 1 . In some embodiments the computing device 34 may be in a location remote from the ultrasound probe 12, and the ultrasound probe 12 transmits the acquired ultrasound images to the remote computing device 34 according to any method previously known in the art. In particular embodiments, the computing device 34 includes a memory 36 for storing acquired ultrasound images and processor instructions, and a signal processor 38 configured to differentiate between sweeps and to classify the sweeps.

[0046] In its various embodiments and implementations, the present disclosure involves image acquisition of an anatomical area of interest of a patient 16 via a blind sweep protocol, a technique used in ultrasound imaging that involves systematically scanning an area of interest without any prior knowledge of the location or orientation of the target anatomical structure. This technique is often used in applications where the target structure is difficult to visualize, such as in deep tissue imaging or when the target is obscured by surrounding structures, or, as described herein, when assisting an inexperienced operator to conduct antenatal screening of gravid abdomen.

[0047] The parameters of a blind sweep protocol can vary depending on the specific imaging application and the type of ultrasound system being used. In general, the protocol will involve systematically scanning the area of interest using a variety of imaging modes and parameters to optimize image quality and increase the chances of detecting the target structure.Advantages of a blind sweep protocol include increased detection rate, because, by systematically scanning the entire area of interest, a blind sweep protocol can increase the chances of detecting the target structure, even if it is small or obscured by surrounding tissues.Further advantage is a reduced operator bias by removing the need for the operator to make assumptions about the location or orientation of the target structure. Further, by increasing the chances of detecting the target structure and reducing operator bias, a blind sweep protocol can help to improve diagnostic accuracy and reduce the likelihood of false negatives or false positives. Finally, a blind sweep protocol can help to ensure that imaging procedures are consistent and repeatable, reducing the likelihood of variability between scans and improving overall image quality.

[0048] FIG. 5 is an illustration of a blind sweep protocol according to some aspects of the present disclosure. The blind sweep protocol utilizes a predetermined pattern 40. The predetermined pattern 40 may be a grid shape, for example, an orthogonal grid, such as the one shown in FIG. 5. In the embodiment shown in FIG. 5, the predetermined pattern 40 is located along the abdomen of patient 16. However, other pattern locations or geometries may be appropriate depending on the particular ultrasound protocol over an anatomical area of interest. The predetermined pattern 40 may be used to guide a novice or inexperienced operator in making a series of longitudinal sweeps 42 and lateral sweeps 44. The example embodiment of FIG. 5 shows predetermined pattern 40 comprising three longitudinal sweeps 42 and three lateral sweeps 44. As part of the ultrasound imaging process, the user will need to execute repositioning movements 46 between sweeps. For example, FIG. 5 shows repositioning movements 46 being made between each of the longitudinal sweeps 42. The repositioning movements 46 introduce a chance for error if the ultrasound operator fails to correctly pause the acquisition of ultrasound images during the repositioning movements 46.

[0049] FIG. 6 is a schematic of a possible sweep heading classification approach for an ultrasound probe 12, according to one embodiment of the invention. In this example, the signal processor 38 classifies movements of the ultrasound probe 12 according to a four-quadrant system to classify the sweep heading, as one of the elements of the sweep classification solution disclosed herein. The four quadrants may comprise an origin 48, a superior quadrant 50, an inferior quadrant 52, a right quadrant 54, and a left quadrant 56. This is just one example of a system for classifying movement of the ultrasound probe 12. As discussed further below, this disclosure contemplates several different ways of classifying movement of the ultrasound probe 12.

[0050] FIG. 7 is a method 100 for communicating feedback in an ultrasound imaging procedure according to some aspects of the present disclosure. Step 102 of the method 100, comprisesreceiving ultrasound images acquired by an ultrasound probe 12. This can be achieved, for example, by sweeping the ultrasound probe 12 previously discussed and transmitting the acquired ultrasound images to signal processor 38. In some embodiments, step 104 of method 100, includes detecting ultrasound probe 12 movement data as the ultrasound probe is moved, via a plurality of sweeps, in a predetermined pattern over the anatomical area of interest in accordance with the blind sweep protocol. Ultrasound probe movement data can include acceleration and angular velocity or their single / double integrals (e.g., position, orientation), and can be detected using any of the methods or devices previously discussed including: an IMU, electromagnetic tracking, external image-based tracking with a camera, or any detection method previously known in the art. The blind sweep protocol, for example, could be realized by sweeping an ultrasound probe 12 along the previously discussed predetermined pattern 40.

[0051] In some cases, for example when the ultrasound probe is detected using an IMU, step 106 of method 100 includes fusing the acceleration and angular velocity data detected by the IMU 14. For example, if the acceleration and the angular velocity are detected at different sampling rates, the sampling must be unified. This may be accomplished by any synchronization algorithm in the art, for example, by resampling one data stream to match the other or by only using samples taken at the same timepoint.

[0052] Once sampling frequencies are unified, there are several approaches to perform sensor fusion. For example, we can define a reference frame for IMU14 where the origin is the IMU’s 14 location within the ultrasound probe 12. We can then define a three-dimensional coordinate system where the first two axes are formed along the long and short lengths of ultrasound probe 12, and the third axis is defined to be perpendicular to these first two axes. Meanwhile, the global reference frame would have as its origin, the position of the ultrasound probe 12 at the image acquisition start time. The first axis of the global reference frame would align with gravity and the other two axes would be defined by the orientation of the probe at the acquisition start time. These additional axes would nominally be aligned with the patient’s 16 longitudinal and transverse axes. The IMU 14 may detect an angular velocity as opposed to directly detecting an angular orientation. This would be the case, for example, if the probe estimation sensor was implemented as a gyroscope 24. The angular velocity detected by the probe estimation sensor would then be integrated to provide the orientation of the IMU 14from both reference frames. Alternatively, angular velocity and acceleration data may be fused with an Extended Kalman Filter, Kalman Filter, Complementary Filter, Madgwick Filter, or similarlyknown sensor fusion methodology. The output in this case would be a more accurate estimate of orientation than from simple integration.

[0053] Once orientation is obtained, it is then used to transform the detected acceleration to the global reference frame. In the global reference frame, the gravitational acceleration is subtracted from the vertical axis acceleration. To obtain velocity and position, acceleration is single and double integrated, respectively.

[0054] Step 108 of method 100 includes differentiating the ultrasound sweeps at least partially based upon the acceleration and angular orientation. This step can be implemented in several ways. For example, using a detected change in acceleration. Repositioning of the ultrasound probe 12 could be identifiable as sudden increase in acceleration along the z axis as the probe is lifted from the abdomen, followed by acceleration along the x or y axis as the probe is moved to the start position of the next sweep. Additionally or alternatively, the method could utilize an image sensor 32 that differentiates the sweeps by tracking the ultrasound probe 12 movement in an acquired video feed. Additionally or alternatively, the method could differentiate ultrasound sweeps based on analyzing changes in the ultrasound images themselves. For example, the interval between the end of the previous sweep to the beginning of the next sweep acquisition may be determined from ultrasound image characteristics. If the area in the far field of the ultrasound image becomes dark and does not show any anatomical inclusions, it can be an indication of the probe being removed from the skin (decoupling). The length of this interval may further be used to differentiate between accidental and intentional decoupling of the probe. Combining coupling / decoupling detection with image-based motion detection detailed above may yield further improvements in accuracy.

[0055] Step 110 of method 100 comprises classifying at least two sweeps of the plurality of sweeps. For example, the individual sweeps can be assigned headings based on being longitudinal sweeps 42 or lateral sweeps 44 as shown in FIG. 5. Additionally or alternatively, the individual sweeps can be classified with headings according to the quadrant classification system shown in FIG. 6. In this embodiment sweeps would receive different headings based on being in the superior quadrant 50, the inferior quadrant 52, the right quadrant 54, or the left quadrant 56.

[0056] In a machine-learning based approach, acceleration and angular velocity data (or other motion data) may be fed directly to a sweep classification algorithm. This could take the formof a traditional machine learning classification or clustering algorithm, or a deep learning classification model. Various levels of pre-processing may also be performed prior to differentiation by the model. For example, continuous orientation measurements may be discretized into quadrants or absolute trajectories according to the previously discussed classification methods.

[0057] In some obstetric imaging applications, maternal anatomical landmarks may be used to differentiate the individual sweeps. For example, the presence of two kidneys on a sweep may be an indication of a lateral sweep 44. Or in another example, if the maternal bladder was seen at the beginning or end of a sweep that sweep may be classified as a longitudinal sweep 42, whereas when it is shown in the middle frames of a cine loop that could an indication of a lower lateral sweep 44. FIG. 8 shows a longitudinal sweep 42 passing through an anatomical landmark 58 once, and a lateral sweep 44 passing through the same anatomical landmark 58 twice.

[0058] Automatic analysis of the images could be done by traditional image processing / pattern recognition approaches or by a controller which uses a neural network trained on the ultrasound images of different sweeps where their appropriate classification is known beforehand. When using machine-learning approaches, the identification of the anatomical landmarks 58 on the ultrasound frames could be done by the same or a separate neural network. At inference time, the anatomical landmarks 58 and their location on the sweep cine loops as well as their relative locations (e.g., when a landmark is captured in more than one sweep) is extracted from the ultrasound data. The network would use this information to output different sweep classifications as well as a confidence metric which indicates network’s confidence in the outputs. Approaches such as Monte Carlo dropout could be used to calculate the confidence of the network. In this approach, the network trained with drop-out nodes may be run multiple times on the same input during inference to generate slightly different outputs (since dropout drops the outputs from a specified number of nodes at random). These different outputs can be used to compute the mean and variance in the estimated numerical labels assigned to each class (e.g., -1, 0, 1 for left, middle, and right, respectively). The variance can be used to indicate a level of confidence in the output (i.e., high variance indicates that network output is not consistent and, therefore, confidence is low, while low variance indicates consistent output and high confidence).

[0059] The classification may further include comparing the relative position of the individual sweeps with the same classification heading. For example, the method would compare the positions of all the lateral sweeps 44 relative to each other. This information could be stored in a memory 36 to be used for downstream applications such as anatomy mapping and detection, or fetal biometry estimates. This information could also be used to provide the feedback to the ultrasound probe 12 operator. For example, this data could be compared to the predetermined grid 40 to determine deviations from the proper blind imaging protocol.

[0060] Step 112 of method 100 includes communicating feedback at least partially based on the classification of the ultrasound sweeps. This could, for example, take the form of visual feedback where the sweeps that were performed correctly are shown on the predetermined pattern 40, displayed on user interface 20, as green and the sweeps that need to be repeated are shown as red. Additionally or alternatively, the feedback could take the form of auditory feedback that outputs an auditory message to the ultrasound probe 12 operator. Additionally or alternatively, the feedback could take the form of haptic feedback. In which case, the ultrasound probe 12 or tablet 18 could vibrate when the user deviates from the predetermined pattern 40. In some examples, the feedback could include the ultrasound probe 12 automatically starting and stopping the acquisition of the ultrasound images. This would completely remove the need for a novice operator to press any buttons or have any control over this feature.

[0061] FIG. 9 is a method 200 of calibrating an IMU 14 according to some aspects of the present disclosure. At step 202 of method 200, the position of the ultrasound probe 12 is input at in a fixed orientation. For example, the ultrasound probe 12 may be placed at the center of the predetermined pattern 40. At step 204 the acceleration and angular velocity of the ultrasound probe 12 are detected and stored in memory 36 for a predetermined time period, while at rest. The calibration time may be determined based on the number of samples needed to reliably determine the IMU 14 bias, and will thus also vary depending on the sampling rate of the probe estimation sensors. For example, a probe estimation sensor with a higher sampling rate would require a shorter calibration time than a probe estimation sensor with a lower sampling rate. At step 206 of method 200, a bias is calculated. One way that this can be achieved is through a comparison of the known orientation and the orientation detected by the probe estimation sensor. For example, the position and orientation can be calculated by double integrating the acceleration and integrating an angular velocity respectively. When calculating the accelerometer bias, the gravitational acceleration component is removed from thecalculation. At step 208 of method 200, the calculated bias is corrected for. In one embodiment, this can be achieved by redefining the origins of the IMU Preference frame and global reference frame by subtracting the calculated bias from the original origin.

[0062] FIG. 10 is an illustration of an ultrasound reference sweep 60 according to some aspects of the present disclosure. For example, the reference sweep 60 shown in FIG. 10 is taken approximately from the pubis to a sternum of a patient 16. Other anatomical landmarks may be used to define the reference sweep 60 depending on the particular ultrasound imaging protocol or medical application. The reference sweeps 60 may be used to define boundaries 62 for the ultrasound imaging. These boundaries 62 can be used as a reference both for differentiating the blind sweeping pattern into individual sweeps, and for scaling position estimates. For example, when processing the blind sweep data, the start and stop of each sweep may be determined by proximity to the boundaries 62. Sensor position estimates are vulnerable to error from bias, drift, and double integration. Rather than relying on absolute estimates for position and orientation, the blind sweep position and orientation estimates may be normalized to the distance to the boundaries 62.

[0063] Additionally or alternatively, the reference sweeps 60 may be used to classify the sweeps immediately after their acquisition without waiting for all the sweeps of one classification heading to be collected. The relative location of the sweeps to the reference sweep 60 (right / left / up / down) as well as the distance from the reference sweep 60 (especially when there is more than one sweep at each side of the reference sweep) may be utilized to perform the classification. The classification may be updated in case more data is collected which indicates the inaccurate prior classification. For example, a sweep could be called "R" because it is shifted towards the right side of the longitudinal reference sweep 60. After acquiring the next couple of sweeps it would be obvious that the label must change to "M” since one sweep is acquired at each side of the initial sweep. This logic or any similar process for sweep re-labelling could be implemented on a signal processing controller. Any deterministic approach or Al-based method may be used to determine the inaccurate classification and relabelling of the sweeps.

[0064] All definitions, as defined and used herein, should be understood to control over dictionary definitions, definitions in documents incorporated by reference, and / or ordinary meanings of the defined terms.

[0065] The indefinite articles "a” and "an,” as used herein in the specification and in the claims, unless clearly indicated to the contrary, should be understood to mean "at least one.”

[0066] The phrase "and / or,” as used herein in the specification and in the claims, should be understood to mean "either or both” of the elements so conjoined, i.e., elements that are conjunctively present in some cases and disjunctively present in other cases. Multiple elements listed with "and / or” should be construed in the same fashion, i.e., "one or more” of the elements so conjoined. Other elements can optionally be present other than the elements specifically identified by the "and / or” clause, whether related or unrelated to those elements specifically identified.

[0067] As used herein in the specification and in the claims, "or” should be understood to have the same meaning as "and / or” as defined above. For example, when separating items in a list, "or” or "and / or” shall be interpreted as being inclusive, i.e., the inclusion of at least one, but also including more than one, of a number or list of elements, and, optionally, additional unlisted items. Only terms clearly indicated to the contrary, such as "only one of” or "exactly one of,” or, when used in the claims, "consisting of,” will refer to the inclusion of exactly one element of a number or list of elements. In general, the term "or” as used herein shall only be interpreted as indicating exclusive alternatives (i.e. "one or the other but not both”) when preceded by terms of exclusivity, such as "either,” "one of,” "only one of,” or "exactly one of.”

[0068] As used herein in the specification and in the claims, the phrase "at least one,” in reference to a list of one or more elements, should be understood to mean at least one element selected from any one or more of the elements in the list of elements, but not necessarily including at least one of each and every element specifically listed within the list of elements and not excluding any combinations of elements in the list of elements. This definition also allows that elements can optionally be present other than the elements specifically identified within the list of elements to which the phrase "at least one” refers, whether related or unrelated to those elements specifically identified.

[0069] It should also be understood that, unless clearly indicated to the contrary, in any methods claimed herein that include more than one step or act, the order of the steps or acts of the method is not necessarily limited to the order in which the steps or acts of the method are recited.

[0070] In the claims, as well as in the specification above, all transitional phrases such as "comprising,” "including,” "carrying,” "having,” "containing,” "involving,” "holding,” "composed of,” and the like are to be understood to be open-ended, i.e., to mean including but not limited to. Only the transitional phrases "consisting of” and "consisting essentially of” shall be closed or semi-closed transitional phrases, respectively.

[0071] The above-described examples of the described subject matter can be implemented in any of numerous ways. For example, some aspects can be implemented using hardware, software, or a combination thereof. When any aspect is implemented at least in part in software, the software code can be executed on any suitable processor or collection of processors, whether provided in a single device or computer or distributed among multiple devices / computers.

[0072] The present disclosure can be implemented as a system, a method, and / or a computer program product at any possible technical detail level of integration. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present disclosure.

[0073] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0074] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0075] Computer readable program instructions for carrying out operations of the present disclosure can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, statesetting data, configuration data for integrated circuitry, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++, or the like, and procedural programming languages, such as the "C” programming language or similar programming languages. The computer readable program instructions can execute entirely on the user’s computer, partly on the user's computer, as a stand-alone software package, partly on the user’s computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some examples, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

[0076] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to examples of the disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchartillustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0077] The computer readable program instructions can be provided to a processor of a, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function / act specified in the flowchart and / or block diagram or blocks.

[0078] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0079] The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various examples of the present disclosure. In this regard, each block in the flowchart or block diagrams can represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the Figures. For example, two blocks shown in succession can, in fact, be executed substantially concurrently, or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustration, and combinations of blocks in the block diagrams and / or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.

[0080] Other implementations are within the scope of the following claims and other claims to which the applicant can be entitled.

[0081] While various examples have been described and illustrated herein, those of ordinary skill in the art will readily envision a variety of other means and / or structures for performing the function and / or obtaining the results and / or one or more of the advantages described herein, and each of such variations and / or modifications is deemed to be within the scope of the examples described herein. More generally, those skilled in the art will readily appreciate that all parameters, dimensions, materials, and configurations described herein are meant to be exemplary and that the actual parameters, dimensions, materials, and / or configurations will depend upon the specific application or applications for which the teachings is / are used. Those skilled in the art will recognize or be able to ascertain using no more than routine experimentation, many equivalents to the specific examples described herein. It is, therefore, to be understood that the foregoing examples are presented by way of example only and that, within the scope of the appended claims and equivalents thereto, examples can be practiced otherwise than as specifically described and claimed. Examples of the present disclosure are directed to each individual feature, system, article, material, kit, and / or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and / or methods, if such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent, is included within the scope of the present disclosure.

Claims

ClaimsWhat is claimed is:1 . An ultrasound system (10), configured for image acquisition of an anatomical area of interest of a patient (16) via a blind sweep protocol comprising: an ultrasound probe (12) configured to acquire ultrasound images as the ultrasound probe (12) is moved through a plurality of sweeps in a predetermined pattern (40), along the anatomical area of interest, in accordance with the blind sweep protocol a signal processor (38) configured to differentiate between sweeps and to classify the sweeps, with reference to the predetermined pattern (40)); and a user interface (20) for communicating the blind sweep protocol and an outcome of the sweep classification to a user.

2. The ultrasound system (10) of claim 1, wherein the predetermined pattern (40) comprises a grid and wherein the signal processor (38) is configured to classify at least two of the sweeps from the plurality of sweeps by heading as a lateral sweep (44) or a longitudinal sweep (42) relative to the anatomical area of interest and, for any two sweeps having the same heading, to determine positioning of the two sweeps relative to each other.

3. The ultrasound system (10) of claim 1, further comprising an inertial measurement unit (14) integrated with, or releasably attached to, the ultrasound probe (12), the inertial measurement unit (14) comprising one or more probe estimation sensors configured to detect an acceleration and an angular velocity of the ultrasound probe (12) when moved through the plurality of sweeps in a predetermined pattern (40), along the anatomical area of interest, in accordance with the blind sweep protocol, the probe estimation sensors comprising an accelerometer (22) and a gyroscope (24).

4. The ultrasound system (10) of claim 3, wherein the probe estimation sensors) further comprise a magnetometer (26).

5. The ultrasound system (10) of claim 3, wherein the signal processor (38) is configured to calibrate the inertial measurement unit (14) by: inputting (202) a position of the ultrasound probe in a fixed orientation; detecting (204) the acceleration and the angular velocity using the inertial measurement unit (14) for a predetermined time period; calculating (206) a bias for the inertial measurement unit (14); and correcting (208) for the calculated bias.

6. The ultrasound system (10) of claim 1, wherein the signal processor (38) is configured to calibrate the inertial measurement unit (14) by determining an imaging boundary (62) based on a reference sweep (60), the reference sweep (60) being between a pubis and a sternum.

7. The ultrasound system (10) of claim 6, wherein the signal processor (38) is further configured to classify the sweeps at least partially based on a relative location of the sweeps to the reference sweep (60).

8. The ultrasound system (10) of claim 1, wherein the signal processor (38) is further configured to classify the sweeps at least partially based on machine learning classification.

9. The ultrasound system (10) of claim 1, further comprising an external image sensor (32) configured to detect movement data of the ultrasound probe (12), and wherein the signal processor (38) is further configured to differentiate between sweeps and to classify the sweeps with reference to the predetermined pattern (40) using detected ultrasound probe movement data the image sensor (32).

10. The ultrasound system (10) of claim 9, wherein the image sensor (32) is a camera configured to generate a video feed of the ultrasound probe (12) and the signal processor (38) is further configured to determine a deviation of the ultrasound probe (12) from the predetermined pattern (40) using the video feed.11 . The ultrasound system (10) of claim 1 , wherein the signal processor (38) is further configured to detect coupling and uncoupling of the ultrasound probe (12), using the acquired ultrasound images, and to differentiate and / or classify the sweeps at least partially based on the detected coupling and uncoupling.

12. The ultrasound system (10) of claim 1 , wherein the signal processor (38) is further configured to classify the sweeps based at least partially on an anatomical landmark (58) within the ultrasound images.

13. The ultrasound system of claim 1 further comprising a magnetic field generator (28) and a coil (30), wherein the coil is coupled to the ultrasound probe (12), and wherein the signal processor (38) is configured to determine a position, velocity, and / or angular orientation of the ultrasound probe (12) using a voltage generated in the coil (30) and differentiate between sweeps and classify the sweeps based at least partially on the determined position, velocity, and / or angular orientation.

14. A method (100) for providing feedback of image acquisition of an anatomical area of interest of a patient in a blind sweep protocol, comprising: receiving (102) ultrasound images acquired by an ultrasound probe (12) as the ultrasound probe (12) is moved through a plurality of sweeps in a predetermined pattern (40), along the anatomical area of interest, in accordance with the blind sweep protocol; differentiating (108) between sweeps and classifying (110) the sweeps with reference to the predetermined pattern (40); communicating (112) an outcome of the sweep classification to a user.

15. The method (100) of claim 1 , further comprising detecting (104) an acceleration and an angular velocity of the ultrasound probe (12) by an inertial measurement unit (14) integrated with, or releasably attached to, the ultrasound probe (12), wherein the step of differentiating (108) between sweeps and classifying (110) the sweeps is based at least in part on detecting the acceleration and the angular velocity of the ultrasound probe.

Images