Ultrasound imaging with ultrasound probe guidance in blind sweep protocol

Abstract

A system includes a processor configured for communication with an ultrasound probe, and to receive a plurality of ultrasound image frames from the ultrasound probe during a sweep of a blind sweep protocol on a patient body, as well as receive probe motion information for the sweep that is representative of a position, a velocity, and / or orientation of the ultrasound probe while it obtains the ultrasound image frames. The processor then performs a comparison between the probe motion information and pre-determined, desired probe motion for the blind sweep protocol, and generates and outputs user guidance based on the comparison, in real time while the ultrasound probe obtains the ultrasound image frames during the sweep.

Description

ULTRASOUND IMAGING WITH ULTRASOUND PROBE GUIDANCE IN BLIND SWEEP PROTOCOLTECHNICAL FIELD

[0001] The subject matter described herein relates to devices, systems, and methods for using ultrasound probe position, velocity, and / or orientation to guide a user to perform high- quality, blind abdominal sweeps for ultrasound imaging.BACKGROUND

[0002] Ultrasound imaging is often used for diagnostic purposes in an office or hospital setting, but may also be used in resource-constrained care settings (e.g., homes, accident sites, ambulances, mobile health facilities, etc.) by emergency personnel, home health nurses, midwives, etc., who may lack ultrasound expertise. To facilitate ultrasound image acquisition by untrained or minimally trained users, a “blind sweep” protocol is often employed, in which the user follows pre-determined probe paths (e.g., sweeping out a pattern on the patient’s abdomen) during imaging. However, deviations in transducer velocity, angle, heading, and position relative to the intended path will result in decreased image quality and failure to image anatomical structures of interest. This is sometimes referred to as a “non-adherent” sweep. Non-adherent sweeps are more likely to produce lower quality or incomplete ultrasound data, which in turn will negatively impact the ability of a local operator, remote clinician, or Al algorithm to segment, characterize, or classify anatomical features or anomalies, and thus to arrive at accurate diagnoses and treatment plans.

[0003] The information included in this Background section of the specification, including any references cited herein and any description or discussion thereof, is included for technical reference purposes only and is not to be regarded as subject matter by which the scope of the disclosure is to be bound.SUMMARY

[0004] Disclosed is an ultrasound sweep guidance system that measures the position, orientation, and / or velocity of the ultrasound probe during a blind imaging sweep, and provides clear instructions to the user about how and where to sweep the probe, as well as guidance to the user about the position, speed, or orientation of the probe during and after the sweep. A benefit of the ultrasound sweep guidance system is to improve imaging quality using user guidance derived from an inertial measurement unit (IMU) or other motion tracking methods. Ultrasound imaging with blind sweep protocol (sweeps at particular locations along body without trying to find specific anatomy) is improved by simultaneously obtaining probe motion information (position, velocity, and / or orientation), determining if probe motion is the desired probe motion (e.g., compliant with the desired paths and orientations of the blind sweep protocol), and providing an output based on comparison between actual probe motion and desired probe motion. The output may be or may include user guidance to correct the probe motion, and may be provided during the sweep itself, so that the user can obtain good quality ultrasound images that result from the probe having the correct position, velocity, and / or orientation (according to the blind sweep protocol). The comparison can also be used to determine if the sweep as a whole is compliant or non- compliant.

[0005] A system of one or more computers can be configured to perform particular operations or actions by virtue of having software, firmware, hardware, or a combination of them installed on the system that in operation causes or cause the system to perform the actions. One or more computer programs can be configured to perform particular operations or actions by virtue of including instructions that, when executed by data processing apparatus, cause the apparatus to perform the actions. One general aspect includes a system that includes a processor configured for communication with an ultrasound probe, where the processor is configured to: receive a plurality of ultrasound image frames obtained by the ultrasound probe during a sweep of a blind sweep protocol on a patient body; receive probe motion information for the sweep, where the probe motion information is representative of at least one of a position, a velocity, or an orientation of the ultrasound probe while the ultrasound probe obtains the plurality of ultrasound image frames; perform a comparison between the probe motion information and a desired probe motion information, where the desired probe motion information is pre-determined based on the blind sweep protocol; generate a first user guidance based on the comparison; and provide an output representativeof the first user guidance. The output is provided in real time while the ultrasound probe obtains the plurality of ultrasound image frames during the sweep. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods.

[0006] Implementations may include one or more of the following features. In some aspects, the first user guidance is representative of a change to at least one of the position, the velocity, or the orientation of the ultrasound probe. In some aspects, the processor is configured to: determine, based on the comparison, whether the sweep is compliant or non- compliant to the blind sweep protocol; generate a second user guidance representative of whether the sweep is compliant or non-compliant; and provide an output representative of the second user guidance. In some aspects, when the sweep is compliant, the second user guidance may include an instruction to perform the next sweep or an indication that the blind sweep protocol is complete, where, when the sweep is non-compliant, the second user guidance may include an instruction to repeat the sweep. In some aspects, the probe motion information is obtained by at least one of an accelerometer, gyroscope, or magnetometer disposed within the ultrasound probe. In some aspects, the probe motion information is obtained by a magnetic coil within the ultrasound probe, where a voltage generated within the magnetic coil is responsive to at least one of a position, a velocity, or an orientation of the magnetic coil relative to an external magnetic field. In some aspects, the probe motion information is obtained by at least one camera, where images generated by the at least one camera are responsive to at least one of a position, a velocity, or an orientation of the ultrasound probe. In some aspects, the processor is configured to: output, to a display in communication with the processor, instructions to a user to perform the sweep. The instructions to the user may include a visual indication of at least one of a position, a direction, a length, or a duration of the sweep on the patient body. In some aspects, the output representative of the first user guidance may include a visual feedback displayed on a display in communication with the processor. In some aspects, the visual feedback may include at least one of a bubble level, a correction arrow, or a visual representation of the ultrasound probe. In some aspects, the output representative of the first user guidance may include an auditory feedback provided through a speaker in communication with the processor. In some aspects, the auditory feedback may include at least one of a volume, a pitch, a frequency, or a duty cycle indicative of how close the probe motion information is to the desired probe motion information. In some aspects, the output representative of the firstuser guidance may include a haptic feedback provided by a haptic motor in communication with the processor. In some aspects, the haptic motor is disposed within the ultrasound probe. In some aspects, the haptic feedback may include at least one of a frequency, an amplitude, or a duty cycle indicative of how close the probe motion information is to the desired probe motion information. In some aspects, the system may include the ultrasound probe. Implementations of the described techniques may include hardware, a method or process, or computer software on a computer-accessible medium.

[0007] One general aspect includes a method, which includes receiving, with a processor in communication with an ultrasound probe, a plurality of ultrasound image frames obtained by the ultrasound probe during a sweep of a blind sweep protocol on a patient body. The method also includes receiving probe motion information for the sweep, where the probe motion information is representative of at least one of a position, a velocity, or an orientation of the ultrasound probe while the ultrasound probe obtains the plurality of ultrasound image frames. The method also includes performing a comparison between the probe motion information and a desired probe motion information, where the desired probe motion information is pre-determined based on the blind sweep protocol. The method also includes generating a user guidance based on the comparison, and providing an output representative of the user guidance. The output is provided in real time while the ultrasound probe obtains the plurality of ultrasound image frames during the sweep. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods.

[0008] Implementations may include one or more of the following features. In some aspects, the probe motion information is obtained by at least one of an accelerometer, gyroscope, or magnetometer disposed within the ultrasound probe. In some aspects, the user guidance may include at least one of: a visual feedback displayed on a display in communication with the processor, where the visual feedback may include at least one of a bubble level, a correction arrow, or a visual representation of the ultrasound probe; an auditory feedback provided through a speaker in communication with the processor, where the auditory feedback may include at least one of a volume, a pitch, a frequency, or a duty cycle indicative of how close the probe motion information is to the desired probe motion information; or a haptic feedback provided by a haptic motor in communication with the processor, where the haptic feedback may include at least one of a frequency, an amplitude, or a duty cycle indicative of how close the probe motion information is to the desired probemotion information. Implementations of the described techniques may include hardware, a method or process, or computer software on a computer-accessible medium.

[0009] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. A more extensive presentation of features, details, utilities, and advantages of the ultrasound sweep guidance system, as defined in the claims, is provided in the following written description of various aspects of the disclosure and illustrated in the accompanying drawings.BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Illustrative aspects of the present disclosure will be described with reference to the accompanying drawings, of which:

[0011] Figure 1 is a schematic, diagrammatic representation of an ultrasound imaging system, according to aspects of the present disclosure.

[0012] Figure 2 is a schematic diagram of a processor circuit, according to aspects of the present disclosure.

[0013] Figure 3 is a schematic, diagrammatic representation of a patient, according to aspects of the present disclosure.

[0014] Figure 4 is a schematic, diagrammatic representation, in block diagram form, of an example ultrasound sweep guidance system, according to aspects of the present disclosure.

[0015] Figure 5 is a schematic, diagrammatic, perspective view of an imaging probe (e.g., an ultrasound imaging probe) in contact with a body surface of a patient, according to aspects of the present disclosure.

[0016] Figure 6 is a schematic, diagrammatic image of a patient undergoing a blind sweep imaging procedure, according to aspects of the present disclosure.

[0017] Figure 7 is a schematic, diagrammatic image of a patient undergoing a blind sweep imaging procedure, according to aspects of the present disclosure.

[0018] Figure 8 is a schematic, diagrammatic representation, in flow diagram form, of an example ultrasound sweep guidance method, according to aspects of the present disclosure.

[0019] Figure 9 is an instruction screen display of an example ultrasound sweep guidance system, according to aspects of the present disclosure.

[0020] Figure 10 is a sweep progress screen display of an example ultrasound sweep guidance system, according to aspects of the present disclosure.

[0021] Figure 11 is a bubble level screen display of an example ultrasound sweep guidance system, according to aspects of the present disclosure.

[0022] Figure 12 is a bubble level screen display of an example ultrasound sweep guidance system, according to aspects of the present disclosure.

[0023] Figure 13 is a bubble level screen display of an example ultrasound sweep guidance system, according to aspects of the present disclosure.

[0024] Figure 14 is a sweep progress screen display of an example ultrasound sweep guidance system, according to aspects of the present disclosure.

[0025] Figure 15 is an orientation feedback demonstration of an example ultrasound sweep guidance system, according to aspects of the present disclosure.

[0026] Figure 16 is a post-sweep screen display of an example ultrasound sweep guidance system, according to aspects of the present disclosure.

[0027] Figure 17 is a schematic, diagrammatic representation, in hybrid flow diagram / block diagram form, of an example ultrasound sweep guidance method, according to aspects of the present disclosure.

[0028] Figure 18 is a schematic, diagrammatic representation, in flow diagram form, of an example ultrasound sweep guidance method, according to aspects of the present disclosure.

[0029] Figure 19 is a schematic, diagrammatic representation, in flow diagram form, of an example ultrasound sweep guidance method, according to aspects of the present disclosure.

[0030] Figure 20A is a schematic, diagrammatic overview, in block diagram form, of a training mode for an untrained neural network, according to aspects of the present disclosure.

[0031] Figure 20B is a schematic, diagrammatic overview, in block diagram form, of an inference mode or clinical usage mode for the trained neural network, according to aspects of the present disclosure.

[0032] Figure 20C is a schematic, diagrammatic overview, in block diagram form, of an inference mode or clinical usage mode for the trained neural network, according to aspects of the present disclosure.

[0033] Figure 21 is a schematic, diagrammatic representation, in hybrid flow diagram / block diagram form, of an example ultrasound sweep guidance method, according to aspects of the present disclosure.DETAILED DESCRIPTION

[0034] In accordance with at least one aspect of the present disclosure, an ultrasound sweep guidance system ultrasound sweep guidance system is provided which measures the attitude and velocity of the ultrasound probe during a sweep and provides clear guidance to the user about problems with the position, speed, or orientation of the probe. The ultrasound sweep guidance system presents a novel approach to imaging quality control by, in some aspects, using an inertial measurement unit (IMU). Other aspects, instead or in addition, leverage alternative motion tracking methods. An IMU measures probe motion, and these data are leveraged to provide both real-time feedback to the user to maintain protocol adherence, and to determine post-sweep acceptance / rej ection. The quality control methodology presented herein is intended for use with standalone, mobile, and cart-based ultrasound imaging systems, but can also be applied to other ultrasound systems and medical imaging and / or device guidance generally.

[0035] To simplify the imaging workflow for minimally trained users, the system instructs the user to perform imaging sweeps along a pre-determined grid on the mother’s abdomen (blind sweeps). This protocol is in contrast to the free hand sweeps that would be performed by a trained sonographer, during which the sonographer moves the probe as needed to localize relevant anatomical structures of interest. To optimize image quality from this blind sweep protocol by novice users, the motion sensor that is already included in some ultrasound probes may be leveraged to provide both real-time and post-sweep quality control.

[0036] During anticipated use, a user may not follow the sweep protocol in the manner that the product was intended. This type of probe motion is non-adherent to the protocol, which has implications for image quality and other downstream processes. For example, rotations of the transducer may result in a loss of coupling, shifts in the image volume such that anatomical structures of interest are missed, or the partial overlap of consecutive image frames. Inconsistent sweep acceleration results in inconsistent spacing of image frames. If the transducer is moved too quickly, an insufficient number of frames may be acquired in the sweep. If the transducer is moved in inconsistent directions (e.g. going back and forth instead of just moving forward), wrong anatomy mapping and visualization may result. Finally, positional deviations may result in anatomies of interest being missed in the acquisition. The performance of downstream medical analysis or Al-based analysis, which rely on these data for many applications (such as anatomy mapping e.g., to estimate fetal presentation, placentalocation estimation and calculation of the maximum vertical pocket of amniotic fluid) may decrease as a result.

[0037] The challenges listed above can be addressed via an inertial measurement unit (IMU) incorporated into or externally mounted to the ultrasound probe, as is already the case with some existing ultrasound probes. The IMU measures 3-dimensional acceleration and angular velocity of the ultrasound probe, representing 6 degrees of freedom (6-DOF). Velocity and position are the first and second integrals of acceleration, respectively. Orientation is the first integral of angular velocity. Therefore, the IMU provides the ability to identify motion and rotation of the ultrasound probe, which a processor can then use to identify whether or not the motion / rotation is compliant with the blind sweep protocol. For example, the processor may determine whether the position, speed, and angle of the probe fall within specified error limits, and may provide guidance to the user to correct the position, speed, and / or angle.

[0038] IMU quality control may for example be implemented in a signal processing controller which can be a stand-alone system or a stand-alone app, or be integrated in the ultrasound system itself. Thresholds for angular velocity, rotation, acceleration, velocity, position, and other relevant IMU-derived metrics may for example be established by analyzing the retrospective data from a large dataset with known sweep qualities. In the realtime quality control implementation, deviation feedback above the threshold will be relayed to the user by auditory, visual, and / or haptic means. The sweep quality might be quantified based on the severity of deviation from the intended protocol or the impact of the deviation from the intended protocol on the quality of the acquired images. This might be communicated to the user through visual output or through changes in the properties (e.g., intensity or frequency, etc.) of the haptic or auditory output. In the post-sweep quality control implementation, completed full sweeps may be categorized as adherent or nonadherent, and the user will be prompted to repeat non-adherent sweeps.

[0039] In some aspects, the categorization of adherent / non-adherent sweeps can also factor in anatomical context for where the protocol deviations happened. For example, if the deviation from protocol happened in a region where no anatomies were detected, then that sweep needn’t necessarily be termed “non-adherent”. However, if the coupling between the probe and the patient’s skin is compromised in the deviant sections of the sweep, then lack of anatomic detections therein may not be an appropriate filter, and the sweep may still be identified as non-adherent. An overall benefit of this approach is that the user will not be required to re-do a sweep if the detected protocol deviation in the original sweep was notfound to result in a corresponding negative impact to the quantifiability or diagnostic value of that sweep.

[0040] Optionally, IMU-derived statistics may be relayed to the user post-sweep as well, thus providing feedback that may help the user improve sweep compliance in the future.

[0041] It is noted that sensor biases of the IMU may be accounted for when processing its measurements. This may be done by various methods known in the art, including by requiring a calibration period prior to use to determine the biases, or by automatically accounting for biases in a Kalman-filter-based or neural-network-based pose / motion prediction algorithm(s). Optionally, the data streams may also be denoised using various hardware and / or software methods known in the art (e.g., low-pass, high-pass or band-pass filters or filtering methods).

[0042] Next, the data streams from the gyroscope and accelerometer are fused. First, if the gyroscope and accelerometer have different sampling frequencies, the sampling can be unified. This may be accomplished by any synchronization algorithm known in the art (e.g., resampling one data stream to match the other, or by only using samples taken at the same timepoint, etc.). Once sampling frequencies are unified, there are a number of approaches to perform sensor fusion, with the common trait but not limited to the assumption that the rotational information provided by the gyroscope is used to transform the accelerometer measurements from the IMU’s frame of reference to a global frame of reference. In the IMU frame of reference, the first two axes may for example be defined along the short and long axes of the probe, with the third axis being orthogonal to the first two axes. The origin of this frame of reference may for example be the physical location of the IMU within the probe. In the global reference frame, the vertical axis may be aligned with the gravitational vector, and the second two axes may be defined by the orientation of the probe at the acquisition start time (nominally aligned with the patient’s longitudinal and transverse axes). The origin in the global reference frame may be the position of the probe at the acquisition start time.

[0043] In the simplest sensor fusion approach, the angular velocity measured by the gyroscope can be integrated to obtain the current IMU orientation. Alternatively, angular velocity and acceleration data may be fused with an extended Kalman filter (EKF), Kalman filter, complementary filter, Madgwick filter, neural network, or other similar sensor fusion methodologies. The output in this case would be more accurate estimates of orientation than from simply integrating the angular velocity. Note that changes in position derived from the IMU measurements may be relative to the initial location of the ultrasound probe. Initialposition may be determined by anatomical landmarks on the patient (e.g., the patient’s bladder, pubic symphysis, etc.) or other means.

[0044] Once orientation is obtained, it can then be used to transform IMU acceleration to the global reference frame. In the global reference frame, the gravitational acceleration is subtracted from the vertical axis acceleration. To obtain velocity and position (relative to acquisition start time), acceleration can be single and double integrated, respectively.

[0045] Rotational deviations of rate or pose angle may be detected when angular velocity and / or orientation exceed clinically significant thresholds. Likewise, motion and position deviations can be identified when acceleration, velocity and relative position exceed clinically significant thresholds. The thresholds for these parameters may be determined by processing of retrospective images and IMU data. When collecting simultaneous image and IMU data, the optimal thresholds may for example be the limits at which no clinically significant difference in image quality is observed. In obstetrics, one of the impacts of an angulated probe is on the calculation of the amniotic fluid maximum vertical pocket (AF- MVP). Therefore, quantifying the impact on the AF-MVP may be one approach to determining thresholds. For example, tilting the probe by 10 degrees causes on average a 15% overestimation of AF-MVP, which may be deemed to be clinically significant by an expert reader. When the probe is tilted by 5 degrees, the overestimation reduces to 8% on average, which may fall below the significance level. In another example, probe angle may be related to the percentage of an image frame’s area with decoupling artifacts. The significance threshold for decoupling may be determined by an expert reader and / or by measuring its impact on the performance of downstream deep learning models (e.g., for anatomy mapping).

[0046] In the real-time quality control scenario, when a threshold is surpassed, the user may be warned of the deviation by auditory, visual, or haptic means, or a combination thereof. For example, a pulsed sound or light may be emitted, optionally with the pulse frequency indicative the severity of the deviation. When the transducer motion returns to compliance, the warning may cease.

[0047] In another aspect, an IMU that includes a magnetometer may be used, to provide even more accurate tracking. A magnetometer may yield more accurate rotation measurements around the global vertical axis than the gyroscope. This can be applied to reducing errors when transforming acceleration measurements to the global frame. In addition, if the system uses a sensor fusion methodology, the magnetometer may be used tocompensate for gyroscope drift around the global vertical axis. The workflow may otherwise be the same as, or similar to, that described previously herein.

[0048] In still another aspect, the data streams from the gyroscope and accelerometer may be fused using a neural network. Various methods for doing this have been described in the art, generally using a recurrent neural network (RNN)-based or temporal convolutional network (TCN)-based approach, which allow for the incorporation of temporal information. Incorporating temporal information is crucial, both due to the time-series structure of the IMU data streams, and the tendency for IMU measurements to experience drift over time, which must be accounted for to obtain accurate measurements. Following sensor fusion by a neural network-based method, the quality control method may be implemented as previously described.

[0049] In still another aspect, video feed of one or multiple cameras (e.g., red-green-blue (RGB), red-green-blue-depth (RGBD), etc.) could be used to track the probe motion and its deviation from the intended path. The intended path might be marked using markers which can be picked up by the camera, or the intended path could be determined based on the images themselves. This may be accomplished with a variety of visual odometry / simultaneous localization and mapping (SLAM) approaches - for example, by tracking the centroid of the probe across the imaging field of known size at a known frame rate, or by using optical flow-based approaches. In another example, a neural network could be used to extract the probe's orientation and velocity information from the videos and calculate the deviation from the intended path indicated by the markers. The network may be a convolutional neural network which is trained on many images of ultrasound probes with different known orientations moving in known directions on a marked surface. The network can learn the association between the visible markers in the consecutive video frames and probe orientation and velocity, or can learn based on human annotation of the video frames, indicating whether the probe position, angle, and velocity are compliant or non-compliant.

[0050] In still another aspect, the captured images using the probe (e.g., the ultrasound probe) may be used in conjunction with the video camera feed to detect the deviation from the intended path and orientation without using any surface markers. In tandem with the visual odometry / SLAM methods described in the previous embodiment, features may be extracted from the ultrasound images. For example, the angular deviation from the position normal to the surface may be captured by presence of shadows on one side of the ultrasound images. The shadow width can be an indication of the angular deviation of the probe from the intended orientation. The probe velocity, position, and pose could be estimated from thevideo camera captures. Extensive decoupling analysis may also be carried out from the video camera data.

[0051] In still another aspect, a processing controller (e.g., a neural network) might be trained on the acquired ultrasound images or external camera images and the corresponding IMU data for each successfully accomplished sweep. A variational encoder decoder architecture, such as an RNN, or TCN, could be used to automatically distinguish the deviations from the successful sweeps. The network may be fed raw ultrasound or external camera image frames, optical flow data derived from the images, or from some other preprocessing method. The IMU data may also be provided as a raw input, pre-processed in the manner previously described, or by some other method. The processing controller may be used to classify adherent vs. non-adherent frames / s weeps, or to output continuous estimates of motion parameters that are subsequently used to determine adherence in the thresholdbased manner previously described.

[0052] In still another aspect, other tracking sensors and systems (e.g., magnetic tracking sensors and systems) could be used to track the probe motion and orientation. For example, in the case of electromagnetic tracking, a fixed external magnetic field generator creates a local magnetic field. In conjunction, three orthogonal coils are installed in or attached to the ultrasound probe. Voltage is induced in the coils as they pass through the magnetic field, from which position and orientation with respect to the magnetic field generator may be determined. Once again, the quality control method may then be implemented as previously described, simply using this alternative source of motion and orientation data.

[0053] The present disclosure aids substantially in the capture of high-quality ultrasound images by minimally trained users, by detecting the position, speed, and angle of the ultrasound probe, providing both real-time and post-sweep feedback to the user, and requiring that the user repeat any non-compliant sweeps. Implemented on a processor in communication with an ultrasound probe, the ultrasound sweep guidance system disclosed herein provides practical improvements in the quality of the captured images and the time required to capture them. This improved imaging transforms a process that is heavily reliant on professional experience into one that is accurate and repeatable even for minimally trained personnel, without the normally routine need to train clinicians such as emergency department personnel to perform quality control on the ultrasound images they capture. This unconventional approach improves the functioning of the ultrasound imaging system, by providing reliable, repeatable imaging in hospital, office, vehicle, field, and home settings.

[0054] The ultrasound sweep guidance system may be implemented as a process at least partially viewable on a display, and operated by a control process executing on a processor that accepts user inputs from a keyboard, mouse, or touchscreen interface, and that is in communication with one or more sensor probes. In that regard, the control process performs certain specific operations in response to different inputs or selections made at different times. Certain structures, functions, and operations of the processor, display, sensors, and user input systems are known in the art, while others are recited herein to enable novel features or aspects of the present disclosure with particularity.

[0055] These descriptions are provided for exemplary purposes only, and should not be considered to limit the scope of the ultrasound sweep guidance system. Certain features may be added, removed, or modified without departing from the spirit of the claimed subject matter.

[0056] For the purposes of promoting an understanding of the principles of the present disclosure, reference will now be made to the aspects illustrated in the drawings, and specific language will be used to describe the same. It is nevertheless understood that no limitation to the scope of the disclosure is intended. Any alterations and further modifications to the described devices, systems, and methods, and any further application of the principles of the present disclosure are fully contemplated and included within the present disclosure as would normally occur to one skilled in the art to which the disclosure relates. In particular, it is fully contemplated that the features, components, and / or steps described with respect to one aspect may be combined with the features, components, and / or steps described with respect to other aspects of the present disclosure. For the sake of brevity, however, the numerous iterations of these combinations will not be described separately.

[0057] Figure 1 is a schematic, diagrammatic representation of an ultrasound imaging system 100, according to aspects of the present disclosure. The ultrasound imaging system 100 may for example be used to acquire ultrasound video sweeps, which can then be analyzed by a human clinician or an artificial intelligence to diagnose medical conditions.

[0058] The ultrasound imaging system 100 is used for scanning an area or volume of a subject’s body. A subject may include a patient of an ultrasound imaging procedure, or any other person, or any suitable living or non-living organism or structure. The ultrasound imaging system 100 includes an ultrasound imaging probe 110 in communication with a host 130 over a communication interface or link 120. The probe 110 may include a transducer array 112, a beamformer 114, a processor circuit 116, and a communication interface 118.The host 130 may include a display 132, a processor circuit 134, a communication interface 136, and a memory 138 storing subject information.

[0059] In some aspects, the probe 110 is an external ultrasound imaging device including a housing 111 configured for handheld operation by a user. The transducer array 112 can be configured to obtain ultrasound data while the user grasps the housing 111 of the probe 110 such that the transducer array 112 is positioned adjacent to or in contact with a subject’s skin. The probe 110 is configured to obtain ultrasound data of anatomy within the subject’s body while the probe 110 is positioned outside of the subject’s body for general imaging, such as for abdomen imaging, liver imaging, etc. In some aspects, the probe 110 can be an external ultrasound probe, a transthoracic probe, and / or a curved array probe.

[0060] In other aspects, the probe 110 can be an internal ultrasound imaging device and may comprise a housing 111 configured to be positioned within a lumen of a subject’s body for general imaging, such as for abdomen imaging, liver imaging, etc.. In some aspects, the probe 110 may be a curved array probe. Probe 110 may be of any suitable form for any suitable ultrasound imaging application including both external and internal ultrasound imaging.

[0061] In some aspects, the ultrasound probe 110 may include an inertial measurement unit (IMU) 115, which may in some cases include an accelerometer 140 (e.g., a 1-axis, 2- axis, or 3-axis accelerometer), a gyroscope 150 (e.g., a 1-axis, 2-axis, or 3-axis gyroscope), and / or a magnetometer 160 (e.g., a 1-axis, 2-axis, or 3-axis magnetometer). In some aspects, the probe 110 also includes a haptic motor 170 capable of, for example, vibrating the probe 110 to provide haptic feedback to a user holding the probe 110. In some aspects, the haptic motor 170 may be located outside of the probe 110, such as for example in a handheld host 130 (e.g., a smartphone or tablet computer).

[0062] In some aspects, aspects of the present disclosure can be implemented with medical images of subjects obtained using any suitable medical imaging device and / or modality. Examples of medical images and medical imaging devices include x-ray images (angiographic images, fluoroscopic images, images with or without contrast) obtained by an x-ray imaging device, computed tomography (CT) images obtained by a CT imaging device, positron emission tomography-computed tomography (PET-CT) images obtained by a PET- CT imaging device, magnetic resonance images (MRI) obtained by an MRI device, singlephoton emission computed tomography (SPECT) images obtained by a SPECT imaging device, optical coherence tomography (OCT) images obtained by an OCT imaging device, and intravascular photoacoustic (IVPA) images obtained by an IVPA imaging device. Themedical imaging device can obtain the medical images while positioned outside the subject body, spaced from the subject body, adjacent to the subject body, in contact with the subject body, and / or inside the subject body.

[0063] For an ultrasound imaging device, the transducer array 112 emits ultrasound signals towards an anatomical object 105 of a subject and receives echo signals reflected from the object 105 back to the transducer array 112. The ultrasound transducer array 112 can include any suitable number of acoustic elements, including one or more acoustic elements and / or a plurality of acoustic elements. In some instances, the transducer array 112 includes a single acoustic element. In some instances, the transducer array 112 may include an array of acoustic elements with any number of acoustic elements in any suitable configuration. For example, the transducer array 112 can include between 1 acoustic element and 10000 acoustic elements, including values such as 2 acoustic elements, 4 acoustic elements, 36 acoustic elements, 64 acoustic elements, 128 acoustic elements, 500 acoustic elements, 812 acoustic elements, 1000 acoustic elements, 3000 acoustic elements, 8000 acoustic elements, and / or other values both larger and smaller. In some instances, the transducer array 112 may include an array of acoustic elements with any number of acoustic elements in any suitable configuration, such as a linear array, a planar array, a curved array, a curvilinear array, a circumferential array, an annular array, a phased array, a matrix array, a one-dimensional (ID) array, a 1.x dimensional array (e.g., a 1.5D array), or a two- dimensional (2D) array. The array of acoustic elements (e.g., one or more rows, one or more columns, and / or one or more orientations) can be uniformly or independently controlled and activated. The transducer array 112 can be configured to obtain one-dimensional, two- dimensional, and / or three-dimensional images of a subject’s anatomy. In some aspects, the transducer array 112 may include a piezoelectric micromachined ultrasound transducer (PMUT), capacitive micromachined ultrasonic transducer (CMUT), single crystal, lead zirconate titanate (PZT), PZT composite, other suitable transducer types, and / or combinations thereof.

[0064] The object 105 may include any anatomy or anatomical feature, such as a kidney, liver, and / or any other anatomy of a subject. The present disclosure can be implemented in the context of any number of anatomical locations and tissue types, including without limitation, organs including the liver, kidneys, gall bladder, pancreas, lungs; ducts; intestines; nervous system structures including the brain, dural sac, spinal cord and peripheral nerves; the urinary tract; as well as valves within the blood vessels, blood, abdominal organs, and / or other systems of the body. In some aspects, the object 105 may include malignancies such astumors, cysts, lesions, hemorrhages, or blood pools within any part of human anatomy. The anatomy may be a blood vessel, as an artery or a vein of a subject’s vascular system, including cardiac vasculature, peripheral vasculature, neural vasculature, renal vasculature, and / or any other suitable lumen inside the body. In addition to natural structures, the present disclosure can be implemented in the context of man-made structures such as, but without limitation, heart valves, stents, shunts, filters, implants and other devices.

[0065] The beamformer 114 is coupled to the transducer array 112. The beamformer 114 controls the transducer array 112, for example, for transmission of the ultrasound signals and reception of the ultrasound echo signals. In some aspects, the beamformer 114 may apply a time-delay to signals sent to individual acoustic transducers within an array in the transducer 112 such that an acoustic signal is steered in any suitable direction propagating away from the probe 110. The beamformer 114 may further provide image signals to the processor circuit 116 based on the response of the received ultrasound echo signals. The beamformer 114 may include multiple stages of beamforming. The beamforming can reduce the number of signal lines for coupling to the processor circuit 116. In some aspects, the transducer array 112 in combination with the beamformer 114 may be referred to as an ultrasound imaging component.

[0066] The processor 116 is coupled to the beamformer 114. The processor 116 may also be described as a processor circuit, which can include other components in communication with the processor 116, such as a memory, beamformer 114, communication interface 118, and / or other suitable components. The processor 116 may include a central processing unit (CPU), a graphical processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a controller, a field programmable gate array (FPGA) device, another hardware device, a firmware device, or any combination thereof configured to perform the operations described herein. The processor 116 may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. The processor 116 is configured to process the beamformed image signals. For example, the processor 116 may perform filtering and / or quadrature demodulation to condition the image signals. The processor 116 and / or 134 can be configured to control the array 112 to obtain ultrasound data associated with the object 105.

[0067] The communication interface 118 is coupled to the processor 116. The communication interface 118 may include one or more transmitters, one or more receivers, one or more transceivers, and / or circuitry for transmitting and / or receiving communicationsignals. The communication interface 118 can include hardware components and / or software components implementing a particular communication protocol suitable for transporting signals over the communication link 120 to the host 130. The communication interface 118 can be referred to as a communication device or a communication interface module.

[0068] The communication link 120 may be any suitable communication link. For example, the communication link 120 may be a wired link, such as a universal serial bus (USB) link or an Ethernet link. Alternatively, the communication link 120 may be a wireless link, such as an ultra-wideband (UWB) link, an Institute of Electrical and Electronics Engineers (IEEE) 802.11 WiFi link, or a Bluetooth link.

[0069] At the host 130, the communication interface 136 may receive the image signals. The communication interface 136 may be substantially similar to the communication interface 118. The host 130 may be any suitable computing and display device, such as a workstation, a personal computer (PC), a laptop, a tablet, or a mobile phone.

[0070] The processor 134 is coupled to the communication interface 136. The processor 134 may also be described as a processor circuit, which can include other components in communication with the processor 134, such as the memory 138, the communication interface 136, an optional speaker 138, and / or other suitable components. The processor 134 may be implemented as a combination of software components and hardware components. The processor 134 may include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a controller, an FPGA device, another hardware device, a firmware device, or any combination thereof configured to perform the operations described herein. The processor 134 may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. The processor 134 can be configured to generate image data from the image signals received from the probe 110. The processor 134 can apply advanced signal processing and / or image processing techniques to the image signals. In some aspects, the processor 134 can form a three-dimensional (3D) volume image from the image data. In some aspects, the processor 134 can perform real-time processing on the image data to provide a streaming video of ultrasound images of the object 105. In some aspects, the host 130 includes a beamformer. For example, the processor 134 can be part of and / or otherwise in communication with such a beamformer. The beamformer in the in the host 130 can be a system beamformer or a main beamformer (providing one or moresubsequent stages of beamforming), while the beamformer 114 is a probe beamformer or micro-beamformer (providing one or more initial stages of beamforming).

[0071] The memory 138 is coupled to the processor 134. The memory 138 may be any suitable storage device, such as a cache memory (e.g., a cache memory of the processor 134), random access memory (RAM), magnetoresistive RAM (MRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), flash memory, solid state memory device, hard disk drives, solid state drives, other forms of volatile and non-volatile memory, or a combination of different types of memory.

[0072] The memory 138 can be configured to store subject information, measurements, data, or files relating to a subject’s medical history, history of procedures performed, anatomical or biological features, characteristics, or medical conditions associated with a subject, computer readable instructions, such as code, software, or other application, as well as any other suitable information or data. The memory 138 may be located within the host 130. Subject information may include measurements, data, files, other forms of medical history, such as but not limited to ultrasound images, ultrasound videos, and / or any imaging information relating to the subject’s anatomy. The subject information may include parameters related to an imaging procedure such as an anatomical scan window, a probe orientation, and / or the subject position during an imaging procedure. The memory 138 can also be configured to store information related to the training and implementation of machine learning algorithms (e.g., neural networks) and / or information related to implementing image recognition algorithms for detecting / segmenting anatomy, image quantification algorithms, and / or image acquisition guidance algorithms, including those described herein.

[0073] The display 132 is coupled to the processor circuit 134. The display 132 may be a monitor or any suitable display. The display 132 is configured to display the ultrasound images, image videos, and / or any imaging information of the object 105.

[0074] The ultrasound imaging system 100 may be used to assist a sonographer in performing an ultrasound scan. The scan may be performed in a point-of-care setting. In some instances, the host 130 is a console or movable cart. In some instances, the host 130 may be a mobile device, such as a tablet, a mobile phone, or portable computer. During an imaging procedure, the ultrasound system can acquire an ultrasound image of a particular region of interest within a subject’s anatomy. The ultrasound imaging system 100 may then analyze the ultrasound image to identify various parameters associated with the acquisition of the image such as the scan window, the probe orientation, the subject position, and / or otherparameters. The ultrasound imaging system 100 may then store the image and these associated parameters in the memory 138. At a subsequent imaging procedure, the ultrasound imaging system 100 may retrieve the previously acquired ultrasound image and associated parameters for display to a user which may be used to guide the user of the ultrasound imaging system 100 to use the same or similar parameters in the subsequent imaging procedure, as will be described in more detail hereafter.

[0075] In some aspects, the processor 134 may utilize deep learning-based prediction networks to identify parameters of an ultrasound image, including an anatomical scan window, probe orientation, subject position, and / or other parameters. In some aspects, the processor 134 may receive metrics or perform various calculations relating to the region of interest imaged or the subject’s physiological state during an imaging procedure. These metrics and / or calculations may also be displayed to the sonographer or other user via the display 132.

[0076] In some aspects, the host 130 may also include a speaker 180. The speaker 180 may for example be used to provide warning tones, beeps, or other auditory feedback to the user.

[0077] Before continuing, it should be noted that the examples described above are provided for purposes of illustration, and are not intended to be limiting. Other devices and / or device configurations may be utilized to carry out the operations described herein.

[0078] Figure 2 is a schematic diagram of a processor circuit 250, according to aspects of the present disclosure. The processor circuit 250 may be implemented in the ultrasound imaging system 100, or other devices or workstations (e.g., third-party workstations, network routers, etc.), or on a cloud processor or other remote processing unit, as necessary to implement the method. As shown, the processor circuit 250 may include a processor 260, a memory 264, and a communication module 268. These elements may be in direct or indirect communication with each other, for example via one or more buses.

[0079] The processor 260 may include a central processing unit (CPU), a digital signal processor (DSP), a controller, or any combination of general-purpose computing devices, reduced instruction set computing (RISC) devices, application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other related logic devices, including mechanical and quantum computers. The processor 260 may also comprise another hardware device, a firmware device, or any combination thereof configured to perform the operations described herein. The processor 260 may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality ofmicroprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0080] The memory 264 may include a cache memory (e.g., a cache memory of the processor 260), random access memory (RAM), magnetoresistive RAM (MRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), flash memory, solid state memory device, hard disk drives, other forms of volatile and nonvolatile memory, or a combination of different types of memory. In an aspect, the memory 264 includes a non-transitory computer-readable medium. The memory 264 may store instructions 266. The instructions 266 may include instructions that, when executed by the processor 260, cause the processor 260 to perform the operations described herein. Instructions 266 may also be referred to as code. The terms “instructions” and “code” should be interpreted broadly to include any type of computer-readable statement(s). For example, the terms “instructions” and “code” may refer to one or more programs, routines, subroutines, functions, procedures, etc. “Instructions” and “code” may include a single computer-readable statement or many computer-readable statements.

[0081] The communication module 268 can include any electronic circuitry and / or logic circuitry to facilitate direct or indirect communication of data between the processor circuit 250, and other processors or devices. In that regard, the communication module 268 can be an input / output (I / O) device. In some instances, the communication module 268 facilitates direct or indirect communication between various elements of the processor circuit 250 and / or the ultrasound imaging system 100. The communication module 268 may communicate within the processor circuit 250 through numerous methods or protocols. Serial communication protocols may include but are not limited to United States Serial Protocol Interface (US SPI), Inter-Integrated Circuit (I2C), Recommended Standard 232 (RS- 232), RS-485, Controller Area Network (CAN), Ethernet, Aeronautical Radio, Incorporated 429 (ARINC 429), MODBUS, Military Standard 1553 (MIL-STD-1553), or any other suitable method or protocol. Parallel protocols include but are not limited to Industry Standard Architecture (ISA), Advanced Technology Attachment (ATA), Small Computer System Interface (SCSI), Peripheral Component Interconnect (PCI), Institute of Electrical and Electronics Engineers 488 (IEEE-488), IEEE-1284, and other suitable protocols. Where appropriate, serial and parallel communications may be bridged by a Universal Asynchronous Receiver Transmitter (UART), Universal Synchronous Receiver Transmitter (USART), or other appropriate subsystem.

[0082] External communication (including but not limited to software updates, firmware updates, model sharing between the processor and central server, or readings from the ultrasound imaging system 100) may be accomplished using any suitable wireless or wired communication technology, such as a cable interface such as a universal serial bus (USB), micro USB, Lightning, or FireWire interface, Bluetooth, Wi-Fi, ZigBee, Li-Fi, or cellular data connections such as 2G / GSM (global system for mobiles) , 3G / UMTS (universal mobile telecommunications system), 4G, long term evolution (LTE), WiMax, or 5G. For example, a Bluetooth Low Energy (BLE) radio can be used to establish connectivity with a cloud service, for transmission of data, and for receipt of software patches. The controller may be configured to communicate with a remote server, or a local device such as a laptop, tablet, or handheld device, or may include a display capable of showing status variables and other information. Information may also be transferred on physical media such as a USB flash drive or memory stick.

[0083] Figure 3 is a schematic, diagrammatic representation of a patient 300, according to aspects of the present disclosure. Visible on the abdomen 310 of the patient 300 is a desired sweep pattern 320 intended to capture images of desired features of the patient’s anatomy. The sweep pattern 320 consists of multiple vertical sweep lines 330 and multiple horizontal sweep lines 340. Each sweep line 330, 340 represents a desired path for one imaging sweep of the abdomen 310. In the example shown in Figure 3, the sweep pattern consists of three vertical sweep lines 330, all in an upward direction with respect to the patient, and three horizontal sweep lines 340, all in a left-right direction with respect to the patient. However, it is understood that a sweep pattern 320 may include more or fewer sweep lines 330, including vertical sweep lines 330, horizontal sweep lines 330, or combinations thereof, in any combination of upward, downward, left, or right directions. Furthermore, a sweep pattern 320 may cover other portions of the patient’s body, including but not limited to the head, neck, spine, limbs, etc.

[0084] These sweep patterns represent desired probe motion information, including desired positions, a desired velocity or velocities, and / or a desired orientation of the ultrasound probe while the ultrasound probe is obtaining a plurality of ultrasound image frames during the sweep. It is noted that the desired sweep patterns or blind sweep protocols stored in a memory of the processor may include only vertical sweeps, only horizontal sweeps, may include a grid (e.g., 3x3, 5x5, etc.) of vertical and horizontal sweeps, and may also include associated parameters such as desired probe motion (e.g., positions, velocities, and / or orientations) stored in the memory. The protocols and their associated parameters canfor example be based on standards established by authorities in the field (physician organizations, sonographer organizations, etc.), published in scholarly joumals / textbooks, etc.

[0085] Figure 4 is a schematic, diagrammatic representation, in block diagram form, of an example ultrasound sweep guidance system 400, according to aspects of the present disclosure. In the example shown in Figure 4, the ultrasound sweep guidance system, 400 acquires ultrasound images 410 from one or more image sweeps, wherein each sweep may for example contain 30 images per second over a 10-second recording period, for a total of 300 image frames. The ultrasound sweep guidance system 400 may include an IMU 420 to record the motion of the ultrasound imaging probe. Either or both of the ultrasound images 410 and the motion data from the IMU 420 may be used by a signal processing unit 430 to analyze the motion of the probe and determine whether it is compliant with the desired blind sweep protocol.

[0086] In some cases, the signal processing unit may include a sensor fusion algorithm such as a Kalman filter or extended Kalman filter. A Kalman filter uses linear quadratic estimation (LQE) to turn a series of noisy measurements observed over time into a series of accurate estimates for the true values of the variables. For a variable x at time k, the Kalman filter includes a state transition model Fk, a control-input model Bk, and a control vector Uk, which collectively describe the evolving system model at time k, such that: xk= FkXk-i + Bkuk+ wk(EQN. 1) where Wk is the (Gaussian) system noise around variable x at time k. The system may include hidden variables, and may reduce a large number of input variables into a smaller number of output variables such that, for example, 6-DOF IMU readings (x, y, and z linear accelerations, and rotation rates around the x, y, and z axes) and 3-DOF magnetometer readings (x, y, and z magnetic field strengths) may yield a 6-element output vector (x, y, and z positions and angles). In other exemplary cases, the Kalman filter may produce more output variables than there are inputs. In an example, the same 6-DOF IMU readings may yield a 15-element output of x, y, and z accelerations, x, y, and z velocities, x, y, and z positions, x, y, and z rotation rates, and x, y, and z angles.

[0087] An extended Kalman filter (EKF) is a nonlinear version of a standard (linear) Kalman filter, which linearizes about an estimate of the current mean and covariance, and is frequently used in well-defined transition models such as navigation based on multiple unrelated input variables. In the EKF, the state transition and observation models don't need to be linear functions of the state but may instead be differentiable functions. In an example, the EKF may use multivariate Taylor series expansions to linearize a model about a workingpoint at any given time k. If the system model is not well known or is inaccurate, then Monte Carlo methods may be employed within the EKF to generate variance around the input values, such that a mean prediction for each output variable can be extracted.

[0088] In some aspects, the signal processing unit 430 may employ other types of sensor fusion algorithms, including but not limited to a complementary filter, Madgwick filter, or a learning network.

[0089] In an example, the signal processing unit 430 may implement or include any suitable type of learning network. For example, in some aspects, the signal processing unit 430 could include a neural network, such as a recurrent neural network (RNN) or temporal convolutional network (TCN) that includes temporal information. In addition, the neural network may additionally or alternatively be an encoder-decoder type network, or may utilize a backbone architecture based on other types of neural networks, such as an object detection network (if images are also used), classification network, etc. The TCN may for example include a set of N convolutional layers, where N may be any positive integer. Fully connected layers can be omitted when the TCN is a backbone. The TCN may also include max pooling layers and / or activation layers. Each convolutional layer may include a set of filters configured to extract features from an input (e.g., from an axis of the IMU). The value N and the size of the filters may vary depending on the aspects. In some instances, the convolutional layers may utilize any non-linear activation function, such as for example a leaky rectified linear unit (ReLU) activation function. In some instances, normalization / regularization methods may be used, such as batch normalization. The max pooling layers gradually adjust the dimension of the input variables (e.g., 6-DOF or 9-DOF IMU outputs) to a dimension of the desired result (e.g., a 15-element probe motion vector). Fully connected layers may be referred to as perception or perceptive layers. In some aspects, fully connected layers may be found in the neural network.

[0090] These descriptions are included for exemplary purposes; a person of ordinary skill in the art will appreciate that other types of learning models, with features similar to or dissimilar to those described above, may be used instead or in addition, without departing from the spirit of the present disclosure.

[0091] The systems and methods disclosed herein are broadly applicable to different types of features, and can for example reduce the effects of noise, bias, drift, vibration, shock, or other error sources in one or more sensors or non-sensor inputs. In some aspects, inputs to the signal processing unit 430 may also include classified ultrasound image features, orientations derived from ultrasound image features, camera images of the probe, cameraimages from the probe, or positioning system inputs external to the ultrasound probe, including but not limited to global positioning system (GPS) inputs.

[0092] A user interface 440 provides instructions to the user regarding the blind-sweep protocol, as well as feedback on whether (and to what degree) the motions of the probe are compliant with the protocol. Feedback may include real-time feedback 450, including live deviation alerts provided during the sweep, and post-sweep feedback 460, indicating whether that sweep was compliant or non-compliant, and in some cases may also include statistics or other information regarding which aspects of the sweep were non-compliant. In some cases, a non-compliant sweep may be rejected, and the user may be asked to repeat that sweep. Additional feedback may also be provided if repeated sweeps are also non-compliant.Feedback is provided in such a manner that, as the user performs the sweeps, the probe motion information gets relatively closer to or becomes the desired probe motion information.

[0093] Once all necessary compliant sweeps have been accepted, the signal processing unit 430 may produce a clinical report 470. Depending on the implementation, the clinical report may include 2D or 3D images, anatomical measurements or statistics, and other clinically relevant information. In some cases, the clinical report 470 may also include performance statistics on the user who performed the sweeps. In some cases, the content of the clinical report 470 may vary depending on geographic region, on the type of anatomy being imaged, or on other factors.

[0094] The clinical report 470 may, in some cases, be available to the user of the ultrasound sweep guidance system 400, who may for example be a physician. In other cases, the user may be a paramedic, midwife, or other person not specifically trained in the interpretation of ultrasound images. In such cases, the clinical report 470 may be stored locally, may be transmitted over a network for review by a remote physician, or may be passed to a downstream Al tool for further analysis.

[0095] It is noted that block diagrams are provided herein for exemplary purposes; a person of ordinary skill in the art will recognize myriad variations that nonetheless fall within the scope of the present disclosure. For example, block diagrams may show a particular arrangement of components, modules, services, steps, processes, or layers, resulting in a particular data flow. It is understood that some aspects of the systems disclosed herein may include additional components, that some components shown may be absent from some aspects, and that the arrangement of components may be different than shown, resulting in different data flows while still performing the methods described herein.

[0096] Figure 5 is a schematic, diagrammatic, perspective view of an imaging probe (e.g., an ultrasound imaging probe) 110 in contact with a body surface 540 of a patient, according to aspects of the present disclosure. The probe 110 (e.g., an IMU located inside the probe) includes its own local coordinate system, with a Y-axis 530 aligned with the long axis of the probe, and orthogonal X-axis 510 and Z-axis 520. The probe 110 may have a different rotational velocity (e.g., rotational velocity 580) around each of the X, Y, and Z axes. The rotational velocities can be integrated to yield a rotation angle around each axis, which represents the orientation of the probe 110 in space.

[0097] Also visible is a global coordinate system that includes a Yg axis 570 and orthogonal Xg axis 550 and Zg axis 560. The global coordinate system may for example align with the caudal-cranial axis, the left-right-axis, and the dorsal-ventral axis of the patient, although this will not always be the case. Furthermore, although an effort may be made to align the probe with the global coordinate system, it will not generally be the case that the probe coordinate system and the global coordinate system are identical. Thus, while ultrasound images are inherently captured relative to the probe’s coordinate system, it may be desirable to rotate the position, velocity, pose angles, and rotation rates of the probe 110 into the global coordinate system to determine whether the probe motion is compliant with the blind-sweep protocol.

[0098] It is noted that positive rotation around each axis, as shown in Figure 5, is determined by the right-hand rule.

[0099] Figure 6 is a schematic, diagrammatic image of a patient 300 undergoing a blind sweep imaging procedure, according to aspects of the present disclosure. Visible are the imaging probe 110 on the patient’s abdomen 310. In the example shown in Figure 6, an angle 610 between the probe Y-axis 530 and the global X axis 550 is approximately 90 degrees, at a time 620 of approximately 10 seconds. A motion graph screen display 600 shows a rotation angle 630 of the probe’s X-axis (e.g., a rotation angle around the X-axis), a rotation angle 640 of the probe’s Y-axis, and a rotation angle 650 of the probe’s Z-axis. At the time 620 of approximately 10 seconds, the motion graph screen display 600 shows all three of these angles are close to zero, indicating that the probe 110 is in the desired orientation.

[0100] Figure 7 is a schematic, diagrammatic image of a patient 300 undergoing a blind sweep imaging procedure, according to aspects of the present disclosure. Visible are the imaging probe 110 on the patient’s abdomen 310, along with the global Y-axis 570 and global X-axis 550. In the example shown in Figure 7, an angle 710 between the probe Y-axis530 and the global X axis 550 is substantially less than 90 degrees, at a time 720 of approximately 10 seconds. A motion graph screen display 600 shows the rotation angle 630 of the probe’s X-axis, the rotation angle 640 of the probe’s Y-axis, and the rotation angle 650 of the probe’s Z-axis. At the time 720 of approximately 11.5 seconds, the motion graph screen display 600 shows that the X angle 630 and Y angle 640 are close to zero, indicating that the probe 110 is in the desired X and Y orientation. However, the Z angle 650 is at approximately -50 degrees, which is non-compliant with the blind sweep protocol.Therefore, a warning icon 730 has appeared on the motion graph screen display 600 (possible along with an audible and / or haptic warning) to indicate to the user that the probe is in a non- compliant orientation.

[0101] Figure 8 is a schematic, diagrammatic representation, in flow diagram form, of an example ultrasound sweep guidance method 800, according to aspects of the present disclosure. It is understood that the steps of method 800 may be performed in a different order than shown in Figure 8, additional steps can be provided before, during, and after the steps, and / or some of the steps described can be replaced or eliminated in other embodiments. One or more of steps of the method 800 can be carried by one or more devices and / or systems described herein, such as components of system 100, processor 134, system 400, signal processing unit 430, and / or processor circuit 250.

[0102] In step 805, the method 800 includes determining the blind sweep protocol. This may for example involve determining the number and position of sweeps based on the patient’s size, and / or initializing / calibrating the IMU such that the global coordinate system aligns with gravity. Execution then proceeds to step 810.

[0103] In step 810, the method 810 includes determining the next sweep. This may for example involve selecting the next non-completed sweep from the blind sweep protocol. Execution then proceeds to step 820.

[0104] In step 820, the method 800 includes displaying, to the user, instructions and prompts to begin the selected sweep. Execution then proceeds to step 825.

[0105] In step 825, the method 800 includes receiving a user input to start the sweep. This may for example involve the user pressing a Start button, Enter key, or other user interface control. Execution then proceeds to step 830.

[0106] In step 830, the method 800 includes generating a screen display to show progress of the current sweep. Execution then proceeds to step 832 and, in parallel, to step 835.

[0107] In step 832, the method 800 includes controlling (e.g., with a processor) the ultrasound probe to acquire ultrasound images of the patient’s anatomy during the sweep. Execution then proceeds to step 834.

[0108] In step 834, the method 800 optionally includes displaying the images obtained during the sweep. Depending on the implementation, displaying the images may violate laws, ethical guidelines, patient privacy, etc., although in other cases, displaying the images may aid in real-time diagnosis and treatment of the patient. Execution then proceeds to step 850.

[0109] In step 835, the method 800 includes controlling (e.g., with a processor) the IMU to obtain IMU data (e.g., X, Y, and X accelerations and rotation rates) during the sweep. Execution then proceeds to step 840.

[0110] In step 840, the method 800 includes determining the probe motion (e.g., X, Y, and Z positions, velocities, and orientations) during the sweep, based on the IMU data. Execution then proceeds to step 845.

[0111] In step 845, the method 800 includes displaying feedback to the user (and / or providing auditory or haptic feedback to the user) on the progress of the sweep, based on the probe motion. This may involve, for example, indicating to the user whether the position, velocity, and orientation of the probe is compliant or non-compliant with the sweep protocol. Execution then proceeds to step 850.

[0112] In step 850, the method 800 includes ceasing to obtain ultrasound images and IMU data. Execution then proceeds to step 855.

[0113] In step 855, the method 800 includes determining whether the sweep is adherend or non-adherent to the blind sweep protocol.

[0114] In some cases, this determination may be based on the IMU data obtained during the sweep. For example, the processor circuit can determine a threshold to determine if the sweep as a whole is adherent or non-adherent. The threshold can be a percentage of IMU data points in a given sweep, a quantity of IMU data points in the given sweep, etc. For example, the threshold can 10% of IMU data points that are non-adherent and / or 90% of IMU data points that are adherent. If the IMU data points for a given sweep satisfy the threshold (e.g., less than 10% non-adherent and / or greater than 90% adherent), then the processor circuit determines the given sweep to be adherent. If the IMU data points for a given sweep do not satisfy the threshold (e.g., greater than 10% non-adherent and / or less than 90% adherent), then the processor circuit determines the given sweep to be non-adherent.

[0115] Instead or in addition, this determination may be based on the ultrasound images themselves. In these cases, it may be desirable to register the ultrasound images to the global coordinate frame based on the probe orientation at the time each image was captured. Thus, the processor circuit registers the ultrasound images with corresponding IMU data points. It is noted that the ultrasound probe and IMU data can have different acquisition rates. Processor circuit can thus select IMU measurements taken only at certain times corresponding to image frame times, or resample IMU measurements accordingly, to account for different acquisition rates, which prepares the ultrasound images and / or IMU data for registration. The processor circuit then rotates each image into the global coordinate frame for registration, thereby establishing one-to-one correspondence between individual ultrasound image and individual IMU data point. Individual ultrasound images can be determined to be adherent or non-adherent based registration between the IMU data and ultrasound images. The processor circuit then identifies individual ultrasound images whose associated rotational, motion, and / or position deviation (based on registered IMU data) is non-adherent (e.g., registered angular velocity and / or orientation do not satisfy rotational threshold, registered velocity and / or position do not satisfy motion and / or position threshold) and vice versa. The processor circuit hen determines the sweep as a whole to be adherent or non-adherent based on individual ultrasound images that are adherent or non-adherent.

[0116] The threshold can be a percentage of ultrasound images in a given sweep, a quantity of ultrasound images in the given sweep, etc. For example, the threshold can be such that 10% of ultrasound images are non-adherent and 90% of ultrasound images are adherent. If the ultrasound images for a given sweep satisfy the threshold (e.g., less than or equal to 10% non-adherent and greater than 90% adherent), then the processor circuit determines the given sweep to be adherent. If the ultrasound images for a given sweep do not satisfy the threshold (e.g., greater than 10% non-adherent and less than 90% adherent), then the processor circuit determines the given sweep to be non-adherent.

[0117] Execution then proceeds to step 860.

[0118] In step 860, the method 800 includes generating a screen display that includes a post-sweep summary and / or the determination of whether the sweep is compliant or non- compliant. Execution then proceeds to step 865.

[0119] In step 865, the method 800 includes determining whether the sweep needs to be repeated, based on whether or not it is compliant with the protocol. If yes, execution returns to step 825. If no, execution proceeds to step 870.

[0120] In step 870, the method 800 includes determining whether the just-completed sweep is the final sweep in the protocol. If yes, execution proceeds to step 875, and in parallel to step 880. If no, execution returns to step 810.

[0121] In step 875, the method 800 includes generating and outputting user feedback for the blind sweep protocol. This may for example include statistics, diagrams, or other information that may be useful to the user in generating compliant sweeps. The method is now complete.

[0122] In step 880, the method 800 includes performing downstream image processing of the ultrasound images. Such image processing may for example include denoising, sharpness adjustments, brightness adjustments, contrast adjustments, object recognition, segmentation, classification, Execution then proceeds to step 885.

[0123] In step 885, the method 800 includes generating and outputting a clinical report. Depending on the implementation, the clinical report may be directed to the user, or may be directed to a remote clinician. The method is now complete.

[0124] It is noted that flow diagrams are provided herein for exemplary purposes; a person of ordinary skill in the art will recognize myriad variations that nonetheless fall within the scope of the present disclosure. For example, the logic of flow diagrams may be shown as sequential. However, similar logic could be parallel, massively parallel, object oriented, real-time, event-driven, cellular automaton, or otherwise, while accomplishing the same or similar functions. In order to perform the methods described herein, a processor may divide each of the steps described herein into a plurality of machine instructions, and may execute these instructions at the rate of several hundred, several thousand, several million, or several billion per second, in a single processor or across a plurality of processors. Such rapid execution may be necessary in order to execute the method in real time or near-real time as described herein. For example, over the course of a 10-second sweep, the ultrasound sweep guidance system 400 may need to poll the IMU and determine the position, velocity, and orientation of the probe 110 multiple times per second (e.g., at a rate of 100 Hz).

[0125] Figure 9 is an instruction screen display 900 of an example ultrasound sweep guidance system 400, according to aspects of the present disclosure. In the example shown in Figure 9, the instruction screen display 900 includes text instructions 910, telling the user to make vertical sweeps with the probe held vertically. The instruction screen display 900 also includes a diagram 920 showing the desired orientation of the ultrasound probe. The instruction screen display 900 also includes a start button, which can be activated by the user in order to start the sweep.

[0126] Figure 10 is a sweep progress screen display 1000 of an example ultrasound sweep guidance system 400, according to aspects of the present disclosure. In the example shown in Figure 10, the sweep progress screen display 1000 includes a stylized patient diagram 1010 over which a sweep progress indicator 1020 is overlaid. The sweep progress indicator 1020 includes an illuminated current sweep line 1030, indicating the position and direction of the current sweep, and an optional bubble level 1040. The bubble level 1040 indicates both the position and the alignment of the ultrasound probe, as shown below. The sweep progress screen display 1000 also includes a control box 1050 that provides instructions to the user, including a countdown timer 1060, a cancel button 1070 (e.g., to abort the current sweep), and an exit button 1080 (e.g., to abort the entire blind sweep protocol).

[0127] Figure 11 is a bubble level screen display 1100 of an example ultrasound sweep guidance system 400, according to aspects of the present disclosure. In the example shown in Figure 11, the bubble level screen display 100 includes an orientation graphic 1150 (e.g., a representation of the probe 110 and its current orientation), as well as a virtual bubble 1110 within an accuracy window 1120. In an example, the bubble 1110 moves relative to the accuracy window based on the orientation of the probe 110. If the probe orientation is accurate, then an angle “anglel” between the probe Y-axis 530 and the global X-axis 550 or global Z-axis is within a threshold “thresholdl” of 90 degrees.

[0128] The bubble level screen display 1100 also includes an inner misalignment zone 1130 and an outer misalignment zone 1140, indicating various degrees of misalignment of the bubble 1120 and thus of the probe 110. Because the bubble 1120 is partly overlapping with the inner misalignment zone, the screen display 1100 also includes a warning icon 1160, indicating that the alignment of the probe (and thus, the quality of the images captured by it) is marginal. Instead of or in addition to the warning icon 1160, the ultrasound sweep guidance system 400 may generate auditory feedback (e.g., a beeping or buzzing) or haptic feedback (e.g., a vibration of the probe 110).

[0129] Figure 12 is a bubble level screen display 1100 of an example ultrasound sweep guidance system 400, according to aspects of the present disclosure. Visible are the probe 110, orientation graphic 1150, virtual bubble 1110, accuracy window 1120, inner misalignment zone 1130, and outer misalignment zone 1140. In the example shown in Figure 12, the bubble 1110 is outside of both the alignment window 1120 and the inner misalignment zone 1130, indicating poor alignment. This occurs because an angle “angle2” between the probe Y-axis 530 and the global X-axis 550 or global Z-axis is outside athreshold “threshold 1” of 90 degrees, but inside a threshold “threshold2” of 90 degrees. Because the bubble 1120 is within the outer misalignment zone 1140, the screen display 1100 also includes a warning icon 1260 that is different from the warning icon 1160, indicating that the alignment of the probe (and thus, the quality of the images captured by it) is poor. Instead of or in addition to the warning icon 1260, the ultrasound sweep guidance system 400 may generate auditory feedback (e.g., a beeping or buzzing) or haptic feedback (e.g., vibration) that is louder, higher amplitude, faster, etc. than in the previous example, such that the feedback is indicative of a degree of compliance or non-compliance with the blind-sweep protocol.

[0130] Figure 13 is a bubble level screen display 1100 of an example ultrasound sweep guidance system 400, according to aspects of the present disclosure. Visible are the probe 110, orientation graphic 1150, virtual bubble 1110, accuracy window 1120, inner misalignment zone 1130, and outer misalignment zone 1140. In the example shown in Figure 13, the bubble 1110 is outside of the alignment window 1120, inner misalignment zone 1130, and outer misalignment zone 1140, indicating non-compliant alignment. This occurs because an angle “angle3” between the probe Y-axis 530 and the global X-axis 550 or global Z-axis is outside a threshold “threshold2” of 90 degrees. Because the bubble 1120 is outside the outer misalignment zone 1140, the screen display 1100 also includes a warning icon 1360 that is different from the warning icons 1160 and 1260, indicating that the alignment of the probe (and thus, the quality of the images captured by it) is not clinically useful, and the sweep will need to be repeated. Instead of or in addition to the warning icon 1360, the ultrasound sweep guidance system 400 may generate auditory feedback (e.g., a beeping or buzzing) or haptic feedback (e.g., vibration) that is louder, faster, of different frequency, or of higher duty cycle than in the previous two examples. Thus, the auditory or haptic feedback is indicative of a degree of compliance or non-compliance with the sweep protocol.

[0131] Figure 13 also includes a correction arrow 1370, indicating to the user which direction to rotate the probe 110 in order to return the bubble 1370 to the accuracy window 1120.

[0132] Figure 14 is a sweep progress screen display 1000 of an example ultrasound sweep guidance system 400, according to aspects of the present disclosure. Visible are the stylized patient diagram 1010, sweep progress indicator 1020, illuminated current sweep line 1030, bubble level 1040, control box 1050, countdown timer 1060, cancel button 1070, and exit button 1080. The bubble level 1040 indicates both the position and the alignment of the ultrasound probe, and is at a different position than in the example of Figure 10 as anadditional 2 seconds have elapsed during the sweep, as shown by the countdown timer 1060. In the example shown in Figure 14, the bubble 1110 of the bubble level 1040 is off center, indicating that the ultrasound probe is tipped too far forward and too far to the right.

[0133] Figure 15 is an orientation feedback demonstration 1500 of an example ultrasound sweep guidance system 400, according to aspects of the present disclosure. Visible are three instances of a bubble level 1040. In the first instance, the bubble 1110a is located fully within the bubble level 1040, indicating good alignment. As a result, the ultrasound sweep guidance system 400 generates a baseline audible tone or haptic vibration 1510. In the second instance, the bubble 1110b is misaligned on the left-right axis. As a result, the ultrasound sweep guidance system 400 generates a left-right misalignment tone or haptic vibration 1520, that is different in pitch, frequency, duty cycle, or other detectable features, from the audible tone or haptic vibration 1510. In the third instance, the bubble 1110c is misaligned along the forward-backward axis. As a result, the ultrasound sweep guidance system 400 generates a front-back misalignment tone or haptic vibration 1530, that is different in pitch, frequency, duty cycle, or other detectable features, from the audible tones or haptic vibrations 1510 and 1520. It is understood that the ultrasound sweep guidance system 400 may employ any combination of visual, auditory, haptic, or other feedback to communicate to the user whether the probe is aligned or misaligned according to the blindsweep protocol, and to provide guidance to the user as to how to adjust the probe orientation to return to return to a compliant pose.

[0134] Figure 16 is a post-sweep screen display 1600 of an example ultrasound sweep guidance system 400, according to aspects of the present disclosure. Visible are the stylized patient diagram 1010, sweep progress indicator 1020, illuminated current sweep line 1030, bubble level 1040, and bubble 1110. In the example shown in Figure 16, the bubble 1110 is mostly outside of the bubble level 1040, indicating non-compliant alignment. Also visible is a feedback box 1650, which includes feedback text 1655, feedback graphics 1060, and, in the case of a non-compliant sweep, a Redo button similar to the Start button shown in Figure 9.

[0135] Other factors that may generate a non-compliant sweep include moving the prove too fast or too slow, or wandering off of the desired sweep path. Thus, the bubble level 1040 may be configured to move along the current sweep path 1030 during the sweep, so that the position of the bubble 1110 may, instead or in addition, be indicative of these errors.

[0136] Figure 17 is a schematic, diagrammatic representation, in hybrid flow diagram / block diagram form, of an example ultrasound sweep guidance method 1700, according to aspects of the present disclosure.

[0137] The method 1700 includes receiving, into a sensor fusion sub-process 1702, data streams from the IMU 115, which may include an accelerometer 140, gyroscope 150, and optional magnetometer 160, as described above. The sensor fusion sub-process 1702 may for example be a Kalman filter, machine learning network, or similar.

[0138] In step 1710, the method 1700 includes accounting for any bias in the gyroscope 150. Execution then proceeds to step 1715.

[0139] In step 1715, the method 1700 optionally includes denoising the data stream from the gyroscope 150. This may be done for example with a highpass, lowpass, or bandpass filter, by averaging multiple data points, or by other means. Execution then proceeds to step 1740.

[0140] In step 1720, the method 1700 includes accounting for any bias in the accelerometer 140. Execution then proceeds to step 1725.

[0141] In step 1725, the method 1700 optionally includes denoising the data stream from the accelerometer. This may be done for example with a highpass, lowpass, or bandpass filter, by averaging multiple data points, or by other means. Execution then proceeds to step 1740.

[0142] In step 1730, the method 1700 includes accounting for any bias in the magnetometer 160. Execution then proceeds to step 832 and, in parallel, to step 1735.

[0143] In step 1735, the method 1700 includes denoising the data stream from the magnetometer. This may be done for example with a highpass, lowpass, or bandpass filter, by averaging multiple data points, or by other means. Execution then proceeds to step 1740.

[0144] In step 1740, the method 1700 includes fusing the data streams from the gyroscope 150, accelerometer 140, and magnetometer 160. Execution then proceeds to step 1750.

[0145] In step 1750, the method 1700 includes determining the orientation, angular velocity, velocity, and / or position of the ultrasound probe, using the fused data streams. Execution then proceeds to step 1780 and, in parallel, to step 1790.

[0146] It is noted that, although steps 1710 through 1750 are shown herein as individual steps, and may be performed as such, it should be appreciated that some aspects of the sensor fusion sub-process 1702 may be capable of performing some or all of these functions in a reduced number of steps. For example, a Kalman filter or neural network can, in some instances, fuse noisy / biased sensor data in a way that yields the desired output vector.

[0147] In step 1760, the method 1700 includes determining a rotational threshold for the ultrasound probe (e.g., a maximum deviation from the desired orientation or rotation speed).In some cases, there may be a separate threshold for each of the X, Y, and Z axes. In other cases, there may be a single threshold that applies to all three axes. Execution then proceeds to step 1780.

[0148] In step 1770, the method 1700 includes determining a motion and / or position threshold for the ultrasound probe (e.g., a maximum deviation from the desired position or velocity at any given time). In some cases, there may be a separate threshold for each of the X, Y, and Z axes. In other cases, there may be a single threshold that applies to all three axes. Execution then proceeds to step 1790.

[0149] In step 1780, the method 1700 includes detecting any rotational deviation (e.g., when the angular velocity and / or orientation do not satisfy the rotational threshold). Execution then proceeds to step 1795.

[0150] In step 1790, the method 1700 includes detecting any motion and / or position deviation (e.g., when the velocity and / or position do not satisfy the motion and / or position threshold). Execution then proceeds to step 1795.

[0151] In step 1795, the method 1700 includes generating an output (whether visual, auditory, haptic, or combinations thereof) that is indicative of any deviation(s), or lack thereof. The method is now complete.

[0152] Figure 18 is a schematic, diagrammatic representation, in flow diagram form, of an example ultrasound sweep guidance method 1800, according to aspects of the present disclosure.

[0153] In step 1810, the method 1800 includes placing markers on the patient’s body identifying the intended path(s) for the ultrasound probe during the blind sweep protocol. Execution then proceeds to step 1830.

[0154] In step 1830, the method 1800 includes receiving video frames from one or more cameras 1820 that are positioned to see both the ultrasound probe and the markers (or controlling, e.g., with a processor, the camera(s) to capture the video frames), and, from the video frames, determining the angle, rotation rate, speed, and / or position deviation from the blind-sweep protocol. This may for example involve image decomposition as described above, or may involve analysis of depth, shadows, Doppler shifts, etc. Execution then proceeds to step 1840.

[0155] In step 1840, the method 1800 includes generating an output indicative of the deviation(s), or lack thereof, as described above. The method is now complete.

[0156] Figure 19 is a schematic, diagrammatic representation, in flow diagram form, of an example ultrasound sweep guidance method 1900, according to aspects of the present disclosure.

[0157] In step 1915, the method 1900 includes receiving image frames from the ultrasound probe 1910, and performing image processing (e.g., object identification, segmentation, classification, etc.) to identify image features that can be used to determine the location and / or orientation of the probe. This image processing may for example be performed using a convolutional neural network (CNN) or other machine learning model. Execution then proceeds to step 810.

[0158] In step 1930, the method 1900 includes receiving video frames from one or more cameras 1920 that are positioned to see both the ultrasound probe and the patient, and, from the video frames, determining the angle, rotation rate, velocity, and / or position of the ultrasound probe. This may for example involve image decomposition as described above, or may involve analysis of depth, shadows, Doppler shifts, etc., or may use a convolutional neural network (CNN) or other machine learning model. Execution then proceeds to step 1940.

[0159] In step 1940, the method 1900 includes, based on the positions, velocities, angles, and rates determined in steps 1915 and 1930, determining any rotational, speed, or positional deviation from the blind-sweep protocol. This may for example involve a sensor fusion subprocess, as described above. Execution then proceeds to step 1950.

[0160] In step 1950, the method 1900 includes generating an output indicative of the deviation(s), or lack thereof, as described above. The method is now complete.

[0161] In some aspects, the camera(s) 1920 are not used, and the image frames generated by the ultrasound probe during the sweep comprise the information relating to positions, velocities, or orientations of the ultrasound probe. In such aspects, the processor is further configured to: associate respective image frames of the ultrasound image stream to the positions, velocities, or orientations of the ultrasound probe at the respective time points at which the respective images were captured, and to recognize features of the respective image frames to determine positions, velocities, or orientations of the respective image frames within a global coordinate system.

[0162] Figure 20A is a schematic, diagrammatic overview, in block diagram form, of a training mode 2000 for an untrained neural network 2010a, according to aspects of the present disclosure. In the example shown in Figure 20A, a set of training data 805a includes noisy sensor input vectors, accurate output (e.g., pose) vectors, and a judgment as to whethereach time step is compliant or non-compliant with the blind-sweep protocol for representative blind-sweep examples. The training data 2010a is fed into an untrained neural network 2010a in an iterative training process that will be familiar to a person of ordinary skill in the art.

[0163] The parameters of a network model (e.g., the weights at each artificial neuron) are initialized with initial values A that may be random values or with results from training on prior datasets. In an iterative process, the network is used to make detection inferences on the training images, the results are compared with the ground truth annotations, and an optimizer is used to adjust the network parameters B until a metric of accuracy is maximized.

[0164] Thus, an output of this training process 2000 is a trained neural network 2010b, wherein the parameters B (e.g., weights) are optimized for generating accurate compliant / non-compliant judgments for the training data 2005a.

[0165] Figure 20B is a schematic, diagrammatic overview, in block diagram form, of an inference mode or clinical usage mode 2004 for the trained neural network 2010b, according to aspects of the present disclosure. In clinical usage, an ultrasound video or 2020 of the blind sweep is fed to the trained and validated neural network 2010b for analysis, along with IMU data 2030. The trained and validated neural network 2010b then produces, as an output, a classification 2040 of the image (or the entire sweep) as either adherent or non-adherent to the blind-sweep protocol. In some aspects, a confidence value can be determined as a normalized value in the range [0-1], where 0 indicates lowest confidence, and 1 indicates highest confidence that the classification is correct.

[0166] An output 2050 is then generated for the user indicating whether the frame (or sweep) is compliant or non-compliant.

[0167] Figure 20C is a schematic, diagrammatic overview, in block diagram form, of an inference mode or clinical usage mode 2006 for the trained neural network 2010b, according to aspects of the present disclosure. In clinical usage, an ultrasound video or 2020 of the blind sweep is fed to the trained and validated neural network 2010b for analysis, along with IMU data 2030. The trained and validated neural network 2010b then produces, as an output, estimates 2060 of the angular velocity, orientation, velocity, and / or position of ultrasound probe. In some aspects, confidence values can be determined as a normalized value in the range [0-1], where 0 indicates lowest confidence, and 1 indicates highest confidence that the classification is correct.

[0168] A detection step 2070 then detects any deviation(s) from the protocol, e.g., points when angular velocity, orientation, velocity, and / or position do not satisfy threshold(s). Anoutput 2080 is then generated for the user indicating whether, and how, and to what degree, the frame (or sweep) is compliant or non-compliant.

[0169] Figure 21 is a schematic, diagrammatic representation, in hybrid flow diagram / block diagram form, of an example ultrasound sweep guidance method 2100, according to aspects of the present disclosure.

[0170] In step 2120, an ultrasound probe 110 that includes magnetic coils 2120 is controlled (e.g., by a processor) to detect a local magnetic field generated by a fixed magnetic field generator 2110 field, and voltages are induced in the coils indicative of the system’s pose and position within the magnetic field. Execution then proceeds to step 2130.

[0171] In step 2130, the method 2100 includes determining (e.g., with the processor) the angular velocity, orientation, velocity, and / or position of the ultrasound probe based on the voltage(s). Some parameters, such as position and orientation, may be determined directly from the magnetic field, whereas others, such as velocity and angular velocity, may be derived. Execution then proceeds to step 2140.

[0172] In step 2140, the method 2100 includes detecting any deviation(s) from the blind sweep protocol, e.g., times when the angular velocity, orientation, velocity, and / or position do not satisfy their respective thresholds. Execution then proceeds to step 2150.

[0173] In step 2150, the method 2100 includes generating an output indicative of the deviation(s), or lack thereof. Execution is now complete.

[0174] As will be readily appreciated by those having ordinary skill in the art after becoming familiar with the teachings herein, the ultrasound sweep guidance system advantageously permits untrained and minimally trained users to perform an ultrasound blind sweep protocol to gather anatomical images of high quality. This may result in higher accuracy and higher clinician trust in the results, while potentially decreasing the amount of time required to perform the sweeps.

[0175] The systems, methods, and devices described herein may be applicable in point of care and handheld ultrasound use cases such as with the Philips Lumify system. The ultrasound sweep guidance system can be used for any handheld imaging applications, including but not limited to obstetrics, lung imaging, and echocardiography. The ultrasound sweep guidance system could be deployed on handheld mobile ultrasound devices, and on portable or cart-based ultrasound systems. The ultrasound sweep guidance system can be used in a variety of settings including emergency departments, ambulances, accident sites, and homes. The applications could also be expanded to other settings.

[0176] The invention is detectable from its functionality and output such as the auditory, visual, and / or haptic feedback it provides in relation to the transducer motion. This invention increases the value proposition of ultrasound applications in the obstetrics context, especially for use by minimally trained users.

[0177] A number of variations are possible on the examples and aspects described above. For example, the systems, methods, and devices described herein are not limited to obstetric ultrasound applications. Rather, the same technology can be applied to images of other organs or anatomical systems such as the lungs, heart, brain, digestive system, vascular system, etc. Furthermore, the technology disclosed herein is also applicable to other medical imaging modalities where 3D data are available, such as other ultrasound applications, camera-based videos, X-ray videos, and 3D volume images, such as computer aided tomography (CT) scans, magnetic resonance imaging (MRI) scans, optical coherence tomography (OCT) scans, or intravenous ultrasound (IVUS) pullback sequences. The technology described herein can be used in a variety of settings.

[0178] Accordingly, the logical operations making up the aspects of the technology described herein are referred to variously as operations, steps, objects, layers, elements, components, algorithms, or modules. Furthermore, it should be understood that these may occur or be performed or arranged in any order, unless explicitly claimed otherwise or a specific order is inherently necessitated by the claim language.

[0179] All directional references e.g., upper, lower, inner, outer, upward, downward, left, right, lateral, front, back, top, bottom, above, below, vertical, horizontal, clockwise, counterclockwise, proximal, and distal are only used for identification purposes to aid the reader’s understanding of the claimed subject matter, and do not create limitations, particularly as to the position, orientation, or use of the ultrasound sweep guidance system. Connection references, e.g., attached, coupled, connected, joined, or “in communication with” are to be construed broadly and may include intermediate members between a collection of elements and relative movement between elements unless otherwise indicated. As such, connection references do not necessarily imply that two elements are directly connected and in fixed relation to each other. The term “or” shall be interpreted to mean “and / or” rather than “exclusive or.” The word “comprising” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude a plurality. Unless otherwise noted in the claims, stated values shall be interpreted as illustrative only and shall not be taken to be limiting.

[0180] The above specification, examples and data provide a complete description of the structure and use of exemplary aspects of the ultrasound sweep guidance system as defined in the claims. Although various aspects of the claimed subject matter have been described above with a certain degree of particularity, or with reference to one or more individual aspects, those skilled in the art could make numerous alterations to the disclosed aspects without departing from the spirit or scope of the claimed subject matter.

[0181] Still other aspects are contemplated. It is intended that all matter contained in the above description and shown in the accompanying drawings shall be interpreted as illustrative only of particular aspects and not limiting. Changes in detail or structure may be made without departing from the basic elements of the subject matter as defined in the following claims.

Claims

CLAIMSWhat is claimed is:

1. A system, comprising: a processor configured for communication with an ultrasound probe, wherein the processor is configured to: receive a plurality of ultrasound image frames obtained by the ultrasound probe during a sweep of a blind sweep protocol on a patient body; receive probe motion information for the sweep, wherein the probe motion information is representative of at least one of a position, a velocity, or an orientation of the ultrasound probe while the ultrasound probe obtains the plurality of ultrasound image frames; perform a comparison between the probe motion information and a desired probe motion information, wherein the desired probe motion information is predetermined based on the blind sweep protocol; generate a first user guidance based on the comparison; and provide an output representative of the first user guidance, wherein the output is provided in real time while the ultrasound probe obtains the plurality of ultrasound image frames during the sweep.

2. The system of claim 1, wherein the first user guidance is representative of a change to at least one of the position, the velocity, or the orientation of the ultrasound probe.

3. The system of claim 1, wherein the processor is configured to: determine, based on the comparison, whether the sweep is compliant or non- compliant to the blind sweep protocol; generate a second user guidance representative of whether the sweep is compliant or non-compliant; and provide an output representative of the second user guidance.

4. The system of claim 3,wherein, when the sweep is compliant, the second user guidance comprises an instruction to perform the next sweep or an indication that the blind sweep protocol is complete, wherein, when the sweep is non-compliant, the second user guidance comprises an instruction to repeat the sweep.

5. The system of claim 1, wherein the probe motion information is obtained by at least one of an accelerometer, gyroscope, or magnetometer disposed within the ultrasound probe.

6. The system of claim 1, wherein the probe motion information is obtained by a magnetic coil within the ultrasound probe, wherein a voltage generated within the magnetic coil is responsive to at least one of a position, a velocity, or an orientation of the magnetic coil relative to an external magnetic field.

7. The system of claim 1, wherein the probe motion information is obtained by at least one camera, wherein images generated by the at least one camera are responsive to at least one of a position, a velocity, or an orientation of the ultrasound probe.

8. The system of claim 1, wherein the processor is configured to: output, to a display in communication with the processor, instructions to a user to perform the sweep.

9. The system of claim 8, wherein the instructions to the user comprise a visual indication of at least one of a position, a direction, a length, or a duration of the sweep on the patient body.

10. The system of claim 1, wherein the output representative of the first user guidance comprises a visual feedback displayed on a display in communication with the processor.

11. The system of claim 10, wherein the visual feedback comprises at least one of a bubble level, a correction arrow, or a visual representation of the ultrasound probe.

12. The system of claim 1, wherein the output representative of the first user guidance comprises an auditory feedback provided through a speaker in communication with the processor.

13. The system of claim 12, wherein the auditory feedback comprises at least one of a volume, a pitch, a frequency, or a duty cycle indicative of how close the probe motion information is to the desired probe motion information.

14. The system of claim 1, wherein the output representative of the first user guidance comprises a haptic feedback provided by a haptic motor in communication with the processor.

15. The system of claim 14, wherein the haptic motor is disposed within the ultrasound probe.

16. The system of claim 14, wherein the haptic feedback comprises at least one of a frequency, an amplitude, or a duty cycle indicative of how close the probe motion information is to the desired probe motion information.

17. The system of claim 1, further comprising: the ultrasound probe.

18. A method, comprising: receiving, with a processor in communication with an ultrasound probe, a plurality of ultrasound image frames obtained by the ultrasound probe during a sweep of a blind sweep protocol on a patient body; receiving probe motion information for the sweep, wherein the probe motion information is representative of at least one of a position, a velocity, or an orientation of the ultrasound probe while the ultrasound probe obtains the plurality of ultrasound image frames; performing a comparison between the probe motion information and a desired probe motion information, wherein the desired probe motion information is pre-determined based on the blind sweep protocol; generating a user guidance based on the comparison; andproviding an output representative of the user guidance, wherein the output is provided in real time while the ultrasound probe obtains the plurality of ultrasound image frames during the sweep.

19. The method of claim 18, wherein the probe motion information is obtained by at least one of an accelerometer, gyroscope, or magnetometer disposed within the ultrasound probe.

20. The method of claim 18, wherein the user guidance comprises at least one of: a visual feedback displayed on a display in communication with the processor, wherein the visual feedback comprises at least one of a bubble level, a correction arrow, or a visual representation of the ultrasound probe; an auditory feedback provided through a speaker in communication with the processor, wherein the auditory feedback comprises at least one of a volume, a pitch, a frequency, or a duty cycle indicative of how close the probe motion information is to the desired probe motion information; or a haptic feedback provided by a haptic motor in communication with the processor, wherein the haptic feedback comprises at least one of a frequency, an amplitude, or a duty cycle indicative of how close the probe motion information is to the desired probe motion information.

Images