System and method for acquiring medical ultrasound images

Abstract

A system for acquiring medical ultrasound images of a subject includes a probe having an ultrasound transducer for capturing ultrasound images, a memory having instruction data representing a set of instructions, and a processor in communication with the memory and configured to execute the set of instructions, which when executed by the processor, cause the processor to receive ultrasound images acquired by the probe, provide the images as inputs to a model to be trained, receive an indication from the model of whether the images comprise images relevant to a medical diagnostic process, and determine whether to bookmark the images based on the received indication.

Description

[Technical Field]

[0001] TECHNICAL FIELD The disclosure herein relates to ultrasound imaging. In particular, but not exclusively, embodiments herein relate to systems and methods for recording ultrasound images. [Background technology]

[0002] Ultrasound imaging is used in a range of medical applications, such as breast tissue examination and fetal monitoring. Medical ultrasound imaging involves moving a probe with an ultrasound transducer that generates high-frequency sound waves on the skin. The high-frequency sound waves traverse the tissue and reflect off internal surfaces (e.g., tissue boundaries). The reflected waves are detected and used to construct an image of the internal structure of interest.

[0003] Ultrasound imaging can be used to create two-dimensional or three-dimensional images. In a typical workflow, a user (e.g., a sonographer, radiologist, clinician, or other medical professional) can use the two-dimensional images to locate an anatomical feature of interest. Once the feature is located in two dimensions, the user can activate a three-dimensional mode to capture a three-dimensional image. Summary of the Invention [Problem to be solved by the invention]

[0004] The aim of the embodiments herein is to improve such methods. [Means for solving the problem]

[0005] In a typical workflow, a sonographer may perform an ultrasound examination on a subject. The sonographer bookmarks / saves images and videos that they consider relevant (e.g., images in which pathology is observed, or standard views for anatomical structure measurements). These bookmarked images are saved by the system for review by more experienced radiologists, who may then complete the medical diagnosis process (e.g., diagnose a specific disease or perform specific measurements).

[0006] Therefore, radiologists rely on sonographer acquisition when reviewing images at a later time. If the sonographer inadvertently does not save relevant images (e.g., small tumors), this can result in either an incomplete diagnosis or a false-negative diagnosis. If the radiologist deems it necessary, the subject may have to be recalled and rescanned. This is inefficient, wastes time and resources, and results in higher medical costs. Therefore, it is desirable to improve such methods.

[0007] Thus, according to a first aspect of the present specification, there is a system for acquiring medical ultrasound images of a subject. The system includes a probe having an ultrasound transducer for capturing ultrasound images, a memory having instruction data representing a set of instructions, and a processor in communication with the memory and configured to execute the set of instructions. The set of instructions, when executed by the processor, cause the processor to receive ultrasound images taken by the probe, provide the images as input to a model to be trained, receive an indication from the model whether the images include images relevant to a medical diagnostic process, and determine whether to bookmark the images based on the received instructions.

[0008] Thus, according to this method, a model can be used to determine the relevance of each ultrasound image frame to better ensure that all image frames relevant to the medical diagnosis process are bookmarked and saved. In this way, fewer relevant images are missed by sonographers, improving the data set made available to radiologists for patient diagnosis, potentially leading to improved patient care. Furthermore, this facilitates improved productivity as diagnoses are made with greater accuracy and there is less need to recall subjects for re-examination.

[0009] According to a second aspect, there is a method for acquiring medical ultrasound images. The method includes receiving ultrasound images, providing the images as input to a model to be trained, receiving an indication from the model whether the images include images relevant to a medical diagnostic process, and determining whether to bookmark the images based on the received indication.

[0010] According to a third aspect, there is a method of training a machine learning model to predict whether an image includes an image relevant to a medical diagnostic process. The method includes providing training data to the machine learning model, the training data including i) example ultrasound images and ii) corresponding ground truth labels, the ground truth labels indicating whether the image includes an image relevant to the medical diagnostic process. The method further includes training the machine learning model to predict whether a new unseen image includes an image relevant to the medical diagnostic process based on the training data.

[0011] According to a fourth aspect, there is provided a computer program product having a computer readable medium having computer readable code embodied therein which, when executed by a suitable computer or processor, causes the computer or processor to carry out the means of the second or third aspects.

[0012] For a better understanding and to show more clearly how embodiments herein may be carried into effect, reference will now be made, by way of example only, to the accompanying drawings in which: [Brief explanation of the drawings]

[0013] [Figure 1] 1 illustrates an exemplary system according to some embodiments of the present disclosure. [Figure 2] FIG. 2 illustrates an exemplary method according to some embodiments of the present disclosure. [Figure 3] FIG. 3 illustrates an exemplary method according to some embodiments of the present disclosure. [Figure 4] FIG. 4 illustrates an exemplary method according to some embodiments of the present disclosure. [Figure 5a] 1 illustrates an exemplary display according to some embodiments of the present disclosure. [Figure 5b] 1 illustrates an exemplary display according to some embodiments of the present disclosure. [Figure 5c] 1 illustrates an exemplary display according to some embodiments of the present disclosure. [Figure 6] 1 illustrates an exemplary system according to some embodiments of the present disclosure. [Figure 7] 1 illustrates an exemplary method according to some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0014] Due to the volume and detail of ultrasound data obtained in an ultrasound examination, it is usually impractical or undesirable to save all images obtained in the examination. Therefore, sonographers typically bookmark (e.g., flag for saving) specific images within an examination, which are then saved and made available for review by radiologists at a later date. As discussed above, it is an object of the present disclosure to provide an improved workflow for acquiring medical ultrasound images of a subject to ensure that all images relevant to a medical diagnostic process or procedure are captured and therefore available to a radiologist for review. This may facilitate improved patient diagnoses and increased productivity in hospitals.

[0015] 1 illustrates a system (e.g., device) 100 for recording ultrasound images, according to some embodiments herein. The system 100 is for recording (e.g., acquiring or photographing) ultrasound images. The system 100 may comprise or be part of a medical device, such as an ultrasound system.

[0016] 1, system 100 includes a processor 102 that controls the operation of system 100 and that can implement the methods described herein. Processor 102 can include one or more processors, processing units, multi-core processors, or modules that are configured or programmed to control system 100 in the methods described herein. In particular implementations, processor 102 can include multiple software and / or hardware modules that are each configured to perform or for performing an individual step or steps of the methods described herein.

[0017] 1, the system 100 may also include a memory 104 configured to store program code that can be executed by the processor 102 to perform the methods described herein. Alternatively, or in addition, one or more memories 104 may be external to the system 100 (i.e., separate or remote). For example, one or more memories 104 may be part of another device. The memory 106 may be used to store images, information, data, signals, and measurements obtained or made by the processor 102 of the apparatus 100 or from any interface, memory, or device external to the apparatus 100.

[0018] 1, the system 100 may further include a transducer 108 for capturing ultrasound images. Alternatively or additionally, the system 100 may receive (e.g., via a wired or wireless connection) a data stream of ultrasound images taken using an ultrasound transducer external to the system 100.

[0019] A transducer may be formed from multiple transducer elements. Such transducer elements may be arranged to form an array of transducer elements. A transducer may be included in a probe, such as a handheld probe, that can be held by a user (e.g., a sonographer, radiologist, or other clinician) and moved over a patient's skin. Those skilled in the art will be familiar with the principles of ultrasound imaging, but simply stated, an ultrasound transducer contains a piezoelectric crystal that can be used to both generate and detect / receive sound waves. Ultrasound waves generated by the ultrasound transducer enter the patient's body and reflect off underlying tissue structures. The reflected waves (e.g., echoes) are detected by the transducer and compiled (processed) by a computer to generate an ultrasound image of the underlying anatomical structures, also known as a sonogram.

[0020] In some embodiments, the transducer may include a matrix transducer capable of interrogating a volume of space.

[0021] 1, the system 100 may also include at least one user interface, such as a user display 106. The processor 102 may be configured to control the user display 106 to, for example, display or render portions of the received data stream or ultrasound images and / or alerts to a user. The user display 106 may include a touchscreen or an application (e.g., on a tablet or smartphone), a display screen, a graphical user interface (GUI), or other visual rendering component.

[0022] Alternatively, or in addition, at least one user display 106 may be external to (i.e., separate or remote from) the system 100. For example, at least one user display 106 may be part of another device. In such an embodiment, the processor 102 may be configured to send instructions (e.g., via a wireless or wired connection) to a system display 106 external to the system display 100 to trigger (e.g., activate) the external system display to cause the system to display an alert indicating that a feature of interest is in the system's view.

[0023] 1 shows only the components needed to illustrate this aspect of the disclosure, and it will be understood that in an actual embodiment, system 100 may have additional components in addition to those shown. For example, system 100 may have a battery or other means for connecting system 100 to a mains power source. In some embodiments, as shown in FIG. 1, system 100 may also include a communications interface (or circuitry) 108 to enable system 100 to communicate with any interfaces, memories, and devices internal or external to system 100, for example, via a wired or wireless network.

[0024] Briefly, the processor 102 is configured to receive ultrasound images taken by the transducer 108 in the probe, provide the images as input to a model to be trained, receive an indication from the model whether the images include images relevant to a medical diagnostic process, and determine whether to bookmark the images based on the received indication.

[0025] As described in detail below, embodiments herein use a model, such as a deep learning model (e.g., a neural network), to determine which ultrasound images or views are relevant to the medical exam being performed. Images deemed relevant by the model can then be saved (in addition to any images flagged by the user / sonographer as relevant). This augments manually saved ultrasound image collections by automatically saving relevant images that may be overlooked by the user. Such deep learning systems can learn image "relevance" from training on breast image data acquired and labeled as relevant or irrelevant by highly skilled sonographers / clinicians during a training phase. Also, as described below, probe position can be added to the model as an optional input parameter to further enhance the image classification process. Some embodiments described herein further provide a way to display relevant images and their location within the body to the user, for example, to improve the user's ability to locate suspicious areas of tissue in subsequent exams. Training and using models in this manner provides an improved workflow, ensuring improved image capture and subsequent diagnosis and patient care.

[0026] More particularly, in this specification, reference to a subject refers to a subject or other person on whom ultrasound imaging is performed, and reference to a user can include a person, such as a clinician, physician, sonographer, medical professional, or any other user performing an ultrasound examination.

[0027] Generally, when an ultrasound examination is performed, it is performed for a specific purpose, referred to herein as a (specific) medical diagnostic process. Examples of medical diagnostic processes include a breast examination to look for abnormal breast tissue or breast lesions; a vascular examination of one of a subject's extremities (e.g., a hand, finger, toe), or a thyroid ultrasound. In some instances, the medical diagnostic process may be performed according to medical guidelines; for example, the medical diagnostic process may include a fetal examination to image a specific part of a fetus according to medical guidelines for fetal examination. However, those skilled in the art will understand that these are merely examples and that the embodiments described herein may be applied to a wide range of ultrasound medical diagnostic processes.

[0028] The term "relevant" is used herein to describe whether an image is relevant to a medical diagnostic process being performed. Relevance may describe whether a sonographer classifies an image as useful, important, necessary, essential, or otherwise relevant to the medical diagnostic process being performed. For example, an image may be relevant to a medical diagnostic process for determining whether a patient has a tumor if it shows abnormal tissue, such as a lesion. Conversely, an image may also be relevant to a cancer diagnosis if it demonstrates that the overall anatomical features are normal. Thus, in this sense, an image may be considered relevant if it is used by a clinician to make a diagnosis or otherwise perform a medical diagnostic process.

[0029] The model to be trained may include any model that is capable of taking an ultrasound image as input and providing as output a classification or prediction of whether the image is relevant to a medical diagnostic process, e.g., whether the image will be used by a user / clinician to perform a medical diagnostic process.

[0030] For example, the model may include a model trained using a machine learning process, such as a deep learning process. In some embodiments, the trained model may include a trained neural network or a deep neural network. Examples of suitable neural networks include, but are not limited to, convolutional networks such as a trained F-net or a trained U-net.

[0031] Those skilled in the art are familiar with neural networks, but simply put, a neural network is a type of supervised machine learning model that can be trained to predict a desired output for given input data. A neural network is trained by providing training data that includes example input data and the corresponding "correct" or ground truth outcomes that are desired. A neural network includes multiple layers of neurons, each representing a mathematical operation that is applied to the input data. The output of each layer in the neural network is fed to the next layer to generate an output. For each portion of the training data, the weights associated with the neurons are adjusted until an optimal weight is found that produces predictions for the training examples that reflect the corresponding ground truth.

[0032] In embodiments in which the trained model includes a neural network, the neural network may include a convolutional neural network, such as a U-net or F-net, suitable for classifying images according to relevance. The trained model may receive ultrasound images as input and provide as output an indication of whether the images include images that are relevant to the medical diagnostic process being performed. The indication may include a binary classification ("relevant" or "not relevant"), a graded classification (e.g., on a scale), a likelihood or percentage likelihood of relevance, or any other indication that the images are relevant (or not relevant).

[0033] In some embodiments, a Siamese neural network can be used. Briefly, a Siamese network is a convolutional neural network (CNN) that utilizes the use of the same weights while simultaneously operating on two separate inputs. Typically, they output a score of similarity between the two inputs. For example, in this specification, a Siamese network can be used to distinguish between images that have already been captured or bookmarked by a user and images that should be added to an auto-save list in order to avoid redundant information (redundant image capture).

[0034] In some embodiments, the trained model is trained to provide an indication of whether an image comprises an image relevant to a medical diagnostic process based on training data comprising i) exemplary ultrasound images and ii) a corresponding ground truth indication of whether each exemplary ultrasound image comprises an image relevant to a medical diagnostic process.

[0035] This is illustrated in Figure 2, which shows training data 202 including annotated ultrasound images being provided as input to a deep learning neural network model 204. The annotations include a ground truth indication of whether each example ultrasound image includes an image relevant to a medical diagnostic process. The annotated ultrasound images are provided to a deep learning neural network 204, which is trained using the training data to predict whether the image includes a relevant image 206 (e.g., relevant to a medical diagnostic process) based on the annotated ultrasound images 202.

[0036] Typically, training data (e.g., ultrasound image frames) may be collected from a wide range of available ultrasound systems and clinically annotated by expert sonographers and clinicians. Images classified as "relevant" can serve as input for a deep learning network trained to learn the characteristics of "relevant" images. To improve training accuracy, additional images (e.g., examples of "irrelevant" images) not saved by expert sonographers in a typical clinical scanning session can be collected. The advantage of training on all images (not just relevant saved images) is that the system learns not only which images are "relevant" and need to be saved, but also which images do not need to be saved. In this way, the network can better predict both images missed by the sonographer and those that will be saved. Furthermore, by training a model on a training dataset compiled (annotated) by multiple sonographers, the expertise of many different sonographers can be used to train the model. Thus, in effect, the model can combine the expertise of all sonographers who contributed to the training dataset when determining whether an image is relevant, which can be advantageous compared to a single sonographer manually classifying images.

[0037] Once the model is trained, inference can use the deep learning neural network to predict whether a new (e.g., unseen) image is relevant to the medical diagnostic process. During inference (live scanning), the network can run in the background. If a suspicious image is not bookmarked by the sonographer, the system can automatically capture it. Images classified as relevant may be bookmarked by the system and displayed to the user. Images may be displayed in groups, for example, as shown in pane 208 of FIG. 2, depending on whether the image is bookmarked by the user or by the system (for relevance classification).

[0038] Returning to FIG. 1, in some embodiments, the processor 102 is further configured to determine a position on the subject's body associated with the position of the probe when the image is captured.

[0039] The position can be determined in various ways. For example, the transducer 108 may be integrated with an electromagnetic (EM) sensor that can be tracked in an EM field generated, for example, by a tabletop field generator that is integrated with the patient table. The recorded position may be calibrated, for example, with reference to a particular feature. For example, in a breast exam, the sonographer can calibrate the position by indicating the nipple location to the system. Such an EM tracking feature allows for the coupling of B-mode images with the probe position to provide a visual mapping of the exam.

[0040] However, it will be understood that this is just one example and the position of the probe (transducer) on the body may be determined in other ways, including but not limited to, motion detectors within the probe, or where the probe position is determined using optical tracking (calibrated in a similar manner as described for the EM sensor).

[0041] In some embodiments, the set of instructions, when executed by the processor, further causes the processor to provide the determined location on the body as a further input to the trained model. In other words, in some embodiments, the trained model is further trained to provide an indication of whether the image comprises an image relevant to the medical diagnostic process based on the determined location. Thus, "relevance" may be determined based on two factors (e.g., inputs): (i) the image, and (ii) the probe location.

[0042] For example, if the system determines that an image is relevant, it may bookmark the image and save the image's location. The user can then continue to maneuver the probe around the region, looking for more "relevant" images to bookmark. The user then attempts to arrive at a probe position that is orthogonal to the probe position originally marked herein. Even if the "relevance" of this image is borderline from the image features themselves (e.g., clearly irrelevant), the model can still determine that the image is relevant (and therefore should be bookmarked) because the probe position indicates that this image should represent another slice through the object of interest (e.g., a tumor).

[0043] This is illustrated in Figure 3, which shows training data including ultrasound images 302 and corresponding probe positions 304 provided as input to a deep learning neural network model 306. Based on the ultrasound images 302 and probe positions 304, the deep learning neural network 306 is trained to predict an indication of whether the images contain relevant images 308 (e.g., relevant to a medical diagnostic process).

[0044] In this approach, probe positions along with B-mode ultrasound images are input into a deep neural network, allowing the deep learning network to determine across adjacent frames (in time and space) which frames are most relevant and intelligently select which frames to auto-capture, avoiding redundant information for the sonographer. Images, along with clinical annotations and probe positioning, may be used as labels to train the deep learning network. This training approach leverages meaningful information contained in the stack of ultrasound images; for example, given a sequence of five frames, the deep learning network can select the most relevant image frame to store. In some embodiments, the network can output a score for each image frame that classifies the best image frame to capture. Similarity across image frames can be learned through the frame's salient features and pose information.

[0045] Furthermore, it is envisioned that 3D deep learning networks can intelligently reconstruct 3D volumes from, for example, breast sweeps and select suspect planes to automatically capture relevant frames. This training option may be useful for 3D deep learning networks used in breast exams, where the entire breast volume can be captured.

[0046] Since each frame is labeled with its relative probe position, a deep learning network may be trained to a) reconstruct the entire volume given the probe position, and b) determine which planes are relevant and bookmark the relevant frames within the stitched volume.

[0047] This is shown in Figure 4, where a deep learning neural network 406 takes as input multiple ultrasound images 402 and corresponding probe positions 404 and outputs a reconstructed 3D volume 408 whereby each slice through the 3D volume is labeled according to relevance 410. Inference can use graphics such as graphics 412 to highlight / display the locations of relevant frames to the user.

[0048] Those skilled in the art are familiar with methods for training neural networks using training data (e.g., gradient descent, etc.) and will understand that the training data may include hundreds or thousands of rows of training data obtained under a diverse range of network conditions.

[0049] It is noted that, in general, training and validation data sets can be improved if they include images from a diverse set of exams containing different diagnostic findings that sonographers want to preserve, and a large population of patients with different tissue characteristics. Training data can be further improved if training images are obtained from ultrasound exams performed and annotated by different expert users. This can better capture inter-user variability and, as discussed above, results in a model that effectively combines the expertise of many sonographers across many different tissue types.

[0050] Once the model is trained, inference can use the deep learning neural network to predict whether new (e.g., unseen) images will be relevant. Images classified as relevant may be bookmarked by the system and displayed to the user according to whether the image is bookmarked by the user or by the system (for relevance classification), e.g., as shown in pane 310 of Figure 3 and pane 412 of Figure 4.

[0051] Thus, in summary, relevance may be determined from the image alone. In such instances, the probe location may be stored in parallel to facilitate user interpretation of the bookmarked image. Alternatively, relevance may be determined from the image and the probe location.

[0052] Those skilled in the art will further appreciate that other inputs may be provided to the model in addition to, or instead of, the examples provided herein.

[0053] Additionally, the model may be trained to have other output channels. For example, in some embodiments, the model may further output suggestions to the user of additional locations the user may wish to scan. For example, in a breast exam, the sonographer may decide to scan the axillary lymph nodes (closer to the armpit) to check for metastases. The system may also be trained to perform this task, where, based on images deemed "relevant" and captured in the previous embodiment, the system may suggest that the user scan the axillary lymph nodes.

[0054] It will be appreciated that alternatively, the system may provide the user with additional locations for the user to perform additional scans based on the instruction(s) received from the model.

[0055] Once the probe is properly positioned for an axillary lymph node scan, "associated" images can also be stored here (and annotated separately).

[0056] Furthermore, although the examples herein describe the use of neural networks, it will be understood that any model capable of taking the above inputs and outputting an indication of whether the image is relevant to the medical diagnostic process may be used, including, but not limited to, a random forest model.

[0057] When a processor receives ultrasound images taken by the probe, provides the images as input to the model to be trained, and receives an indication from the model whether the images include images relevant to a medical diagnostic process, as described above, the processor is then configured to determine whether to bookmark the images based on the received indication.

[0058] In some embodiments, the processor is further configured to determine whether the image is selected by a user as an image to be bookmarked (manually bookmarked). The processor may then decide to bookmark the image if the image is not selected by the user as an image to be bookmarked and if an indication received from the trained model indicates that the image is relevant to the medical diagnostic process. In other words, if the model determines that the image is relevant but not selected to be bookmarked by the user (sonographer), the system can decide to bookmark the image independently of the sonographer.

[0059] Depending on whether a bookmarked image is bookmarked by a user or by the system, the bookmarked image may be recorded (e.g., tagged), e.g., due to being classified by the model as relevant, and such tags may be used by the system when displaying the image to the user.

[0060] For example, the processor may send instructions to the display to display the image to the user and further indicate to the user whether the image is i) selected by the user as an image to be bookmarked or ii) bookmarked by the system based on instructions received from the model being trained.

[0061] Images may be grouped and displayed in different panes according to whether they are bookmarked manually or by the system, as described above and shown by reference numeral 208 in FIG. 2, reference numeral 310 in FIG. 3, and reference numeral 412 in FIG. 4.

[0062] Additionally, the images may be grouped and / or displayed according to their location on the body. An example is shown in Figure 5a, which shows a schematic diagram of a breast 52 overlaid with the location of the ultrasound images taken. Each displayed ultrasound image may be color-coded according to whether the image is bookmarked by the user or sonographer 54 or by the system 56.

[0063] Turning now to other embodiments, the processor may be further configured to determine a location on the subject's body associated with the position of the probe when the image is captured and send instructions to the display to display an indication of the determined location on the body to a user. This may be performed for each image classified as relevant by the model.

[0064] In the example shown in Figure 5b, the medical diagnostic process includes a breast examination and the body part includes the breast. In this embodiment, the processor causes a schematic diagram of the breast 502 to be displayed. During the examination, the position of the probe (and / or the previous positions of the probe) may be indicated by means of a line or trajectory 501. If an additional output channel is provided that provides an indication of another position on the breast where imaging may be performed, this may be indicated, for example, by an arrow 503 that provides an indication of the direction in which the probe should be moved. The different frames may be presented in groups 412 as described above.

[0065] More particularly, the processor may be adapted to send instructions to the display to display a representation of the body part and to overlay an indication of the determined location on the body over the representation of the body part. For example, a cartoon or other schematic diagram of the body part may be displayed. The location of the associated image may be represented as a pinpoint, bullet point, dot, cross, or any other means of indicating a location on the representation of the body part.

[0066] 5c, the medical diagnostic process includes a breast exam and the body part includes a breast. Accordingly, the processor causes a schematic representation of a breast 502 to be displayed. Overlaid on the representation of the breast are icons indicating the locations of image frames determined by the model as being relevant to the medical diagnostic process of a breast exam.

[0067] Such graphics can be used in subsequent (e.g., follow-up) ultrasound scans to help the sonographer more easily locate the relevant image locations. For example, the processor may be adapted to receive a real-time sequence of subsequent ultrasound images (e.g., from a subsequent scan) and send instructions to the display to display an indication of the position of the probe when each ultrasound image in the real-time sequence of subsequent ultrasound images is captured compared to the position of the probe when the image is captured. This may help the sonographer find the relevant image plane more quickly, making image processing more efficient.

[0068] In the embodiment shown in FIG. 5c, the current probe position is indicated by a line 506 relative to its position on the breast representation. An image currently being captured by the probe (e.g., a live image 508) may be shown next to the body part representation. In some embodiments, the model is further trained, e.g., has additional output channels, to indicate the location of lesions in the image, as described above. As an example, the You Only Look Once (YOLO) neural network is suitable for real-time lesion detection. More information on YOLO networks can be found here: https: / / arxiv.org / abs / 1506.02640. Detected lesions may be encapsulated in a box 510 (e.g., a colored box) and / or labeled with a score indicating the certainty of the decision. In this way, lesions can be more quickly and reliably identified during breast examinations.

[0069] Turning now to other embodiments, in general, the processor may be caused to send instructions to the probe to cause it to change to a different imaging mode if instructions received from the model indicate that the images include images relevant to a medical diagnostic process. For example, if relevant images are determined, the system may suggest that the sonographer image a particular anatomical structure to be captured, which may be (automatically) bookmarked by the system in additional ultrasound modes (e.g., microflow imaging, color flow, and / or shear wave elastography, among others). For example, if a tumor is suspected on a given image, activating a color mode may provide supplemental information for tumor diagnosis by studying the tumor vasculature. In such embodiments, the model's training data may also include a ground truth field corresponding to whether the imaging model should be changed and to which mode, in order to train the model to suggest imaging modes.

[0070] In one embodiment combining several of the above elements, once trained, the system 100 can be used in an inference mode in a clinic. In this embodiment, the system can be fed raw (e.g., real-time) ultrasound images and corresponding probe positions. Information about user manual image saving is also available to the system. The model is used to output a binary variable for each ultrasound image, i.e., "relevant" or "not relevant," as well as probe pose (if and when available). If the image is deemed "not relevant," no action is taken. Otherwise, in this embodiment, the system checks whether the user has already manually saved the image. If yes, no action is taken. If not, the system auto-saves the image. Auto-saved images may be kept in a buffer in case the sonographer later bookmarks the same image (to avoid duplication). If the sonographer does not take a bookmarking action within a threshold time interval, the system adds this frame to an auto-save list and saves it. The examination may be performed via deep learning with pose regression and image redundancy validation on images already captured by the sonographer. A list of manually saved and autosaved images is maintained and then displayed to the user individually, such as in a grouped image box.

[0071] In this embodiment, the system 100 continues to monitor and update the manually saved and automatically saved images. When the system automatically saves an image, it is possible for the sonographer to return to the same location and manually save an equivalent image in the same location after a subsequent visual inspection. In such cases, if a duplicate is identified, the list of manually saved images may be updated accordingly, and the original automatically saved image may be deleted. Alternatively, the sonographer may be notified when additional images are captured that include that location within the scan field. This may prevent duplicate images from being saved and presented to the radiographer for subsequent review.

[0072] Image matching methods such as correlation and sum of differences can be combined with probe position information to check for image overlap.

[0073] Thus, various systems are disclosed that provide improved workflows for ultrasound systems. In clinical use, the systems described herein generally (1) augment current clinical workflows and improve diagnostic reliability by tracking sonographer actions during a given scan, including, but not limited to, live ultrasound images and corresponding probe positions during the ultrasound scan, manually saved US images, and probe positions corresponding to manually saved images. (2) Automatically save images determined by the system if the user has not manually saved images during the scan session. (3) Display options can enable subsequent reviewers of images, such as radiologists, to first review images archived by the sonographer and then have the option to review autosaved images. (4) To avoid redundancy, the systems described herein can maintain complementary lists of images in "Manually Saved" and "Autosaved" sections. (5) The systems described herein can further suggest to the sonographer to use additional ultrasound modes (e.g., microflow imaging, color flow, and shear wave elastography) to image specific anatomical structures captured by Autosaved.

[0074] Additionally, embodiments herein may be used for training purposes, as the list of "uncaptured" images may be expected to be longer for novice users compared to more highly trained sonographers. Thus, the systems herein may improve ultrasound examination efficiency, provide more objective data for first-time correct diagnosis, and reduce procedure time.

[0075] Referring now to Figure 6, Figure 6 illustrates an exemplary embodiment of an ultrasound system 600 constructed in accordance with the principles described herein. One or more components illustrated in Figure 6 may be included in a system configured to receive a data stream of two-dimensional images acquired using an ultrasound transducer, determine from the data stream that a feature of interest is within the field of view of the transducer, trigger an alert to be sent to a user indicating that the feature of interest is within the field of view of the transducer, and send instructions to the transducer to trigger the transducer to capture a three-dimensional ultrasound image after a predetermined time interval.

[0076] For example, any of the above-described functions of processor 102 may be programmed, e.g., via computer-executable instructions, into a processor of system 600. System 600 may further be programmed to perform method 700, described below. In some examples, functions of processor 102 or method 700 may be implemented and / or controlled by one or more processing components shown in FIG. 6, including, for example, image processor 636.

[0077] In the ultrasound imaging system of FIG. 6 , an ultrasound probe 612 includes a transducer array 614 for transmitting ultrasound waves into a region of the body and receiving echo information in response to the transmitted waves. The transducer array 614 may be a matrix array including multiple transducer elements configured to be individually activated. In other embodiments, the transducer array 614 may include a one-dimensional linear array. The transducer array 614 is coupled to a microbeamformer 616 within the probe 612, which may control the transmission and reception of signals by the transducer elements in the array. In the illustrated example, the microbeamformer 616 is coupled by a probe cable to a transmit / receive (T / R) switch 618 to switch between transmit and receive and to protect the main beamformer 622 from high-energy transmit signals. In some embodiments, the T / R switch 618 and other elements of the system may be included within the transducer probe rather than within a separate ultrasound system base.

[0078] In some embodiments herein, the ultrasound probe 612 may further include a motion detector, as described above, to detect movement of the probe.

[0079] The transmission of ultrasound beams from the transducer array 616 under the control of the microbeamformer 616 may be directed by a transmit controller 620 coupled to the T / R switch 618 and a beamformer 622 that receives input, for example, from a user operating a user interface or control panel 624. One of the functions controlled by the transmit controller 620 is the direction in which the beam is steered. The beam may be steered straight (orthogonally) forward from the transducer array or at a different angle for a wider field of view. The partially beamformed signals generated by the microbeamformer 616 are coupled to a main beamformer 622 in which the partially beamformed signals from individual patches of transducer elements are combined into a fully beamformed signal.

[0080] The beamformed signals are coupled to a signal processor 626, which may process the received echo signals in various ways, such as bandpass filtering, decimation, I and Q component separation, and harmonic signal separation. Data generated by the different processing techniques employed by the signal processor 626 may be used by a data processor to identify internal structures, such as lesions within the breast, ribs, or anatomical features of the newborn, and their parameters.

[0081] The signal processor 626 may also perform additional signal enhancements, such as speckle reduction, signal compounding, and noise removal. The processed signals may be coupled to a B-mode processor 628, which may use amplitude detection for imaging internal structures, including, for example, the ribs, heart, and / or pleural interfaces. The signals generated by the B-mode processor are coupled to a scan converter 630 and a multiplanar reformatter 632. The scan converter 630 arranges the echo signals in a spatial relationship to be received in a desired image format. For example, the scan converter 630 may arrange the echo signals in a two-dimensional (2D) fan-shaped format. The multiplanar reformatter 632 may convert echoes received from points in a common plane within a volumetric region of the body into an ultrasound image of that plane, as described in U.S. Pat. No. 6,443,896 (Detmer). The volume renderer 636 converts the echo signals of the 3D dataset into a projected 3D image viewed from a given reference point, as described in U.S. Pat. No. 6,530,885 (Entrekin et al.).

[0082] The 2D or 3D image processor is coupled from the scan converter 630, multiplanar reformatter 632, and volume renderer 634 to an image processor processor 636 where further enhancement, buffering, and temporary storage are performed for display on an image processor display 638.

[0083] The graphics processor 640 can generate graphic overlays for display with the ultrasound images. These graphic overlays can include, for example, a representation of the body part and / or an indication of the position of the probe when each ultrasound image in a real-time sequence of subsequent ultrasound images is captured compared to the position of the probe when the image is captured, as described with respect to FIG.

[0084] The graphic overlay may further include other information, such as standard identification information, such as the patient's name, the date and time of the image, imaging parameters, etc. The graphic overlay may also include one or more signals indicating that a target image frame is being acquired and / or that the system 600 is in the process of identifying the target image frame. The graphics processor may receive input, such as a typed patient name, from a user interface 624. The user interface 624 may also receive input prompts for adjustments in settings and / or parameters used by the system 600. The user interface may also be coupled to a multiplanar reformatter 632 for selection and control of the display of multiplanar reformat (MPR) images.

[0085] Those skilled in the art will appreciate that the embodiment shown in FIG. 6 is merely an example and that the ultrasound system 600 may include additional components to those shown in FIG. 6, such as a power supply or battery.

[0086] 7, in some embodiments, there is a method 700 for acquiring medical ultrasound images. The method includes receiving 702 ultrasound images, providing 704 the images as input to a model to be trained, receiving 706 an indication from the model that the images include images relevant to a medical diagnostic process, and determining 708 whether to bookmark the images based on the received indication.

[0087] The steps of receiving ultrasound images, providing the images as input to a model to be trained, receiving an indication from the model whether the images contain images relevant to a medical diagnostic process, and determining whether to bookmark the images based on the received indication are all described above with respect to system 100, and it will be understood that the details therein apply equally to method 700.

[0088] In some embodiments, there is also a method for training a machine learning model to predict whether an image comprises an image relevant to a medical diagnostic process. The method can be performed in addition to or separately from method 700. The method includes providing training data to the machine learning model, the training data including i) example ultrasound images and ii) corresponding ground truth labels, the ground truth labels indicating whether the image comprises an image relevant to the medical diagnostic process, and training the machine learning model to predict whether a new, unknown image comprises an image relevant to the medical diagnostic process based on the training data.

[0089] Training a model in this manner is described in detail above with respect to system 100, and it will be understood that the details apply equally to this method embodiment.

[0090] In another embodiment, a computer program product is provided that includes a computer readable medium having computer readable code embodied therein, the computer readable code being configured, upon execution of instructions by a suitable computer or processor, to cause the computer or processor to perform the method or methods described herein.

[0091] It will therefore be understood that the present disclosure also applies to computer programs adapted to carry out the embodiments, in particular computer programs on or in a carrier. The program may be in the form of object code, such as source code, object code, code intermediate sources, and partially compiled forms, or in any other form suitable for use in the implementation of the methods according to the embodiments described herein.

[0092] It will also be understood that such programs can have many different architectural designs. For example, program code implementing the functionality of a method or system may be subdivided into one or more subroutines. Many different ways of distributing functionality among these subroutines will be apparent to those skilled in the art. The subroutines may be stored together in an executable file to form a self-contained program. Such an executable file may include computer-executable instructions, such as processor instructions and / or interpreter instructions (e.g., Java interpreter instructions). Alternatively, one or more or all of the subroutines may be stored in at least one external library file and linked with the main program statically or dynamically, such as at run time. The main program includes at least one call to at least one subroutine. The subroutines may also include function calls to each other.

[0093] The carrier of a computer program may be any entity or device capable of carrying the program. For example, the carrier may include a data storage device such as a ROM, for example a CD ROM or a semiconductor ROM, or a magnetic recording medium, for example a hard disk. Furthermore, the carrier may be a transmissible carrier such as an electric or optical signal, which may be conveyed via an electric or optical cable or by radio or other means. When the program is embodied in such a signal, the carrier wave may be constituted by such cable or other device or means. Alternatively, the carrier may be an integrated circuit in which the program is embedded, the integrated circuit being adapted for, or used for, performing the relevant method.

[0094] Variations to the disclosed embodiments can be understood and effected by those skilled in the art from an examination of the drawings, the disclosure, and the appended claims. In the claims, the word "comprise" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain means are recited in mutually different dependent claims does not indicate that a combination of these means cannot be used to advantage. A computer program can be stored / distributed on a suitable medium, such as an optical storage medium or a solid-state medium, supplied together with or as part of other hardware, but can also be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems. Any reference signs in the claims should not be construed as limiting the scope.

Claims

1. 1. A system for acquiring medical ultrasound images of a subject, the system comprising: a probe having an ultrasound transducer for capturing ultrasound images; a memory having instruction data representing a set of instructions; a processor in communication with the memory and configured to execute the set of instructions, the set of instructions, when executed by the processor, causing the processor to: receiving an ultrasound image captured by the probe; providing the ultrasound images as input to a trained model; receiving an indication from the trained model whether the ultrasound image comprises an image relevant to a medical diagnostic process; determining whether to bookmark the ultrasound image based on the received indication; determining to bookmark the ultrasound image if the received indication from the trained model indicates that the ultrasound image is relevant to the medical diagnostic process; The processor and and the trained model is trained to provide an indication of whether the ultrasound image comprises an image relevant to the medical diagnostic process based on training data comprising: i) example ultrasound images; and ii) a corresponding ground truth indication of whether each example ultrasound image comprises an image relevant to the medical diagnostic process. system.

2. The processor further comprises: sending instructions to a display to display the ultrasound image to a user; sending instructions to the display to indicate to the user whether the ultrasound image is i) selected by the user as an image to be bookmarked, or ii) bookmarked by the system based on the received indication from the trained model; The system of claim 1 configured to execute:

3. The processor further comprises: determining a position on the subject's body relative to the position of the probe when the ultrasound image is captured; The system of claim 1 configured to perform:

4. The processor further comprises: sending instructions to a display to display an indication of the determined location on the body; The system of claim 3 configured to execute:

5. The processor further comprises: sending instructions to the display to display a representation of a body part and overlaying an indication of the determined location on the body over the representation of the body part; The system of claim 4 configured to execute:

6. The processor further comprises: receiving a subsequent real-time sequence of ultrasound images; sending instructions to the display to display an indication of the position of the probe as each ultrasound image in a subsequent real-time sequence of ultrasound images is captured compared to the position of the probe when the ultrasound image was captured; 6. The system of claim 4 or 5, configured to execute:

7. The set of instructions, when executed by the processor, causes the processor to: providing the determined location on the body as a further input to the trained model. Execute the trained model is further trained to provide the indication of whether the ultrasound image has an image relevant to the medical diagnostic process based on the determined location. A system according to any one of claims 3 to 6.

8. The set of instructions, when executed by the processor, further causes the processor to: sending a command to the probe to change the probe to a different imaging mode if the indication received from the trained model indicates that the ultrasound image has an image relevant to the medical diagnostic process. The system according to any one of claims 1 to 7, wherein the system executes the following:

9. The set of instructions, when executed by the processor, further causes the processor to: Suggesting additional locations to the user for the user to perform additional scans based on the indication received from the trained model. The system of claim 1 ,

10. 10. The system of claim 1, wherein the medical diagnostic process comprises a breast examination to determine breast tissue abnormalities.

11. 1. A method for acquiring medical ultrasound images, the method comprising: receiving an ultrasound image; providing the ultrasound image as an input to a trained model; receiving an indication from the trained model whether the ultrasound image comprises an image relevant to a medical diagnostic process; determining whether to bookmark the ultrasound image based on the received indication; and The step of determining whether to bookmark the ultrasound image comprises: determining to bookmark the ultrasound image if the received indication from the trained model indicates that the ultrasound image is relevant to the medical diagnostic process; and the trained model is trained to provide an indication of whether the ultrasound image comprises an image relevant to the medical diagnostic process based on training data comprising: i) example ultrasound images; and ii) a corresponding ground truth indication of whether each example ultrasound image comprises an image relevant to the medical diagnostic process. method.

12. training a machine learning model to predict whether the ultrasound images have images relevant to a medical diagnostic process, said training comprising: i) providing training data to the machine learning model, the training data comprising i) example ultrasound images and ii) corresponding ground truth labels, the ground truth labels indicating whether the ultrasound images comprise images relevant to the medical diagnostic process; training the machine learning model to predict whether a new unseen image has an image relevant to the medical diagnostic process based on the training data; using the trained model; 12. The method of claim 11, comprising:

13. 13. A computer program product having a computer readable medium having computer readable code embodied therein, the computer program product being configured, when executed by a suitable computer or processor, to cause said computer or said processor to perform the method of claim 11 or 12.

Images