Machine-Assisted Workflow in Ultrasound Imaging

Abstract

The present application relates to machine-assisted workflows in ultrasound imaging. By using computer-assisted classification and / or computer-assisted segmentation with or without monitoring changes in the field of view, the workflow for ultrasound imaging can be made more efficient. Classification and / or segmentation are used to perform the next action in a series of actions that constitute an ultrasound examination. Instead of requiring the user to determine an action and implement the action, the ultrasound scanner determines and implements the action based on the identification and / or location of the imaged object. For example, identifying an object as a kidney using a machine learning classifier triggers a color flow scan, and the location of the object determines the placement of a region of interest (ROI) for the color flow, avoiding the user from having to perform ROI initiation and / or placement and improving workflow efficiency.

Description

[0001] This application is a divisional application. The invention name of the parent case is “Machine-Assisted Workflow in Ultrasound Imaging”. The application date is November 9, 2018, and the application number is 201811330956.2. Background Art

[0002] This embodiment relates to ultrasound imaging. Machine learning has made great progress in solving image classification and segmentation. For example, deep convolutional neural networks can achieve reasonable performance on hard visual recognition tasks in ultrasound, matching or exceeding human performance in some domains. Classification or segmentation from ultrasound images is used to provide physicians with more information for diagnosis, prognosis or treatment.

[0003] Automatic classification or segmentation can assist the sonographer in reducing some of the time spent examining patients. The increasing and aging patient population is creating a need for improved healthcare efficiencies. This leads to a huge need for improvements in ultrasound imaging workflows from the perspective of increased patient throughput, reduced examination time, and reduced user stress through repetitive motions. While standardization of ultrasound examinations and automatic segmentation may help, greater efficiencies may be found that reduce examination time and resulting user stress. Summary of the invention

[0004] By way of introduction, the preferred embodiments described below include methods, instructions, and systems for machine-assisted workflow assistance for ultrasound scanners. The use of computer-assisted classification and / or computer-assisted segmentation (with or without monitoring changes in the field of view) can make the workflow for ultrasound imaging more efficient. Classification and / or segmentation are used to perform the next action in a series of actions that constitute an ultrasound examination. Instead of requiring the user to determine an action and implement the action, the ultrasound scanner determines and implements the action based on the identification and / or location of the imaged object. For example, identifying an object as a kidney using a machine learning classifier triggers a color flow scan, and the location of the object determines the placement of a region of interest (ROI) for the color flow, avoiding the user from having to perform ROI initiation and / or placement and improving workflow efficiency.

[0005] In a first aspect, a method for machine-assisted workflow assistance for an ultrasound scanner is provided. An internal region of a patient is scanned with an ultrasound transducer of the ultrasound scanner. The scan is repeated during an ongoing ultrasound examination. A machine learning classifier applied to scan data from the scan during the ongoing ultrasound examination identifies an object in the internal region. A machine learning segmentor is selected based on the identification of the object. The selection occurs during the ongoing ultrasound examination. The selected machine learning segmentor segments the object in the scan data during the ongoing ultrasound examination. The ultrasound scanner implements a next action in the workflow based on the segmentation of the object. The next action is implemented during the ongoing ultrasound examination and without user input to the ultrasound scanner between the segmentation and implementation. The ultrasound scanner generates an image of the object.

[0006] In a second aspect, a system for machine-assisted workflow assistance in ultrasound imaging is provided. An ultrasound scanner is configured to scan a region of a patient with ultrasound using a transducer. The scanning is repeated during an examination of the patient. An image processor is configured to: apply computer-assisted classification of one or more objects of the region from scanned scan data, apply computer-assisted segmentation of the one or more objects based on the classification, and change an operation of the ultrasound scanner based on the segmentation of the one or more objects. The change in operation is a next step in a workflow for the examination of the patient. A display is operable to display an image of the region with a graphic that shows the change and / or the next step in the workflow.

[0007] In a third aspect, a method for machine-assisted workflow assistance for an ultrasound scanner is provided. An ultrasound scanner scans a patient as part of a workflow for ultrasound examination of the patient. The workflow includes a series of actions. The ultrasound scanner monitors a change in a field of view of the scan. In response to the change, an object is detected from the scan using a machine learning network. The detection includes identifying the object and / or locating the object. Without user control, the operation of the ultrasound scanner is altered to be incremental along the sequence of the workflow. The alteration is based on the identification and / or location of the object.

[0008] The present invention is defined by the following claims, and nothing in this section should be taken as a limitation on those claims.Further aspects and advantages of the invention are discussed below in conjunction with the preferred embodiments, and may be claimed later, either individually or in combination. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The components and drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the invention. Moreover, in the drawings, like reference numerals designate corresponding parts throughout the different views.

[0010] Figure 1is a flow chart of one embodiment of a method for machine-assisted workflow assistance for an ultrasound scanner;

[0011] Figure 2 is a flow chart of another embodiment of a method for machine-assisted workflow assistance for an ultrasound scanner;

[0012] Figure 3 illustrates an exemplary basic workflow for ultrasound examination;

[0013] Figure 4 illustrating an exemplary ultrasound image with a region of interest indicator placed as the next action in a workflow;

[0014] Figure 5 illustrating an exemplary ultrasound image with a measurement caliper indicator placed as the next action in the workflow; and

[0015] Figure 6 An embodiment of a system for machine-assisted workflow assistance for ultrasound imaging. DETAILED DESCRIPTION

[0016] Image recognition is applied not only for recognition, but also to assist and improve ultrasound imaging workflow. Machine learning-based techniques have been increasingly successful in image classification and segmentation. At the same time, the demand for improved patient throughput has increased. Image recognition is used to improve and assist ultrasound imaging workflow for increased patient throughput and efficiency.

[0017] In one embodiment, machine learning based image classification and segmentation techniques are utilized to assist and improve ultrasound imaging workflows. Deviations from standard workflows are suggested and / or implemented based on pathology identification. As opposed to providing classification after the fact, image classification and segmentation occurs during live scanning to provide real-time workflow assistance during the actual live examination. Real-time operation allows presets, measurements, and annotations to be automatically loaded by the ultrasound scanner as opposed to manual input by the user. Automatic workflow modifications based on pathology detection are implemented, and the ultrasound scanner clearly communicates to the user when any workflow modifications are made.

[0018] Figure 1 A method for machine-assisted workflow assistance for an ultrasound scanner is shown. During an ultrasound examination of a patient, a series of actions are performed. At least some of the actions are performed in response to classification and / or segmentation of one or more objects in an internal region of the patient. The ultrasound scanner automatically performs the actions in response to the computer-assisted classification and / or segmentation, allowing a user to avoid changing the ultrasound scanner to implement the actions.

[0019] Figure 2 Show Figure 1Another embodiment of the method of claim 1. The segmentation may be repeated for a stable field of view relative to the patient. When the field of view changes significantly, one or more classifiers are applied to identify objects in the field of view and select an appropriate segmentation. In response to the identification and / or segmentation, actions of a workflow dependent on the identification and / or segmentation of the object are performed without user control.

[0020] This method is Figure 6 The ultrasound imaging system may be implemented in a system or a different system. For example, an ultrasound imaging system generates ultrasound images using a beamformer and a transducer. An image processor of the ultrasound imaging system applies computer-aided detection to identify and / or segment objects and implements actions in an inspection workflow in response to the identification and / or segmentation. The beamformer, memory, detector, and / or other devices may be used to acquire data, perform one or more of the actions, and / or output data. The image processor may control the device to perform Figure 1 and / or method 2.

[0021] Additional, different, or fewer actions may be provided. For example, the method is performed in action 40 without generating an image. As another example, computer-aided detection is performed in action 30 without specific actions 32-36. In yet another example, action 28 is not provided, such as where classification and / or segmentation is performed regardless of any changes in the field of view. In other examples, action 32 is provided without actions 34 and 36, or vice versa.

[0022] The actions are performed in the order described or shown (e.g., top to bottom or digital), but may be performed in other orders. For example, the segmentation of action 36 is performed before the classification identification by action 32. As another example, the monitoring of changes to action 28 may be performed after action 32.

[0023] In act 26, the ultrasound scanner scans the internal area of ​​the patient with the ultrasound transducer. The ultrasound scanner generates an ultrasound image of the patient. The image is generated by scanning the patient.

[0024] Any type of ultrasound scan and corresponding image may be generated, such as B-mode, flow or color mode (e.g., Doppler velocity or power), contrast agent, harmonics, pulsed wave Doppler (i.e., spectral Doppler), M-mode, Doppler tissue (i.e., tissue motion), or other ultrasound imaging modes representing acoustic interaction with the patient. Different modes detect different types of information, such as the intensity of the acoustic return (e.g., B-mode and M-mode), or the velocity (e.g., flow mode or Doppler tissue).

[0025] The scan samples the field of view. Electronic steering, the position of the transducer relative to the patient, and / or the structure of the transducer define the field of view being scanned. The field of view is a point, line, area, or volume inside the patient. The depth of the field of view (i.e., the distance from the transducer for that field of view) is set by the beamformer.

[0026] Any anatomical structure, inserted object, or abnormality may be in the field of view. For example, the sonographer is performing an ultrasound examination of a specific organ, such as the kidney or heart. Other objects include inserted devices, such as catheters or stents. An abnormality is a tissue or flow pattern that is not typical for the organ or patient, such as a lesion, cyst, or reflux.

[0027] The scan is repeated to continue imaging the patient within the examination period, such as within a five to twenty minute examination. The transducer and corresponding field of view are moved to locate the object of interest in the investigation mode. Once the organ is located, the scan continues. The type of scan may vary, depending on what is found in the area or due to examining different objects at different times. The workflow for the examination may require different types of scans or the same scan to perform other actions, such as measurements. The repetitions are continuous, such as scanning the field of view five, ten, twenty or more times a second. Pauses in scanning may occur during a single visit or appointment, during an ongoing ultrasound examination of a patient, while the sonographer is performing actions in the workflow.

[0028] During an ongoing ultrasound examination, a workflow may include various steps. A sequence of steps or a decision tree with or without branching decisions guides the ultrasound examination. Each action is typically performed by a sonographer. The sonographer controls the ultrasound scanner to cause the steps to occur, such as manually tracing a boundary for segmentation, positioning a box on the displayed image for ROI placement, annotating the image, selecting a measurement, indicating a measurement location, starting a measurement, starting image storage, selecting a patient record, and / or naming an image. Any given step can be automated, such as segmenting an image. Such automation can speed up the workflow. Further automation is provided by triggering and implementing at least a portion of the next step in the workflow based on the automatic implementation of the current step. Segmentation and / or classification are used to at least partially trigger and implement one or more other steps in the workflow.

[0029] The workflow may be unique to the sonographer, the hospital or medical practice's guidelines, the professional organization's guidelines, or other arrangements of actions to be performed for a given type of ultrasound examination. Any level of detail may be provided in the workflow. The workflow sequence addresses actions to be performed from the beginning of the examination (such as initially beginning with selecting an appropriate transducer for a given examination) to the end of the examination (such as storing or transmitting the results).

[0030] Figure 3An exemplary basic workflow for abdominal ultrasound examination is shown. In step 50, an initial scan is performed to locate the anatomical structure of interest (e.g., kidney, bladder, or liver). When the anatomical structure is located, a representative image is stored in step 52. The anatomical structure is then measured in step 54. The measurements may include placing a region of interest (ROI) and measuring with a different type of ultrasound scan (e.g., color flow), or placing calipers to measure a size (e.g., length, area, or volume). In step 56, the measurements are stored. Other workflows include other actions or more detailed steps to be performed. Exemplary workflow steps include configuring the ultrasound scanner to store images, reviewing images, taking measurements, updating patient records, annotating images, changing scan parameters (e.g., transmit frequency, receive frequency, depth of field, and / or type of scan), changing scan presets or applications (e.g., switching from a liver scan to a lesion scan), placing a Doppler gate, locating pathology, annotating images, and / or sending results. These various steps provide or result in providing information that will be used for diagnosis, prognosis, and / or treatment, but the various steps take time for the ultrasound examiner to implement. Rather than just automating known steps, real-time automation is performed based on classification and / or segmentation.

[0031] Each scan of a plane or volume provides a frame of scan data. An image may be generated from each frame. The scan data is imaging data. An image may be scan data useful for imaging, data formatted for display, and / or an image displayed. In persistent or other composite situations, an image may be generated from multiple frames. Tens, hundreds, or thousands of frames and / or images may be created for use in examining a patient in a single examination (e.g., during a given patient's visit to a sonographer).

[0032] While performing act 30, the scanning of act 26 is continued or repeated. For example, the scanning continues with the identification of act 32, the selection of act 34, the segmentation of act 36, and the implementation of act 38. During live scanning, object detection is performed in real time. To assist with workflow, the detection occurs in real time with the continuous scanning. The detection of act 30 in real time with the scanning of act 26 provides a solution for workflow enhancement during live scanning or during a given ultrasound examination of a patient.

[0033] In action 28, the ultrasound scanner monitors changes in the scan field of view. Due to transducer and / or patient movement, the field of view of the transducer may shift from one area of ​​the patient to another. This action may be intentional, such as to investigate the patient, or unintentional, such as due to sonographer tension or patient movement.

[0034] Once an area or object is found, the transducer may remain in position relative to the patient to continue scanning the area or object for more detailed inspection and / or execution of steps of the workflow. Although the field of view is maintained at the area or object, some actions may not need to be repeated. For example, identifying the object from act 32 is not repeated. After some inspections, the field of view may shift. The shift may result in the need to repeat the identification of act 32 because a different object may be in the field of view. For example, Figure 2 The branching of the method depending on the amount of change in the field of view is shown, where for large changes, action 32 is performed, and for small changes or no changes, actions 32 and 34 are skipped. The amount of change is detected in action 28. The amount is compared to a threshold to distinguish large changes (e.g., changes in the field of view) from small changes (e.g., changes due to physiological or accidental sonographer or patient movement). To initiate the method, action 32 is performed after completing an investigation of the patient to find the object or during the investigation to indicate successful discovery of the object.

[0035] Any monitoring of the level of change in the field of view may be used. For example, performing a correlation (e.g., a minimum sum of absolute differences or a cross-correlation) between recently acquired temporally adjacent scan data frames. As another example, a correlation is between a template or reference scan data frame and a recently acquired scan data frame. Other measures of the level of change may be used, such as tracking transducer motion with a magnetic or other position sensor.

[0036] The amount or level of change is compared to a threshold. When the level of change is below the threshold, a small change has occurred. When the level of change is above the threshold, a large change has occurred. For changes above the threshold level, the change may be considered large or small. The ultrasound scanner uses the threshold to identify the level of change. A pattern of change may be used, such as identifying a large change followed by a small change for a stable field of view over a period of time (e.g., 10 seconds). The pattern indicates a shift in the field of view to a new area and an interest in examining the new area. The pattern is identified from one or more thresholds.

[0037] The threshold and / or pattern of change links the real-time image movement detection to the detection of act 30. For example, in response to the level of change being above the threshold, the identification of act 32, the selection of act 34, and the segmentation of act 36 are performed (i.e., a change in the field of view with or without subsequent stable positioning), and in response to the level of change being below the threshold, the segmentation of act 36 is performed without repeating the identification of act 32 and the selection of act 34 (i.e., continuing to scan the previous field of view and the corresponding one or more previously identified objects).

[0038] In act 30, the ultrasound scanner detects the object from the scan. The detection is the classification (eg, identification) and / or location (eg, segmentation) of the object. In one embodiment, both classification and location detection are performed in any order.

[0039] The detection is performed in response to a determined change or pattern of changes. Alternatively, the detection is performed periodically, continuously or in response to another trigger. The monitoring of action 28 is performed or not performed.

[0040] The ultrasound scanner or image processor applies computer-aided detection to each, one, a subset, or all frames or images in a sequence acquired by scanning. Any computer-aided detection now known or later developed may be applied. For example, pattern matching is used to determine whether a pattern indicative of an organ, an anatomical landmark, an inserted device, and / or an abnormality (e.g., a tumor or cyst) is located in a frame or image. As another example, thresholding, random walks, or other image processing is applied.

[0041] For faster detection, a machine learning detector or network may be applied. In one embodiment, the machine learning detector is a deep learning neural network. Any machine learning or training may be used. Probability boosting trees, support vector machines, neural networks, sparse autoencoding classifiers, Bayesian networks, or other machine learning now known or later developed may be used. Any semi-supervised, supervised, or unsupervised learning may be used. Hierarchical or other methods may be used.

[0042] For machine training and application of a machine learning network, values ​​of any number of features are extracted from a sample of the scan data. Values ​​for texture of tissue represented in the scan data are extracted. The texture of the tissue is represented by measurements of the scan data.

[0043] Each feature defines a kernel for convolution with the data. The result of the convolution is the value of the feature. By placing the kernel at different positions, the values ​​for that feature at different positions are provided. Given a feature, the values ​​of that feature at different positions are calculated. Features for other texture information besides convolution can be used, such as identifying maximum or minimum values. Other features besides texture information can be used.

[0044] In one embodiment, features are manually designed. One or more features to be used are predetermined based on the programmer's experience or testing. Exemplary features include scaled invariant feature transforms, histograms of oriented gradients, local binary patterns, gray level co-occurrence matrices, Haar wavelets, steerables, or combinations thereof. Feature extraction computes features from ultrasound images to better capture information that distinguishes one or more organs or regions.

[0045] In another embodiment, deep learned features are used. Values ​​are extracted from the scan data for features learned from machine learning. Rather than training a detector only from manually specified features, deep machine learning learns features represented in the training data as well as training the detector. Relevant features are automatically determined as part of the training. This capability allows for generic training on arbitrary data (i.e., training data with known results) for which features such as texture can be internally determined. By training the network with labeled results (i.e., ground truth for detection (e.g., identity or location of an object), the network learns what features are relevant or can be ignored for detection.

[0046] Any deep learning method or architecture can be used. For example, a convolutional neural network is used. The network can include convolutions, subsampling (e.g., max pooling), fully connected layers, and / or other types of layers. By using convolutions, the number of possible features to be tested is limited. The fully connected layers operate to fully connect the features as limited by the convolutional layers after max pooling. Other features can be added to the fully connected layers, such as non-imaging or clinical information. Any combination of layers can be provided.

[0047] With or without deep learning, a machine learning network is trained to associate classification labels (output identification and / or segmentation) with extracted values ​​for one or more features. Machine learning uses training data with ground truth (such as values ​​for features extracted from a data frame for a patient with known objects and / or segmentation) to learn detections based on input feature vectors. The resulting machine learning network is a matrix that is used to input, weight, and combine to output detections. Using one or more matrices, the image processor inputs the values ​​extracted for the features and outputs detections.

[0048] Additional information other than the scan data can be used for detection. For example, values ​​of clinical measurements for the patient are used. The detector is trained to detect based on values ​​extracted for features in the scan data as well as the additional measurements. Genetic data, blood-based diagnosis, family history, gender, weight, and / or other information are input as features.

[0049] In the case of computer-aided detection based on machine learning, self-learning or feedback learning can be used. Once the physician reviews the object of the test and indicates whether the test was correct, the indication and the sample can later be used as training data. This information can be used as further training data to relearn or update the detector with additional ground truth.

[0050] Different computer-assisted detectors may detect different types of objects and / or objects in different situations. Multiple detectors may be applied to each frame. In one embodiment, the detector to be applied is selected based on the type of examination. For example, a user configures an ultrasound scanner for breast examination. One or more detectors for detecting suspicious objects (e.g., tumors and / or cysts) in the breast, for detecting specific breast anatomical structures or landmarks, and / or for detecting the breast are selected and applied. As another example, a user configures an ultrasound scanner for breast examination to detect cancer. One or more detectors are selected for detecting cancer objects in the breast. The selection is automatic by a processor, or the user selects the detector. In another embodiment, the detector detects different anatomical structures so that detection occurs for any area of ​​the patient being scanned. A hierarchy of one detector and / or binary detectors that detect different objects is applied.

[0051] The detector classifies (e.g., identifies) and / or segments one or more objects. For example, one or more machine learning classifiers identify one or more objects represented in the scan data. The classifier classifies an organ or region of the patient from the extracted feature values. The value is input to a machine learning classifier implemented by an image processor. By applying the classifier, the organs or other objects represented in the scan data are classified. As another example, one or more machine learning segmenters locate one or more objects represented in the scan data. The segmenter determines a location from the extracted feature values, such as all tissues belonging to an organ, a boundary, and / or a center. The value is input to a machine learning segmentor implemented by an image processor. Different machine learning detectors or networks are provided for classification and segmentation. Alternatively, one detector is trained to both classify and segment.

[0052] Computer aided detection is applied during acquisition of frames or images (i.e., during scanning and / or ultrasound examination). The application is real-time. The period of processing a frame is equal to or less than the period of acquiring a limited number of new frames (such as only one, two, five, or ten frames). Real-time application allows the application to occur within a second or two of creating a scan. Machine learning detectors can operate faster than template matching or other detection.

[0053] Acts 32 , 34 , and 36 illustrate one exemplary detection of act 30 . Figure 2 Further processing flows for these three exemplary detection actions are shown. In action 32, objects are identified using a machine learning or other computer-assisted classifier, then a machine learning segmenter is selected based on the identification in action 34, and the identified objects are then segmented using the selected machine learning or other computer-assisted segmenter. In other embodiments, only action 32 is performed, only action 36 is performed, or both actions 32 and 36 are performed without action 34.

[0054] In act 32, the ultrasound scanner or image processor identifies one or more objects in the internal region of the patient. The one or more objects are identified by applying a machine learning classifier to the scan data (i.e., inputting values ​​for feature vectors derived from the scan data). In one embodiment, an ImageNet-based classifier is used for identification.

[0055] One or more objects are identified in any number of frames or images. One or more objects are identified during the scan where the computer-assisted characterization is run in real-time with the scan. The machine learning classifier can be trained to classify multiple objects in a region. Alternatively, the machine learning classifier can be trained to identify one specific object. More than one machine learning classifier can be applied to identify multiple objects. One or more machine learning classifiers are selected based on the imaging application or preset for the scan, such as applying a classifier of whether the object is liver based on performing a liver scan.

[0056] Identify an object. Determine a label that identifies an object. For example, determine a type of organ or anatomical landmark (e.g., kidney or carotid artery, or left ventricular apex). As another example, a catheter is identified as a catheter. In yet another example, identify an abnormality. An abnormality can be identified as an abnormality or more specifically (e.g., cyst, tumor, or cancer).

[0057] In act 34, the ultrasound scanner or image processor selects a machine learning segmentor. The machine learning segmentor is trained to a specific type or particular object, such as a kidney segmentor. Alternatively, the segmentor is trained to segment more than one type of object.

[0058] An object-based identification is selected. A classifier determines that the field of view includes one or more specific objects. One or more corresponding segmenters are selected for segmenting the one or more objects. For example, a machine learning segmenter for a catheter, an organ, or an anatomical landmark is selected.

[0059] Selection occurs during an ongoing ultrasound examination. Classification is performed while the patient is being scanned. Similarly, selection occurs within a very short time (a fraction of a second) after classification and while the patient is being scanned. Less rapid selection may occur, such as within a few seconds while other classifications are being performed on other objects.

[0060] The selection of a machine learning segmenter is the selection of one or more matrices. Each segmenter may use a different network architecture. A set of segmenters trained for different objects may be obtained, and one or a subset may be selected based on the classification. Alternatively, the same or similar architecture is used for different segmenters and / or classifiers. Different segmenters and / or classifiers have different weights. The selection is for the weights of the nodes of the network. In one embodiment, one network is well trained for segmenting the heart, and another network is well trained for segmenting one or more organs of the abdomen (i.e., liver, spleen, kidneys). By selecting which weights to deploy to the network or trained networks, the appropriate network is applied for segmentation.

[0061] In action 36, the ultrasound scanner or image processor segments the object in the scan data. The selected machine learning segmenter is applied. In one embodiment, a SegNet-based machine learning segmenter is used. Values ​​for the features are extracted and input to the machine learning segmenter. In the case of using deep learning, the segmenter extracts values ​​for the features.

[0062] The segmenter is applied during an ongoing ultrasound examination. The segmenter is applied and position information is output within a very short time or seconds from the selection of the segmenter. The segmenter is applied while the next frame of scan data is acquired or within the time of acquiring ten or twenty frames. More or less rapid applications can be provided.

[0063] The segmenter may continue to be applied. As each new frame of data is acquired by repeated scanning, the segmenter is applied. Position information is provided for each frame. This continues until there is a large change or pattern of changes in the field of view. Once the field of view is stabilized, the same or a different segmenter is applied to each frame of scan data. In an alternative embodiment, the segmenter is applied to each other or another subset of frames acquired in the ongoing examination.

[0064] The machine learning segmenter indicates the location of the object. The classifier identifies the object in the field of view, and the segmenter determines the location of the object in the field of view. The location is a boundary, a binary mask (i.e., each pixel, voxel, or sample point belonging to the object), a center, a spatial extent (e.g., a fitted approximation or model shape), or other representation of spatial position, orientation, and / or scale. The machine learning segmenter determines the location of the object based on input values ​​for a feature vector, such values ​​being extracted from scan data for one or more frames.

[0065] In action 38, the ultrasound scanner or image processor implements the next action in the workflow based on the segmentation and / or classification of the object. The implementation does not require user input between detection, classification and / or segmentation and implementation. The operation of the ultrasound scanner is changed to increment along the order of the workflow without user control of the change. The user or image processor can select the workflow, such as based on classification. During at least one of the steps of the workflow, the ultrasound scanner performs all or part of the steps without user input activation or other information to perform the part. For example, an ROI or measurement indicator is placed on the image, and an ROI scan or measurement is performed after segmentation without the user having to place the ROI or measurement indicator and / or the user having to activate the ROI scan or measurement. Once the anatomical structure has been successfully identified and / or segmented, the identity or location information is used to improve the imaging workflow for the user by avoiding at least one user interaction with the ultrasound scanner. This allows the user to continue to focus on the scan, the visual interpretation of the scanned results, and the monitoring examination.

[0066] Any number of steps of the workflow may be triggered by and / or use identity and / or location information. Any given step that has sufficient information to be triggered and / or executed occurs without user input. The next one or more actions of the workflow are implemented during the ongoing ultrasound examination. An action to be taken in a series of actions for an ongoing ultrasound examination is automatically executed.

[0067] The change of the operation of the ultrasound scanner to implement the steps or actions in the workflow is based on the identification and / or position of the object. The next action may be to scan the object more intensively, so both the identification and the position are used. The ultrasound scanner is configured to scan the object. For example, the Doppler gate is positioned based on the identification and the position (for example, the position in the regurgitant flow area of ​​the heart). When entering the Doppler or spectral imaging mode, the gate is automatically positioned in the blood vessel, and / or the gate is positioned, and the Doppler or spectral imaging mode is started without the user having to position and / or start. As another example, the imaging or scanning preset is changed. The user may have initially selected a preset that configures the ultrasound scanner for scanning, such as selecting liver imaging. The identification and / or position can be used to select a more refined set of presets suitable for the identified object and / or position, such as selecting a preset for imaging a liver with a specific shape and / or position relative to other organs. As another example, the user may have selected a preset for imaging the cardiac system, and then the image processor changes the preset to be used for imaging a specific identified artery, ventricle and / or valve. In yet another example, the position information indicates the orientation of the blood vessel or fluid region. Changing the scan configuration for flow imaging to steer the beam more in the direction of the flow and / or including velocity corrections to account for the angle of the scan line from the transducer to the direction of the flow. The user may be prevented from having to perform these actions during scanning due to the identification and / or position determination and automatic incrementation of the workflow.

[0068] The next action may be to store an image or frame of the scanned data. For example, identification is used to trigger the storage of a representative image of the object of interest. As another example, locating the object after identification can be used to trigger the storage of an image of the object annotated with segmentation, identification, and / or measurement information. The previously entered patient information is used to automatically store one or more images in the patient's medical record. Automatic image storage occurs without the user having to trigger and / or enter additional information, saving the user time and effort.

[0069] The next action may be to measure. Based on the identification of the object, the associated one or more measurements are loaded or implemented. By preloading, it is possible to avoid the user having to select which measurements to perform. By implementing, it is possible to avoid the user having to activate. Rather than the user having to place the caliper or trace the boundary and / or having to activate the measurement, the ultrasound scanner uses segmentation to place the caliper at the given landmark or as a track and activate the measurement. Figure 4 and Figure 5 An example is shown in which the kidneys are identified and a maximum length measurement is automatically performed based on segmentation of the identified kidneys. Figure 4 and Figure 5 An indication of the measurement is shown as automatic placement of the calipers and a line between the calipers. The measurement of the identified object is performed based on the identification and / or location without requiring a user to control the measurement step in the workflow.

[0070] The next action may be annotation of the image. Any annotation may be included in the workflow, such as marking the image with an identification of an anatomical structure, showing segmentation (e.g., highlighting an object with a color or highlighting different objects with different colors, extracting an object, or showing boundary trajectories), and / or indicating the value of a measurement. Annotations are alphanumeric text and / or other graphics. Annotations are added to the displayed image based on identification and / or location without the user having to add the annotation. Annotations may be positioned based on the location of the object.

[0071] The next step may be identification and / or color flow scanning of the vessel or fluid region boundaries. The location information provides the boundaries, allowing for color flow scanning with flow information being shown for locations within the boundaries and B-mode being shown for other locations. Rather than the user having to identify and track the vessel or fluid region and then activate color flow imaging, the ultrasound scanner uses identification and / or segmentation to establish a flow display volume (containment) relative to the B-mode location and activate flow scanning.

[0072] The next action may be to place the ROI. The identification indicates that the object of interest is in the field of view. The position information indicates the location of the object in or around which the ROI will be placed. For example, the boundary of the object is used. The ROI is placed to cover the boundary with any tolerance outside of that boundary. The location, boundary and / or size of the object from the segmentation is used to position the ROI. Figure 4 An example of a ROI is shown as a box placed around or enclosing the entire kidney. Rather than the user having to identify the kidney and interact with a user interface to place the ROI, the ultrasound scanner enables placement of the ROI, such as a color region of interest. ROIs are used for different types of imaging or scanning, such as color flow. Given the scanner places the ROI, a ROI-based scan may also be started without user initiation. Color Doppler imaging automatically starts with the ROI located at the anatomy of interest.

[0073] Continuous real-time segmentation may additionally provide tracking information to allow the ROI position to be updated as the probe or patient moves. Based on repeated segmentation of subsequent scan data frames, the ROI or other position (eg, Doppler gate position) is maintained on anatomical landmarks.

[0074] The next action might be to change the depth of the scan. The frequency of the transmitted waveform can also be changed with depth. Higher frequencies decay faster but provide better resolution. By matching frequency to depth, a more optimized scan is provided. The identification of an object (e.g., a diaphragm) and the location of that object can be used to establish the depth. The depth of field of view is changed as a step in the workflow to refine the inspection. Figure 5An example is shown where ultrasound examination is used to study the kidney. The diaphragm (usually designated by an ellipse) is identified and located. The depth of the scan is reduced to avoid scanning deeper than the diaphragm, resulting in more detail for imaging the kidney. In other embodiments, the deepest extent of the kidney is used to establish the depth. Rather than the user having to make the depth change, the depth change occurs automatically.

[0075] The next action may be associated with the identification of the abnormality (i.e., identifying the pathology). For example, a classifier identifies a lesion or tumor during a liver, kidney, or breast exam. The workflow includes a branch of steps to be performed upon detection of an abnormality. Implementation of the branch of steps is triggered based on the detection of the classifier. The steps may include placement of an ROI or magnification box (e.g., Res box) around the pathology, measurement of the pathology (e.g., size, shape, flow, and / or elasticity), annotation of the pathology, storage of an image showing the pathology, or other steps. The ultrasound scanner performs one or more of the steps based on the identification and / or location of the pathology without the user having to control the steps.

[0076] Portions of the steps may be automated, such as showing a list of measurements that may be performed on a user interface for user selection. The user does not have to activate display of the list. Varying modes may be used, such as once the field of view is stable at a pathology or object, measurements, annotations, and image storage are sequentially performed without user input. Other step orders may be provided, such as automatically advancing the quantification workflow to the next task based on completion of image storage.

[0077] exist Figure 1 In action 40, an ultrasound scanner or image processor generates an image of the object. Imaging may correspond to scanning, such that each acquired frame of scan data is used to generate an image. Images for only one or less than all frames may be generated from the scan.

[0078] The generated image may be responsive to the execution or initiation of the next action to be automatically implemented. Since the user is not implementing a step on the workflow, a visual indication is provided to the user that the ultrasound scanner is implementing the action. For example, annotations, measurement locations, measurements, or ROIs are included in the image. As another example, a confirmation of saving to the patient's medical record or image storage is included. In yet another example, a value for a setting (e.g., a preset value, depth of field, or scan setting) is displayed. A notification of the change without details may be provided on the image or user interface.

[0079] The execution of the next step in the workflow is shown by text or other graphics. Lines, boxes or other graphics indicate the execution or results of execution of one or more steps in the workflow. A graphic representing the workflow with the current step highlighted and / or completed steps distinguished from unfinished steps can be displayed.

[0080] Identification and / or location information may be included on the image. For example, color shading is added. Different objects are colored differently, or a given object is colored (where other objects are not colored). Markers, lines, boxes, or other graphics may be used to illustrate the segmentation. Annotations for identification may be used.

[0081] Due to automation of the workflow based on classification and segmentation, one or more steps in the workflow may be performed more quickly. Delays for a user to log in or configure the ultrasound scanner are reduced or avoided. The user may confirm correct placement, measurement, or other information as part of the step, but the user's initial placement or log in for measurement may be avoided. Greater throughput per sonographer may result.

[0082] Figure 6 One embodiment of a system for machine-assisted workflow assistance in ultrasound imaging is shown. The system is an ultrasound scanner 10 that applies computer-assisted or machine learning classification and segmentation. The ultrasound scanner 10 is configured by a machine learning detector to identify objects and / or segment objects. Steps of a workflow for ultrasound examinations are automatically implemented based on the classification and / or segmentation. Classification, segmentation, and workflow implementation occur in real time with scanning and / or during ultrasound examinations.

[0083] The ultrasound scanner 10 is a medical diagnostic ultrasound imaging system. The ultrasound scanner 10 is configured to scan an area of ​​a patient with ultrasound using a transducer 14. The scan is repeated during an examination of the patient. In an alternative embodiment, the ultrasound scanner 10 includes a personal computer, a workstation, a PACS station, or other devices in the same location or distributed on a network for real-time imaging from scan data acquired from the patient through a connection with the beamformers 12, 16 and the transducer 14.

[0084] Ultrasound Scanner 10 Implementation Figure 1 methods, Figure 2 The ultrasound scanner 10 includes a transmit beamformer 12, a transducer 14, a receive beamformer 16, a detector 18, a display 20, a memory 22, and an image processor 24. Additional, different, or fewer components may be provided. For example, user input is provided for adjusting automatic placement and / or for configuring the ultrasound system 10 for user-controlled workflow steps.

[0085] The transmit beamformer 12 is an ultrasound transmitter, a memory, a pulse generator, an analog circuit, a digital circuit or a combination thereof. The transmit beamformer 12 is configured to generate waveforms with different or relative amplitudes, delays and / or phasing for multiple channels. A waveform is generated and applied to a transducer array with any timing or pulse repetition frequency. For example, the transmit beamformer 12 generates a pulse sequence for B-mode scanning in a linear, fan-shaped or Vector® format. As another example, the transmit beamformer 12 generates a pulse sequence for color flow scanning, such as a pulse for forming 2-12 beams for an ROI within the B-mode field of view in the ongoing flow sample count of each scan line. In yet another example, the transmit beamformer 12 generates pulses for elastic or shear imaging. The transmit beamformer 12 can generate a beam for acoustic radiation force impulses. The intensity of the beam causes shear waves or longitudinal waves to be generated from the focus. The transmit beamformer 12 then generates a beam for tracking the response of tissue to the generated waves.

[0086] The transmit beamformer 12 is connected to the transducer 14, such as by a transmit / receive switch. One or more beams are formed during a given transmit event as acoustic waves are transmitted from the transducer 14 in response to the generated waves. The beams are used for B-mode, color flow mode, elastic, shear wave and / or other imaging modes. A series of transmit beams are generated to scan a one-dimensional, two-dimensional or three-dimensional area. Sector, Vector®, linear or other scanning formats may be used.

[0087] The transducer 14 is a 1-dimensional, 1.25-dimensional, 1.5-dimensional, 1.75-dimensional or 2-dimensional array of piezoelectric or capacitive film elements. The transducer 14 includes a plurality of elements for transducing between acoustic energy and electrical energy. For example, the transducer 14 is a one-dimensional PZT array having approximately 64-256 elements.

[0088] The transducer 14 is connected to the transmit beamformer 12 for converting an electrical waveform into an acoustic waveform and to the receive beamformer 16 for converting acoustic echoes into electrical signals. To scan with ultrasound, the transducer 14 transmits acoustic energy and receives echoes. In response to the ultrasonic energy (echoes) impinging on the elements of the transducer 14, a receive signal is generated.

[0089] The receive beamformer 16 includes: a plurality of channels with amplifiers, delays and / or phase rotators, and one or more summers. Each channel is connected to one or more transducer elements. The receive beamformer 16 applies relative delay, phase and / or apodization in response to each transmission for imaging to form one or more receive beams. Dynamic focus reception can be provided. The relative delay and / or phasing and summing of the signals from different elements provide beamforming. The receive beamformer 16 outputs data representing the spatial position using the received acoustic signal. In an alternative embodiment, the receive beamformer 16 is a processor for generating samples using Fourier or other transforms.

[0090] The receive beamformer 16 may include filters, such as filters for isolating information at the second harmonic, transmit (i.e., fundamental), or other frequency bands relative to the transmit frequency band. Such information may be more likely to include desired tissue, contrast agent, and / or flow information. In another embodiment, the receive beamformer 16 includes a memory or buffer and a filter or adder. Two or more receive beams are combined to isolate information at a desired frequency band (such as the second harmonic, cubic fundamental, or another frequency band).

[0091] The receive beamformer 16 outputs beam summation data representing spatial or sample positions. Data is output for a single position, a position along a line, a position for a region, or a position for a volume. The data beamformed in response to a complete scan of the region is a frame of data.

[0092] The detector 18 is a B-mode detector, a Doppler detector, a pulsed wave Doppler detector, a correlation processor, a Fourier transform processor, a filter, other processors now known or later developed for implementing ultrasound imaging modes, or a combination thereof. The detector 18 provides detection of imaging modes, such as including a Doppler detector (e.g., an estimator) and a B-mode detector.

[0093] Other post-beamforming components may be provided. Spatial filters, temporal filters, and / or scan converters may be included. Detector 18 or other components output display values, such as detections, mapping detection values ​​to display values, and formatting display values ​​or detection values ​​into a display format.

[0094] The image processor 24 is a control processor, a general purpose processor, a digital signal processor, a graphics processing unit, an application specific integrated circuit, a field programmable gate array, a network, a server, a processor group, a combination thereof, or other devices now known or later developed for detecting objects in the image and controlling the ultrasound system 10 to assist in the examination workflow. The image processor 24 is separate from the detector 18 or is part of the detector 18. As a separate device, the image processor 24 requests, receives, accesses or loads data for detection and control at any processing stage (e.g., beamforming, detection, scan conversion, display mapping or image processing). The image processor 24 is configured by software, firmware and / or hardware to perform or cause to be performed Figure 1 or Figure 2 action.

[0095] The image processor 24 is configured to monitor the scan data over time for changes in the field of view. Using correlation or other methods, the amount of change or the pattern of change is used to determine whether to apply segmentation based on previous object identification or to apply a new segmenter selection and classification. The image processor 24 is configured to classify by applying computer-assisted classification in response to changes above a threshold or the pattern of change defined by one or more thresholds.

[0096] The image processor 24 is configured to apply computer-assisted classification of one or more objects of the region from the scanned scan data. For example, the machine learning classifier is configured to identify the one or more objects as organs and / or abnormalities. The image processor 24 is configured to apply computer-assisted segmentation of the one or more objects based on the classification. The segmenter is selected based on the identification of the objects. The segmenter can be a machine learning segmenter that is configured to locate the one or more organs and / or abnormalities.

[0097] The image processor 24 is configured to alter the operation of the ultrasound scanner based on the classification and / or segmentation of one or more objects. The alteration is to implement the next one or more steps in the workflow for the examination of the patient. The workflow is a series of steps in a decision tree from the initial scan to the completion of the examination of the patient with ultrasound. Based on the object identification and / or segmentation performed in real time with the scan, the next one or more steps in the decision tree can be implemented by the ultrasound scanner 10 with no or little user interaction (also in real time). For example, the ultrasound scanner places an area of ​​interest based on and in response to the classification and segmentation, changes the depth used for scanning, and / or measures one or more objects.

[0098] The image processor 24 or detector 18 generates an image or value and outputs the image or value to the display 20. For example, a B-mode or mixed mode (e.g., B-mode and flow mode) image is output. Text, numerical indications, or graphics can be added and displayed to the user. A chart can be displayed. For example, annotations marking detected objects are output on the image, flags indicating the image as including detected objects, derived values ​​of measurements, ROIs, or other object related information. The image includes an indication of the execution or completion of execution of one or more steps in the workflow. Other indications, such as flashing buttons, can be provided.

[0099] The display 20 is a CRT, LCD, monitor, plasma, projector, printer or other device for displaying an image or a sequence of images. Any display 20 now known or later developed may be used. Each image is a scanned region of a patient. Other information is output on, with or as part of the image, such as annotations (e.g., graphics) showing changes and / or indications of steps in a workflow (e.g., ROI placement indicated by an ROI added to an image). The display 20 is operable to display one image or a series of images. The display 20 displays a two-dimensional image or a three-dimensional representation.

[0100] The detector 18, image processor 24, receive beamformer 16, and transmit beamformer 12 operate according to instructions stored in memory 22 or another memory. The instructions configure the system for Figure 1 The instructions configure the detector 18, the image processor 24, the receive beamformer 16 and / or the transmit beamformer 12 for operation by being loaded into the controller, by causing a table of values ​​to be loaded and / or by being executed.

[0101] The memory 22 is a non-temporary computer-readable storage medium. Instructions for implementing the processes, methods and / or techniques discussed herein are provided on a computer-readable storage medium or memory (such as a cache, buffer, RAM, removable media, hard drive or other computer-readable storage medium). Computer-readable storage media include various types of volatile and non-volatile storage media. In response to one or more groups of instructions stored in or on a computer-readable storage medium, the functions, actions or tasks illustrated in the accompanying drawings or described herein are executed. Functions, actions or tasks are independent of a specific type of instruction set, storage medium, processor or processing strategy, and can be performed by software, hardware, integrated circuits, firmware, microcode, etc. operating alone or in combination. Similarly, processing strategies may include multi-processing, multi-tasking, parallel processing, etc. In one embodiment, instructions are stored on a removable media device for reading by a local or remote system. In other embodiments, instructions are stored in a remote location for transmission over a computer network or over a telephone line. In yet another embodiment, instructions are stored in a given computer, CPU, GPU or system.

[0102] Although the invention has been described above by reference to various embodiments, it should be understood that many changes and modifications may be made without departing from the scope of the invention. Therefore, it is intended that the foregoing detailed description be considered illustrative rather than restrictive, and it should be understood that it is the following claims including all equivalents that are intended to define the spirit and scope of the invention.

Claims

1. A system for machine-assisted workflow assistance for an ultrasound scanner, the system comprising: an ultrasound scanner configured to scan an internal region of a patient with an ultrasound transducer of said ultrasound scanner, the scanning being repeated during an ongoing ultrasound examination; The ultrasound scanner includes an image processor configured to: identifying objects in the interior region by a machine learning classifier applied to scan data from a scan during an ongoing ultrasound examination; selecting a machine learning segmentor based on the identity of the object, the selection occurring during an ongoing ultrasound examination; and segmenting objects in scan data during an ongoing ultrasound examination by a selected machine learning segmentor; wherein the ultrasound scanner is further configured to implement a next action in the workflow based on the segmentation of the object, the next action being implemented during the ongoing ultrasound examination and without user input to the ultrasound scanner between the segmentation and the implementing; and Wherein the ultrasound scanner is further configured to generate an image of the object.

2. The system of claim 1, wherein scanning comprises: The scanning is performed continuously with identification, selection, segmentation and implementation occurring in real time along with the continuous scanning.

3. The system of claim 1, wherein the identification comprises: An organ or an inserted device in the internal region is identified, and wherein the selecting comprises: selecting a machine learning segmentor trained for the organ or the inserted device.

4. The system of claim 1, wherein the identification comprises: An anomaly in the inner region is identified, and wherein implementing includes: implementing a next action based on the anomaly.

5. The system of claim 1 , wherein selecting comprises: Choose to configure the weights of the machine learning segmenter for the identified objects.

6. The system of claim 1, wherein segmenting comprises: Locating a boundary of the object, and wherein implementing comprises: implementing according to the boundary.

7. The system of claim 1, wherein implementing comprises: The next action is implemented as configuring the ultrasound scanner to scan the object, as storing the image in the patient's medical record, as establishing a region of interest around or within the object, and / or as making arrangements on a user interface of the ultrasound scanner for measurement of the object.

8. The system of claim 1, wherein implementing comprises: The next action in the workflow is implemented as the action taken in the series of actions for the ongoing ultrasound examination, which is being triggered by the segmentation.

9. The system of claim 1 , wherein scanning comprises B-mode scanning, and wherein implementing comprises: The next action is implemented as setting the color region of interest based on the identity of the object, the location of the object from the segmentation, and the size of the object from the segmentation.

10. The system of claim 1, wherein implementing comprises: Based on the identification of the object and the location of the object from the segmentation, the next action is implemented as a change in the scanning depth of the scan.

11. The system of claim 1 , wherein generating an image comprises: Generate an image in response to the execution of the next action.

12. The system of claim 1 , wherein generating an image comprises: An image is generated to graphically illustrate the execution of the next action.

13. A system according to claim 1, wherein the image processor is further configured to: detect the level of change between scan data from different times, and perform identification, selection and segmentation in response to the level of change being higher than a threshold, and perform segmentation without repeating identification and selection in response to the level of change being lower than a threshold.

14. A system for machine-assisted workflow assistance for an ultrasound scanner, the system comprising: An ultrasound scanner, configured to: scanning a patient as part of a workflow for ultrasonographically examining a patient, the workflow comprising a series of actions; and The ultrasound scanner includes an image processor configured to: Monitor changes in the scanned field of view; In response to the change, detecting an object from the scan with a machine learning network, the detecting comprising identifying the object and / or locating the object, wherein the detecting comprises: identifying the object with a machine learning classifier, and then segmenting the identified object with a machine learning segmenter selected based on the identification; as well as The operation of the ultrasound scanner is altered incrementally along the sequence of the workflow without user control, the alteration being based on the identification and / or location of the object.

15. The system of claim 14, wherein monitoring comprises: A change is identified as being above a threshold, and wherein the change comprises storing the image with an annotation of the identification and location, changing a depth of field of view, placing a region of interest, and / or placing a gate.

Images