Dynamic ultrasound recommendations

Abstract

An ultrasound system includes a memory that stores instructions; and a processor that executes the instructions. The instructions cause the ultrasound system to implement a process. The process includes receiving static information relating to an ultrasound procedure and dynamic information dynamically obtained from performing the ultrasound procedure. The static information and the dynamic information are input to a trained machine learning model during the ultrasound procedure. The trained machine learning model generates a recommendation for performing the ultrasound procedure based on the static information and the dynamic information. The ultrasound system overlays a marker for the recommendation on an ultrasound image to recommend an adjustment to the ultrasound procedure.

Description

DYNAMIC UUTRASOUND RECOMMENDATIONSBACKGROUND

[0001] Ultrasound imaging is increasingly being performed by low-cost, ultra-mobile ultrasound devices in addition to ultrasound imaging devices with ultrasound carts. The growing capabilities of cart-based ultrasound systems and the advent of the ultra-mobile ultrasound devices places increased training burden on user groups. In many instances, ultrasound users are not adequately trained to administer an ultrasound examination in full compliance with device capabilities, guidelines, state of the art knowledge and understanding, and stakeholder preferences.

[0002] An example of capabilities of cart-based ultrasound systems is the availability of a new transducer that would enable or require a new scan technique and thus adaptation of the scan protocol by ultrasound users. While manufacturers may accompany the installation of a new feature with user training, adoption of lesser-used features remains difficult. The same is true with respect to adoption of new guidelines and advances in the state of the art since new guidelines and advances in the state of the arts are communicated in workshops and conferences and therefore difficult to inform everyday routine. With respect to stakeholder preferences, clinics and reading / referring physicians may have their own preferences that are hard to document and track.

[0003] Ultrasound examinations are typically dynamic in that a sonographer may dynamically determine which ultrasound images to capture next and for how long during the ultrasound examination. The variations may be based on findings and insights during the ultrasound examination, and the ultrasound examination is only complete when the sonographer deems the ultrasound images and measurements to be complete.

[0004] Currently there is no software assistance program for guiding execution of an ultrasound examination, such as which techniques to apply and what views to capture in the report in the contexts of given diagnostic questions, applicable guidelines, patient characteristics and physician preferences.SUMMARY

[0005] According to an aspect of the present disclosure, an ultrasound system includes a memorythat stores instructions; and a processor that executes the instructions. When executed by the processor, the instructions cause the ultrasound system to: receive static information relating to an ultrasound procedure and dynamic information dynamically obtained from performing the ultrasound procedure; input the static information and the dynamic information to a trained machine learning model; generate, by the trained machine learning model during the ultrasound procedure, a recommendation for performing the ultrasound procedure based on the static information and the dynamic information; and overlay a marker for the recommendation on an ultrasound image to recommend an adjustment to the ultrasound procedure.

[0006] According to another aspect of the present disclosure, a method for ultrasound imaging, comprising includes receiving, by a controller comprising a memory that stores instructions and a processor that executes the instructions, static information relating to an ultrasound procedure and dynamic information dynamically obtained from performing the ultrasound procedure; inputting the static information and the dynamic information to a trained machine learning model; generating, by the trained machine learning model during the ultrasound procedure, a recommendation for performing the ultrasound procedure based on the static information and the dynamic information; and overlaying a marker for the recommendation on an ultrasound image to recommend an adjustment to the ultrasound procedure.

[0007] According to another aspect of the present disclosure, a tangible non-transitory computer- readable storage medium stores a computer program. When executed by a processor, the computer program causes a system to: receive static information relating to an ultrasound procedure and dynamic information dynamically obtained from performing the ultrasound procedure; input the static information and the dynamic information to a trained machine learning model; generate, by the trained machine learning model during the ultrasound procedure, a recommendation for performing the ultrasound procedure based on the static information and the dynamic information; and overlay a marker for the recommendation on an ultrasound image to recommend an adjustment to the ultrasound procedure.BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The example embodiments are best understood from the following detailed description when read with the accompanying drawing figures. It is emphasized that the various features arenot necessarily drawn to scale. In fact, the dimensions may be arbitrarily increased or decreased for clarity of discussion. Wherever applicable and practical, like reference numerals refer to like elements.

[0009] FIG. 1 illustrates a system for dynamic ultrasound recommendations, in accordance with a representative embodiment.

[0010] FIG. 2 illustrates another system for dynamic ultrasound recommendations, in accordance with a representative embodiment.

[0011] FIG. 3 illustrates a method for dynamic ultrasound recommendations, in accordance with a representative embodiment.

[0012] FIG. 4 illustrates inputs and outputs for dynamic ultrasound recommendations, in accordance with a representative embodiment.

[0013] FIG. 5 illustrates a dynamic ultrasound recommendation, in accordance with a representative embodiment.

[0014] FIG. 6 illustrates another dynamic ultrasound recommendation, in accordance with a representative embodiment.

[0015] FIG. 7 illustrates a user interface for dynamic ultrasound recommendation, in accordance with a representative embodiment.

[0016] FIG. 8 illustrates another user interface for dynamic ultrasound recommendation, in accordance with a representative embodiment.

[0017] FIG. 9 illustrates another user interface for dynamic ultrasound recommendation, in accordance with a representative embodiment.

[0018] FIG. 10 illustrates another user interface for dynamic ultrasound recommendation, in accordance with a representative embodiment.

[0019] FIG. 11 illustrates a computer system, on which a method for dynamic ultrasound recommendations is implemented, in accordance with another representative embodiment.DETAILED DESCRIPTION

[0020] In the following detailed description, for the purposes of explanation and not limitation, representative embodiments disclosing specific details are set forth in order to provide a thorough understanding of embodiments according to the present teachings. However, otherembodiments consistent with the present disclosure that depart from specific details disclosed herein remain within the scope of the appended claims. Descriptions of known systems, devices, materials, methods of operation and methods of manufacture may be omitted so as to avoid obscuring the description of the representative embodiments. Nonetheless, systems, devices, materials and methods that are within the purview of one of ordinary skill in the art are within the scope of the present teachings and may be used in accordance with the representative embodiments. It is to be understood that the terminology used herein is for purposes of describing particular embodiments only and is not intended to be limiting. Definitions and explanations for terms herein are in addition to the technical and scientific meanings of the terms as commonly understood and accepted in the technical field of the present teachings.

[0021] It will be understood that, although the terms first, second, third etc. may be used herein to describe various elements or components, these elements or components should not be limited by these terms. These terms are only used to distinguish one element or component from another element or component. Thus, a first element or component discussed below could be termed a second element or component without departing from the teachings of the inventive concept.

[0022] As used in the specification and appended claims, the singular forms of terms ‘a’, ‘an’ and ‘the’ are intended to include both singular and plural forms, unless the context clearly dictates otherwise. Additionally, the terms "comprises", and / or "comprising," and / or similar terms when used in this specification, specify the presence of stated features, elements, and / or components, but do not preclude the presence or addition of one or more other features, elements, components, and / or groups thereof. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0023] Unless otherwise noted, when an element or component is said to be “connected to”, “coupled to”, or “adjacent to” another element or component, it will be understood that the element or component can be directly connected or coupled to the other element or component, or intervening elements or components may be present. That is, these and similar terms encompass cases where one or more intermediate elements or components may be employed to connect two elements or components. However, when an element or component is said to be “directly connected” to another element or component, this encompasses only cases where the two elements or components are connected to each other without any intermediate or interveningelements or components.

[0024] The present disclosure, through one or more of its various aspects, embodiments and / or specific features or sub-components, is thus intended to bring out one or more of the advantages as specifically noted below.

[0025] As described herein, ultrasound devices, including ultra-mobile ultrasound devices, may be used to receive guidance before, during and after ultrasound examinations, to assist in reaching adequate diagnostic relevance. Dynamic ultrasound recommendations may be used by both highly trained professional sonographers as well as ultrasound users from other specialties. The teachings herein may also be applicable to both relatively simple ultrasound examinations as well as complex ultrasound examinations. Guidance may be provided by a user interface specific to ultrasound examinations, and may be used to guide capturing of additional ultrasound images or measurements based on the capture history of the current ultrasound examination and the capture history of previous, similar ultrasound examinations.

[0026] FIG. 1 illustrates a system 100 for dynamic ultrasound recommendations, in accordance with a representative embodiment.

[0027] The system 100 in FIG. 1 is a system for dynamic ultrasound recommendations and includes components that may be provided together or that may be distributed. The system 100 includes an ultrasound probe 110, an ultrasound base 120, and a display 180.

[0028] The ultrasound probe 110 includes a processing circuit 115 and a transducer array 113. The processing circuit 115 may comprise a memory for storing data and instructions, an application-specific integrated circuit (ASIC) and / or a processor for processing data and instructions. The transducer array 113 includes an array of transducer elements including at least a first transducer element 1131, a second transducer element 1132, and an Xth transducer element 113X. The transducer array 113 converts electrical energy into sound waves which bounce off of body tissue, and receives echoes of the sound waves and converts the echoes into electrical energy. The transducer array 113 may include dozens, hundreds or thousands of individual transducer elements. The ultrasound probe 110 may transmit a beam to produce images and may detect the echoes. The processing circuit 115 may process ultrasound images captured by the transducer array 113 of the ultrasound probe 110.

[0029] The ultrasound base 120 may comprise an ultrasound cart, and the memory and theprocessor may be implemented in the ultrasound cart. The ultrasound base 120 includes a first interface 121, a second interface 122, a third interface 123, and a controller 150. A computer that can be used to implement the ultrasound base 120 is depicted in FIG. 11, though an ultrasound base 120 may include more or fewer elements than depicted in FIG. 1 or FIG. 11. One or more of the interfaces may include ports, disk drives, wireless antennas, or other types of receiver circuitry that connect the controller 150 to other electronic elements. The first interface 121 connects the ultrasound base 120 to the ultrasound probe 110, and may comprise a port, an antenna, and / or another type of physical component for wired or wireless communications. The second interface 122 connects the ultrasound base 120 to the display 180, and may also comprise a port, an antenna, and / or another type of physical component for wired or wireless communications. The third interface 123 is a user interface, and may comprise buttons, keys, a mouse, a microphone, a speaker, switches, a touchscreen or other type of display separate from the display 180, and / or other types of physical components that allow a user to interact with the ultrasound base 120 such as to enter instructions and receive output.

[0030] The controller 150 includes at least a memory 151 that stores instructions and a processor 152 that executes the instructions. The instructions stored in the memory 151 may comprise a software program including instructions for implementing a model such as an artificial intelligence model, as well as the same or a different software program for generating a user interface on an ultrasound device. In the system 100, the user interface may be generated by the instructions stored in the memory 151 and may be displayed on the display 180. The software program(s) in the memory 151 generate recommendations to be displayed on the display 180 for further ultrasound scans and for which images to capture and which measurements to take.

[0031] The display 180 may be local to the ultrasound base 120 or may be remotely connected to the ultrasound base 120, such as wirelessly. The display 180 includes or otherwise provides a graphical user interface 181 which is configured to display various of the user interfaces described herein, for example with respect to FIG. 5, FIG. 6, FIG. 7, FIG. 8, FIG. 9 and FIG. 10. The display 180 may be connected to the ultrasound base 120 via a local wired interface such as an Ethernet cable or via a local wireless interface such as a Wi-Fi connection. The display 180 may be interfaced with other user input devices by which users can input instructions, including mouses, keyboards, thumbwheels and so on. The display 180 may be a monitor such as acomputer monitor, a display on a mobile device, an augmented reality display, a television, an electronic whiteboard, or another screen configured to display electronic imagery. The display 180 may also include one or more input interface(s) such as those noted above that may connect to other elements or components, as well as an interactive touch screen configured to display prompts to users and collect touch input from users.

[0032] The controller 150 may perform some of the operations described herein directly and may implement other operations described herein indirectly. For example, the controller 150 may indirectly control operations such as by generating and transmitting content to be displayed on the display 180. The controller 150 may directly control other operations such as logical operations performed by the processor 152 executing instructions from the memory 151 based on input received from electronic elements and / or users via the interfaces. Accordingly, the processes implemented by the controller 150 when the processor 152 executes instructions from the memory 151 may include steps not directly performed by the controller 150.

[0033] FIG. 2 illustrates another system for dynamic ultrasound recommendations, in accordance with a representative embodiment.

[0034] In FIG. 2, an ultrasound probe 210 and a smartphone A and a smartphone B are connected over a network 201.

[0035] The network 201 may comprise a local wireless network such as a WiFi network, though the network 201 may also or alternatively include wired elements such as wires connected to smartphone A and smartphone B via USB cables.

[0036] The ultrasound probe 210 may comprise a portable transducer. The ultrasound probe 210 includes a transducer array 213, a lens 214, a user interface 223, a controller 250 and a wireless communication circuit 290. The transducer array 213 includes at least a first transducer element 2131, a second transducer element 2132, and an Xth transducer element 213X. The transducer array 213 converts electrical energy into sound waves which bounce off of body tissue, and receives echoes of the sound waves and converts the echoes into electrical energy. The transducer array 213 may include dozens, hundreds or thousands of individual transducer elements. The ultrasound probe 210 may transmit a beam to produce images and may detect the echoes. The processor 252 may process ultrasound images captured by the transducer array 213 of the ultrasound probe 210. The lens 214 may be used to transmit the ultrasound beams and toreceive echoes of ultrasound beams. The user interface 223 may be used by a user to interact with the ultrasound probe 210. The wireless communication circuit 290 may be used to communicate with smartphone A and smartphone B via the network 201. The ultrasound probe may be configured to link to an external device such as the smartphone A and smartphone B via applications installed on the external device(s).

[0037] Smartphone A stores and executes an ultrasound application 299A. Smartphone B stores and executes an ultrasound application 299B. The ultrasound application 299A and the ultrasound application 299B may be configured to enable smartphone A and smartphone B to interact with the ultrasound probe 210 via the network 201. For example, ultrasound application 299A and ultrasound application 299B may be configured to enable displays of ultrasound images from the ultrasound probe 210. Ultrasound application 299 A and ultrasound application 299B may also be configured to overlay, or generate and overlay, markers on the displays of ultrasound images from the ultrasound probe 210. In some embodiments, determinations of whether recommendations are to be made may be performed by the ultrasound application 299A and the ultrasound application 299B. For example, the ultrasound application 299A and the ultrasound application 299B may each comprise a model that can be applied to static information and dynamic information as described below.

[0038] The controller 250 includes at least a memory 251 that stores instructions and a processor 252 that executes the instructions. The instructions stored in the memory 251 may comprise a software program with a model such as an artificial intelligence model, as well as the same or a different software program for generating a user interface on an ultrasound device. In FIG. 2, the user interface may be generated by the instructions stored in the memory 251 and may be displayed on a display of the smartphone A or smartphone B. The software program(s) in the memory 251 generate recommendations to be displayed on displays of the smartphone A and / or the smartphone B for further ultrasound scans and which images to capture and which measurements to take.

[0039] The controller 250 may perform some of the operations described herein directly and may implement other operations described herein indirectly. For example, the controller 250 may indirectly control operations such as by generating and transmitting content to be displayed on a display of the smartphone A or smartphone B. The controller 250 may directly control otheroperations such as logical operations performed by the processor 252 executing instructions from the memory 251 based on input received from electronic elements and / or users via the interfaces. Accordingly, the processes implemented by the controller 250 when the processor 252 executes instructions from the memory 251 may include steps not directly performed by the controller 250.

[0040] FIG. 3 illustrates a method for dynamic ultrasound recommendations, in accordance with a representative embodiment.

[0041] The method of FIG. 3 may be performed by the system 100 in FIG. 1 or by the ultrasound probe 210 in FIG. 2. In some embodiments based on FIG. 2 and FIG. 3, the method of FIG. 3 may be performed by a combination of the ultrasound probe 210 and one or both of the smartphone A and the smartphone B in FIG. 2.

[0042] At S310, the method of FIG. 3 includes obtaining static information. The static information is information relating to an ultrasound procedure, such as a type of diagnosis, the type of exam prescribed by the physician, the type of transducer, demographic information of the patient, identity and experiences levels and training levels of the sonographer performing the ultrasound procedure, and other types of static information that may be obtained before an ultrasound procedure. The static information may also comprise a feature of the ultrasound system, such as a type of the ultrasound probe 110, the ultrasound base 120 or the ultrasound probe 210.

[0043] At S320, an ultrasound procedure is started. For example, the ultrasound probe 110 in FIG. 1 or the ultrasound probe 210 in FIG. 2 may be turned on, and may be initiated or activated, and a sonographer may begin obtaining ultrasound images from a patient.

[0044] At S330, dynamic information is obtained. The dynamic information is information dynamically obtained from performing the ultrasound procedure. The dynamic information may be generated based on usage of the ultrasound system during the ultrasound procedure. The dynamic information is information directly or derived indirectly from the ultrasound procedure started at S320. The dynamic information may include the actual content of the ultrasound images, information derived from the content of the ultrasound images, as well as the context of the ultrasound imaging examination such as diagnosis.

[0045] At S340, the static information and the dynamic information are input to a model. Themodel may comprise a trained machine learning model stored in and executed by the controller 150 in FIG. 1 or the controller 250 in FIG. 2. The model may be stored in the memory 151 or the memory 251 as a software program. The software program(s) in the memory 151 or the memory 251 generate recommendations to be displayed on the display 180 or on smartphone A or smartphone B for further ultrasound scans and for which images to capture and which measurements to take.

[0046] The model may operate in two modes, either continuous or summary. The continuous mode involves continuous update of recommendations based on the captured information of the current examination, from start until the current time. The summary mode involves summary recommendations when the user indicates that the examination is complete in the opinion of the user. The model may comprise an artificial intelligence model trained on archive ultrasound imaging reports. Inputs for training the artificial model may include metadata and image and measurement contents. The metadata may comprise static information such as a diagnostic question, patient characteristics such as body mass index (BMI) or age, and reading / referring physician preferences. The image and measurement contents (IMC) may comprise dynamic information that can be obtained during an ultrasound examination, such as from image analysis by an image analysis program, and measurements by the image analysis program. The ultrasound base 120 or the ultrasound probe 210 may be configured to analyze an ultrasound image generated during the ultrasound procedure, and generate the dynamic information based on analyzing the ultrasound image. Also, or alternatively, the ultrasound base 120 or the ultrasound probe 210 may be configured to identify their usage during the ultrasound procedure, and generate the dynamic information based on the usage during the ultrasound procedure. The training may use a complete metadata set and leave one out subset of the image and measurement content (IMC) to predict the difference set between the full set of image and measurement content and the current subset. As an example, a full set may be A-B-C-D-E, A-B- C-D may be trained to predict E, A-B-C-E may be trained to predict D, and so on. The predicted images may be images that get recommended during inference in the deployment of the trained artificial intelligence model.

[0047] At S350, a determination is made as to whether a recommendation is to be made. The determination at S350 is based on applying the model at S340. If no recommendation is to bemade (S350 = No), the method returns to S330 to continue obtaining dynamic information from the ultrasound procedure started at S320. The determination at S350 may also include a determination as to how to bring an recommendation(s) to the attention of the sonographer.

[0048] If a recommendation is to be made (S350 = Yes), at S360 the recommendation is generated. The trained machine learning model may generate a recommendation for performing the ultrasound procedure based on the static information and the dynamic information. The recommendation is generated by the trained machine learning model during the ultrasound procedure insofar as the recommendation if for one or more next steps to take or consider taking during the ultrasound procedure. The recommendation is generated by the model applied at S340. The recommendation may be provided as guidance to be output on a user interface used during ultrasound examinations, such as the display 180 in FIG. 1 or smartphone A and / or smartphone B in FIG. 2. In some embodiments, recommendations may be output audibly instead of or together with visualizations of the recommendations. The recommendation may be a recommendation to capture additional ultrasound images or measurements based on the capture history of the current ultrasound examination and the capture history of previous, similar examinations, as well as other types of static and dynamic information. A recommendation may be generated by the trained machine learning model based on actions taken by one or more sonographer(s) in one or more matching previous context used in training the trained machine learning model. The recommendation may be implemented to alert a sonographer to take additional measurements that other sonographers in a matching imaging context have taken. The alerts and recommendations described herein may indicate how to integrate the recommendations into the examination workflow.

[0049] At S370, a marker is generated. The marker is for the recommendation and is used to recommend an adjustment to the ultrasound procedure. The marker may comprise text and / or icons to be overlaid on or adjacent to ultrasound images on the display 180 or smartphone A and / or smartphone B. The marker may be generated by the controller 150 of the ultrasound base 120 in FIG. 1, or by either the controller 250 of the ultrasound probe 210 in FIG. 2 or else by the ultrasound application 299A on smartphone A or by the ultrasound application 299B of smartphone B in FIG. 2.

[0050] At S380, the marker is overlaid on or adjacent to ultrasound images from the ultrasoundprocedure. In FIG. 1, the marker is overlaid on the ultrasound images on the display 180. In FIG. 2, the marker may be overlaid on the ultrasound images on smartphone A or smartphone B.

[0051] At S390, a determination is made as to whether the ultrasound procedure started at S320 is now complete. If the ultrasound procedure is complete (S390 = Yes), the method finishes at S399. If the ultrasound procedure is not complete (S390 = No), the method returns to S330 to obtain additional dynamic information from the ultrasound procedure.

[0052] As set forth above, the method of FIG. 3 may use software executing in the background during and immediately after the acquisition phase of an ultrasound examination. The use of a suitable user interface on the display 180 or smartphone A or smartphone B enables automatic recommendations for acquisition of additional scans and / or for capture of additional images and measurements. The model applied at S340 may comprise an artificial intelligence model that matches the current examination to an archive of ultrasound reports. The model may apply a diagnostic question similar to metadata, patient characteristics and preferences of the reading / referring physician. The model may also be applied dynamically to the image-level content of the actual ultrasound images from scans, image captures and measurements already completed during the current examination.

[0053] FIG. 4 illustrates inputs and outputs for dynamic ultrasound recommendations, in accordance with a representative embodiment.

[0054] In FIG. 4, a series of factors used as inputs for a dynamic ultrasound recommendation include information from other sonographers 401 who have performed ultrasound examinations and on similar patients 402. The series of factors may also or alternatively include information particular to the sonographer 403 and particular to the ultrasound device(s) 404 being used by the sonographer. The series of factors may also or alternatively include information particular to the diagnosis 405 for the current patient. The output in FIG. 4 comprises a recommendation 406 of a particular capture for the sonographer to take. The label for the input for information from other sonographers 401 in FIG. 4 states “other sonographers who scanned”. The label for the input for similar patients 402 in FIG. 4 states “such patients”. The label for the input for information particular to the sonographer 403 in FIG. 4 states “for Dr. Doe”. The label for the input for the ultrasound device(s) 404 in FIG. 4 states “with this equipment”. The label for the input for the diagnosis 405 in FIG. 4 states “for this diagnosis”. The label for the output of therecommendation in FIG. 4 states “also took the following capture”. The various inputs in FIG. 4 may correspond to some or all of the inputs used as training inputs when training a trained machine learning algorithm.

[0055] FIG. 5 illustrates a dynamic ultrasound recommendation, in accordance with a representative embodiment.

[0056] In FIG. 5, an ultrasound image on the user interface 581 A is analyzed to recommend capture of an ultrasound image on the user interface 581B. Nodularity is not shown in the ultrasound image on the user interface 581 A. Typically, a curvilinear transducer is used for liver imaging to provide coverage and penetration. However, the curvilinear transducers are lower frequency and not optimized for near field visualization. The ultrasound image on the user interface 58 IB is from a high frequency transducer optimized for near field visualization and reveals nodularity at the liver capsule, as shown by the line below the portal vein as indicated by the arrow on the user interface 581B.

[0057] In FIG. 5, an overlaid icon 582 may be provided with text and is separate from the ultrasound images. The overlaid icon 582 may be provided on the display 180 in FIG. 1 as a recommendation to capture an image with a high frequency probe to check for nodularity. The recommendation may include text to suggest that a sonographer changes transducers. For example, the overlaid icon 582 may have text that states “Recommended: image with high freq, probe to check for nodularity”. With sufficient data, the machine learning model used to generate the recommendation may also be trained to provide the reason for the recommendation. The context for the dynamic ultrasound recommendation in FIG. 5 is for a sonographic examination of the liver. Insofar as the liver is a complex organ with many vessels, multiple and different views may be necessary during a liver ultrasound examination, depending on the diagnostic question. Variations may be based, for example, on confirming and characterizing liver tumor versus assessing severity of a fatty liver disease. Variations may be based on, for instance patient characteristics such as for obese patients requiring increasing signal and depth. Variations may also be based on the dynamically generated content during the examination, resulting in the ultrasound image on the user interface 581 A.

[0058] The model used to generate the recommendation on the overlaid icon 582 may comprise an artificial intelligence model trained on the metadata and image and measurement contents ofnative historical archive data. The historical archive data may be native in the sense that there is no human annotation of salient features. For example, in the ultrasound image on the user interface 581 A, the echogenic thrombus does not need to be marked up in the image by a human reader. Given a sufficient amount of input data, the artificial intelligence model may learn that such image characteristics triggered Doppler imaging in all or at least some cases and thus the model may be trained to recommend color and spectral Doppler mode for a current examination that exhibits similar characteristics in a B-mode image.

[0059] An approach of matching using only metadata from the content of the archive report is not as effective because the current examination may only contain relevant metadata after the report is made. For example, if metadata from the content of the archive report were to be solely relied on and were to recommend a Doppler examination, the metadata would not be effective since the omission of a Doppler examination could not easily be remedied.

[0060] FIG. 6 illustrates another dynamic ultrasound recommendation, in accordance with a representative embodiment.

[0061] In FIG. 6, a different imaging mode is recommended based on what is seen during the ultrasound examination. The ultrasound image on the user interface 681 A is analyzed, and results in an overlaid icon 682 with a recommendation resulting in the ultrasound image on the user interface 68 IB. The ultrasound image with the overlaid icon 682 may be provided on the display 180 in FIG. 1. For example, the overlaid icon 682 may have text that states “Recommended: switch to color and spectral Doppler”. The ultrasound image on the user interface 681 A includes a portal vein and reveals an echogenic thrombus indicated by the arrow on the user interface 681 A. The echogenic thrombus nearly occludes the vessel lumen. The recommendation on the overlaid icon 682 is to switch to color and use a spectral Doppler imaging mode. The ultrasound image on the user interface 68 IB comprises a Doppler image that demonstrates low-resistance arterial vascularity within the thrombus, consistent with a tumor in a vein. A new thrombus in a vein, whether a tumor in the vein or a bland thrombus, qualifies an hepatocellular carcinoma (HCC) surveillance ultrasound examination as “US-3 Positive”, based on the Ultrasound Liver Reporting & Data System (LLRADS) scoring metric. The ultrasound image on the user interface 681B may result in a recommendation for a further diagnostic evaluation with a multiphase contrast-enhanced imaging study based on identification of the newthrombus in the vein.

[0062] FIG. 7 illustrates a user interface for dynamic ultrasound recommendation, in accordance with a representative embodiment.

[0063] Algorithms may be used to label a predicted image(s) so that the recommendation can be made using the label rather than an image. The image labeling may be automated. FIG. 7 illustrates a current view of an EPIQ main screen with a variety of labels. The ultrasound image shown on the user interface 781 is from a continuous-update mode of the model used to generate recommendations. The model is trained on historical ultrasound examinations including exam metadata such as the various labels shown on the user interface 781. Exam metadata may consist of diagnostic question, the hosting department, the referring physician, and patient characteristics. The diagnostic question may be represented by one or more International Statistical Classification of Diseases and Related Health Problems 10threvision (ICD-10) or newer codes for easier indexation. All captured ultrasound images in the training data may also be labeled to link the captured ultrasound images with anatomy and view.

[0064] In continuous-update mode, the software program used to implement the model analyzes each screen capture or measurement acquired by the user immediately after the screen capture or measurement is taken. The model predicts the next view within the context of the examination metadata and screens and measurements captured previously during the present exam. Either the label of the predicted view or measurement is communicated to the user right away or after the following screen capture, in the event that the following screen capture does not match the prediction. The continuous-update mode may be appropriate for novice or less-well trained users and may improve overall exam diagnostic quality and completeness, as well as reduce exam time. In the continuous-update mode, it may be useful to display the label of the next recommended view or measurement as well as a picture rendition for further guidance also a picture rendition. For example, the thumbnail views labeled 1, 2, 3, 4, 5, 6 and 7 may each correspond to other labelled ultrasound images used as training inputs for a trained machine learning model.

[0065] FIG. 8 illustrates another user interface for dynamic ultrasound recommendation, in accordance with a representative embodiment.

[0066] FIG. 8 illustrates an ultrasound image on the user interface 881. The ultrasound image inFIG. 8 may be the same as or similar to the ultrasound image in FIG. 7. However, in FIG. 8, suggested views are provided with indicators for corresponding recommendations. In FIG. 8, the ultrasound image on the user interface 881 is overlaid with alert icons to indicate for which captures or measurements a recommendation is found. For example, the exclamation point used as an alert icon 882 may be colored yellow on the user interface 881. Similar alert icons on thumbnails on the right may also be colored yellow, including the alert icon 883 and the alert icon 884. The user can then click on these icons to obtain further guidance on the additional recommended captures or measurements.

[0067] FIG. 9 illustrates another user interface for dynamic ultrasound recommendation, in accordance with a representative embodiment.

[0068] In FIG. 9, a series of ultrasound images are shown as a dynamic view for suggested recommendations on a user interface 981. The series of ultrasound images in FIG. 9 provide an example of how completed follow-ups to recommendations may be indicated. Icons in FIG. 9 may be shown in different color codes. For example, yellow color may indicate a recommendation identified by the model, as demonstrated by the low-quality image shown as alert icon 982. If the user ignores the yellow recommendation at the moment but captures the recommended views or measurements in the rest of the examination, the exclamation mark color code may change to green, as shown in alert icon 983 with the improved image-quality. If the user interrupts the examination flow to follow a recommendation, clicks / taps the icon and performs the recommended task then the icon may turn into a green check mark to indicate completion via recommender interaction. This is demonstrated as alert icon 984, showing an image with optimal image quality.

[0069] FIG. 10 illustrates another user interface for dynamic ultrasound recommendation, in accordance with a representative embodiment.

[0070] In FIG. 10, a live scanning screen is shown on the user interface 1081. The live scanning screen is from after a user clicked on a recommendation. The portion labelled A is a reference image 1082. The portion labelled B is the recommended capture 1083. The portion labelled C is the actual capture 1084 which triggered the recommendation. The portion labelled D is the live image on the primary window of the user interface 1081.

[0071] Upon clicking / tapping an exclamation mark such as a yellow exclamation markindicating a new recommendation, the model may change the view on the display 180 or the smartphone A and / or the smartphone B. The view may change to overlay an image pair as shown in FIG. 5 and FIG. 6 on top of the current captured image. The arrow with text in FIG. 5 and FIG. 6 may be replaced by a set or arrows such as in FIG. 4 to show the meta information that the recommendation is based on. An example of the live scanning screen after the user activated a recommendation is shown in FIG. 10.

[0072] Interaction with an exclamation mark of a different color such as green may be slightly different. Since there is another capture which satisfies the recommendation, the exclamation mark of the different color will be shown together with the recommendation pair, i.e. the live scanning screen in the portion labelled D in FIG. 10 replaced by the capture which the system found to be close enough to the recommended image. If the user agrees that the capture is indeed equivalent to the recommended image and they are satisfied, the recommendation may be considered completed. Alternatively, a user may find a recommended image different than the other acquisition and may want to acquire another image based on the recommendation, in that case the center portion of the user interface 1081 turns into the live scanning screen again and the user continues with the acquisition.

[0073] The summary-recommendation mode may be more appropriate for highly trained users and is akin to a review of examination completeness performed at the end of the examination. The advantage of the summary-recommendation mode is that it is more unobtrusive, yet the user may find the summary-recommendation mode more cumbersome to perform an extra task - looking at the summary recommendations - at the end of an exam, and then potentially restart scanning to capture missed screens or measurements.

[0074] In some embodiments, user input is not particularly required by the prediction software. In some embodiments, one of the two modes may be implemented without requiring the other of the two modes.

[0075] FIG. 11 illustrates a computer system, on which a method for dynamic ultrasound recommendations is implemented, in accordance with another representative embodiment.

[0076] Referring to FIG. 11, the computer system 1100 includes a set of software instructions that can be executed to cause the computer system 1100 to perform any of the methods or computer-based functions disclosed herein. The computer system 1100 may operate as astandalone device or may be connected, for example, using a network 1101, to other computer systems or peripheral devices. In embodiments, a computer system 1100 performs logical processing based on digital signals received via an analog-to-digital converter.

[0077] In a networked deployment, the computer system 1100 operates in the capacity of a server or as a client user computer in a server-client user network environment, or as a peer computer system in a peer-to-peer (or distributed) network environment. The computer system 1100 can also be implemented as or incorporated into various devices, such as a workstation that includes a controller, a stationary computer, a mobile computer, a personal computer (PC), a laptop computer, a tablet computer, or any other machine capable of executing a set of software instructions (sequential or otherwise) that specify actions to be taken by that machine. The computer system 1100 can be incorporated as or in a device that in turn is in an integrated system that includes additional devices. In an embodiment, the computer system 1100 can be implemented using electronic devices that provide voice, video or data communication. Further, while the computer system 1100 is illustrated in the singular, the term “system” shall also be taken to include any collection of systems or sub-systems that individually or jointly execute a set, or multiple sets, of software instructions to perform one or more computer functions.

[0078] As illustrated in FIG. 11, the computer system 1100 includes a processor 1110. The processor 1110 may be considered a representative example of a processor of a controller and executes instructions to implement some or all aspects of methods and processes described herein. The processor 1110 is tangible and non-transitory. As used herein, the term “non- transitory” is to be interpreted not as an eternal characteristic of a state, but as a characteristic of a state that will last for a period. The term “non-transitory” specifically disavows fleeting characteristics such as characteristics of a carrier wave or signal or other forms that exist only transitorily in any place at any time. The processor 1110 is an article of manufacture and / or a machine component. The processor 1110 is configured to execute software instructions to perform functions as described in the various embodiments herein. The processor 1110 may be a general-purpose processor or may be part of an application specific integrated circuit (ASIC). The processor 1110 may also be a microprocessor, a microcomputer, a processor chip, a controller, a microcontroller, a digital signal processor (DSP), a state machine, or a programmable logic device. The processor 1110 may also be a logical circuit, including aprogrammable gate array (PGA), such as a field programmable gate array (FPGA), or another type of circuit that includes discrete gate and / or transistor logic. The processor 1110 may be a central processing unit (CPU), a graphics processing unit (GPU), or both. Additionally, any processor described herein may include multiple processors, parallel processors, or both. Multiple processors may be included in, or coupled to, a single device or multiple devices.

[0079] The term “processor” as used herein encompasses an electronic component able to execute a program or machine executable instruction. References to a computing device comprising “a processor” should be interpreted to include more than one processor or processing core, as in a multi-core processor. A processor may also refer to a collection of processors within a single computer system or distributed among multiple computer systems. The term computing device should also be interpreted to include a collection or network of computing devices each including a processor or processors. Programs have software instructions performed by one or multiple processors that may be within the same computing device or which may be distributed across multiple computing devices.

[0080] The computer system 1100 further includes a main memory 1120 and a static memory 1130, where memories in the computer system 1100 communicate with each other and the processor 1110 via a bus 1108. Either or both of the main memory 1120 and the static memory 1130 may be considered representative examples of a memory of a controller, and store instructions used to implement some or all aspects of methods and processes described herein. Memories described herein are tangible storage mediums for storing data and executable software instructions and are non-transitory during the time software instructions are stored therein. As used herein, the term “non-transitory” is to be interpreted not as an eternal characteristic of a state, but as a characteristic of a state that will last for a period. The term “non- transitory” specifically disavows fleeting characteristics such as characteristics of a carrier wave or signal or other forms that exist only transitorily in any place at any time. The main memory 1120 and the static memory 1130 are articles of manufacture and / or machine components. The main memory 1120 and the static memory 1130 are computer-readable mediums from which data and executable software instructions can be read by a computer (e.g., the processor 1110). Each of the main memory 1120 and the static memory 1130 may be implemented as one or more of random access memory (RAM), read only memory (ROM), flash memory, electricallyprogrammable read only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, a hard disk, a removable disk, tape, compact disk read only memory (CD-ROM), digital versatile disk (DVD), floppy disk, blu-ray disk, or any other form of storage medium known in the art. The memories may be volatile or non-volatile, secure and / or encrypted, unsecure and / or unencrypted.

[0081] “Memory” is an example of a computer-readable storage medium. Computer memory is any memory which is directly accessible to a processor. Examples of computer memory include, but are not limited to RAM memory, registers, and register files. References to “computer memory” or “memory” should be interpreted as possibly being multiple memories. The memory may for instance be multiple memories within the same computer system. The memory may also be multiple memories distributed amongst multiple computer systems or computing devices.

[0082] As shown, the computer system 1100 further includes a video display unit 1150, such as a liquid crystal display (LCD), an organic light emitting diode (OLED), a flat panel display, a solid-state display, or a cathode ray tube (CRT), for example. Additionally, the computer system 1100 includes an input device 1160, such as a keyboard / virtual keyboard or touch-sensitive input screen or speech input with speech recognition, and a cursor control device 1170, such as a mouse or touch-sensitive input screen or pad. The computer system 1100 also optionally includes a disk drive unit 1180, a signal generation device 1190, such as a speaker or remote control, and / or a network interface device 1140.

[0083] In an embodiment, as depicted in FIG. 11, the disk drive unit 1180 includes a computer- readable medium 1182 in which one or more sets of software instructions 1184 (software) are embedded. The sets of software instructions 1184 are read from the computer-readable medium 1182 to be executed by the processor 1110. Further, the software instructions 1184, when executed by the processor 1110, perform one or more steps of the methods and processes as described herein. In an embodiment, the software instructions 1184 reside all or in part within the main memory 1120, the static memory 1130 and / or the processor 1110 during execution by the computer system 1100. Further, the computer-readable medium 1182 may include software instructions 1184 or receive and execute software instructions 1184 responsive to a propagated signal, so that a device connected to a network 1101 communicates voice, video or data over the network 1101. The software instructions 1184 may be transmitted or received over the network1101 via the network interface device 1140.

[0084] In an embodiment, dedicated hardware implementations, such as application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), programmable logic arrays and other hardware components, are constructed to implement one or more of the methods described herein. One or more embodiments described herein may implement functions using two or more specific interconnected hardware modules or devices with related control and data signals that can be communicated between and through the modules. Accordingly, the present disclosure encompasses software, firmware, and hardware implementations. Nothing in the present application should be interpreted as being implemented or implementable solely with software and not hardware such as a tangible non-transitory processor and / or memory.

[0085] In accordance with various embodiments of the present disclosure, the methods described herein may be implemented using a hardware computer system that executes software programs. Further, in an exemplary, non-limited embodiment, implementations can include distributed processing, component / object distributed processing, and parallel processing. Virtual computer system processing may implement one or more of the methods or functionalities as described herein, and a processor described herein may be used to support a virtual processing environment.

[0086] Accordingly, dynamic ultrasound recommendations enables ultrasound imaging technologists and other users who administer ultrasound imaging examinations to comply with device capabilities, guidelines, state of the art of ultrasound imaging techniques and stakeholder preferences.

[0087] Although dynamic ultrasound recommendations has been described with reference to several exemplary embodiments, it is understood that the words that have been used are words of description and illustration, rather than words of limitation. Changes may be made within the purview of the appended claims, as presently stated and as amended, without departing from the scope and spirit of dynamic ultrasound recommendations in its aspects. Although dynamic ultrasound recommendations has been described with reference to particular means, materials and embodiments, dynamic ultrasound recommendations is not intended to be limited to the particulars disclosed; rather dynamic ultrasound recommendations extends to all functionally equivalent structures, methods, and uses such as are within the scope of the appended claims.

[0088] The illustrations of the embodiments described herein are intended to provide a general understanding of the structure of the various embodiments. The illustrations are not intended to serve as a complete description of all of the elements and features of the disclosure described herein. Many other embodiments may be apparent to those of skill in the art upon reviewing the disclosure. Other embodiments may be utilized and derived from the disclosure, such that structural and logical substitutions and changes may be made without departing from the scope of the disclosure. Additionally, the illustrations are merely representational and may not be drawn to scale. Certain proportions within the illustrations may be exaggerated, while other proportions may be minimized. Accordingly, the disclosure and the figures are to be regarded as illustrative rather than restrictive.

[0089] One or more embodiments of the disclosure may be referred to herein, individually and / or collectively, by the term “invention” merely for convenience and without intending to voluntarily limit the scope of this application to any particular invention or inventive concept. Moreover, although specific embodiments have been illustrated and described herein, it should be appreciated that any subsequent arrangement designed to achieve the same or similar purpose may be substituted for the specific embodiments shown. This disclosure is intended to cover any and all subsequent adaptations or variations of various embodiments. Combinations of the above embodiments, and other embodiments not specifically described herein, will be apparent to those of skill in the art upon reviewing the description.

[0090] The Abstract of the Disclosure is provided to comply with 37 C.F.R. §1.72(b) and is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing Detailed Description, various features may be grouped together or described in a single embodiment for the purpose of streamlining the disclosure. This disclosure is not to be interpreted as reflecting an intention that the claimed embodiments require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter may be directed to less than all of the features of any of the disclosed embodiments. Thus, the following claims are incorporated into the Detailed Description, with each claim standing on its own as defining separately claimed subject matter.

[0091] The preceding description of the disclosed embodiments is provided to enable any person 1skilled in the art to practice the concepts described in the present disclosure. As such, the above disclosed subject matter is to be considered illustrative, and not restrictive, and the appended claims are intended to cover all such modifications, enhancements, and other embodiments which fall within the true spirit and scope of the present disclosure. Thus, to the maximum extent allowed by law, the scope of the present disclosure is to be determined by the broadest permissible interpretation of the following claims and their equivalents and shall not be restricted or limited by the foregoing detailed description.

Claims

CLAIMS:

1. An ultrasound system (100), comprising: a memory (151) that stores instructions; and a processor (1110) that executes the instructions, wherein, when executed by the processor (1110), the instructions cause the ultrasound system (100) to: receive static information (S310) relating to an ultrasound procedure and dynamic information (S330) dynamically obtained from performing the ultrasound procedure; input (S340) the static information and the dynamic information to a trained machine learning model; generate (S360, by the trained machine learning model during the ultrasound procedure, a recommendation for performing the ultrasound procedure based on the static information and the dynamic information; and overlay (S380) a marker for the recommendation on an ultrasound image to recommend an adjustment to the ultrasound procedure.

2. The ultrasound system (100) of claim 1, further comprising: an ultrasound cart; and a display (180), wherein the memory (151) and the processor (1110) are implemented in the ultrasound cart, and the ultrasound image with the marker overlaid on the ultrasound image is displayed on the display (180).

3. The ultrasound system (100) of claim 1, further comprising: a portable transducer, wherein the portable transducer is configured to link to an external device for the ultrasound procedure via an application installed on the external device, and wherein the memory (151) and the processor (1110) are implemented in the portable transducer, and the ultrasound image with the marker overlaid on the ultrasound image is displayed on the external device.

4. The ultrasound system (100) of claim 1, wherein, when executed by the processor (1110), the instructions further cause the ultrasound system (100) to: analyze an ultrasound image generated during the ultrasound procedure; and generate the dynamic information based on analyzing the ultrasound image.

5. The ultrasound system (100) of claim 1, wherein, when executed by the processor (1110), the instructions further cause the ultrasound system (100) to: identify usage of the ultrasound system (100) during the ultrasound procedure; and generate the dynamic information based on usage of the ultrasound system (100) during the ultrasound procedure.

6. The ultrasound system (100) of claim 1, wherein the static information comprises a training level of a sonographer performing the ultrasound procedure.

7. The ultrasound system (100) of claim 1, wherein the static information comprises a feature of the ultrasound system (100).

8. The ultrasound system (100) of claim 1, wherein the recommendation is generated by the trained machine learning model based on actions taken by a sonographer in a matching previous context used in training the trained machine learning model.

9. The ultrasound system (100) of claim 1, wherein the marker for the recommendation comprises at least one of a recommendation to capture at least one additional image.

10. An method for ultrasound imaging, comprising: receiving, by a controller (150) comprising a memory (151) that stores instructions and a processor (1110) that executes the instructions, static information (S310) relating to an ultrasound procedure and dynamic information (S330) dynamically obtained from performing the ultrasound procedure;inputting (S340) the static information and the dynamic information to a trained machine learning model; generating (S360), by the trained machine learning model during the ultrasound procedure, a recommendation for performing the ultrasound procedure based on the static information and the dynamic information; and overlaying (S380) a marker for the recommendation on an ultrasound image to recommend an adjustment to the ultrasound procedure.

11. The method of claim 10, wherein the controller (150) is implemented in an ultrasound cart, and the method is performed by the controller (150) and a display (180) which receives the marker from the ultrasound cart, and wherein the method further comprises displaying on the display (180) the marker overlaid on the ultrasound image.

12. The method of claim 10, wherein the method is performed by a controller (150) implemented in a portable transducer, wherein the portable transducer is configured to link to an external device for the ultrasound procedure via an application installed on the external device, and wherein the ultrasound image with the marker overlaid on the ultrasound image is displayed on the external device.

13. The method of claim 10, further comprising: analyzing an ultrasound image generated during the ultrasound procedure; and generating the dynamic information based on analyzing the ultrasound image.

14. The method of claim 10, further comprising: identifying usage of an ultrasound system (100) that includes the controller (150) during the ultrasound procedure; and generating the dynamic information based on usage of the ultrasound system (100) during the ultrasound procedure.

15. The method of claim 10, wherein the static information (3) comprises a training level of a sonographer performing the ultrasound procedure.

16. The method of claim 10, wherein the static information comprises a feature of an ultrasound system (100) that includes the controller (150).

17. The method of claim 10, wherein the recommendation is generated by the trained machine learning model based on actions taken by a sonographer in a matching previous context used in training the trained machine learning model.

18. The method of claim 10, wherein the marker for the recommendation comprises at least one of a recommendation to capture at least one additional image.

19. A tangible non-transitory computer-readable storage medium that stores a computer program, wherein the computer program, when executed by a processor (1110), causes a system (100) to: receive static information (S310) relating to an ultrasound procedure and dynamic information (S330) dynamically obtained from performing the ultrasound procedure; input (S340) the static information and the dynamic information to a trained machine learning model; generate (S360), by the trained machine learning model during the ultrasound procedure, a recommendation for performing the ultrasound procedure based on the static information (3) and the dynamic information (3); and overlay (S380) a marker for the recommendation on an ultrasound image to recommend an adjustment to the ultrasound procedure.

Images