SYSTEM, METHOD AND APPARATUS FOR ANNOTATING MEDICAL IMAGES - Patent application

Abstract

The ultrasound imaging system analyzes the acquired images, the ultrasound probe position and orientation, and usage data to provide suggested annotations for the acquired images. The annotations can be in a variety of forms, including text and body marker icons.

Description

[Technical field]

[0001]

[0001] The present disclosure relates to imaging systems and methods for automatically or semi-automatically annotating medical images. In particular, an imaging system or method for automatically or semi-automatically generating annotations for ultrasound images is disclosed. [Background technology]

[0002] During a typical ultrasound examination, a user (e.g., a sonographer) acquires multiple images of an anatomical structure of interest in various imaging modes and orientations. For example, in a typical liver ultrasound examination in North America, a user acquires multiple images of the liver, kidneys, gallbladder, etc., for simultaneous or later review by a physician. To aid in the reading of the ultrasound images and for archival purposes, the images are annotated by the sonographer to inform the reviewer of the details of the scan. In a typical workflow, the sonographer scans with the ultrasound probe until the desired image plane is achieved. The sonographer then "freezes" the image at the desired image plane. The user makes the desired annotations on the frozen image. Annotations may include, among other things, text labels, body marker icons, and / or measurements. Once the user is satisfied, the annotated image is "acquired." That is, the image is stored (saved) in the computer-readable memory of the ultrasound imaging system and / or provided to a medical picture and communication system (PACS).

[0003]

[0003] Figure 1 is an example of a display from an ultrasound imaging system including annotations on an ultrasound image. Display 100 includes an ultrasound image 102 of a liver and annotations 104 and 106. Label 104 is a textual annotation that provides information about the anatomy in ultrasound image 102, the region of the anatomy, and the imaging plane. In the example shown in the example of Figure 1, label 104 indicates that the image is of the sagittal (SAG) plane of the left (LT) portion of the liver. Body marker 106 is an annotation that provides a graphical representation of a part of a subject's body 108 and an icon indicating the position and orientation of an ultrasound probe 110 relative to the body 108. In the example shown in Figure 1, body marker 106 includes an illustration of a torso with a beam directed toward the navel and an ultrasound probe positioned near the top of the torso. As shown in FIG. 1, an ultrasound image 102 may include annotations with both labels 104 and body markers 106, although currently, labels 104 are more commonly used in North America, while body markers 106 are more commonly used in Europe and Asia.

[0004]

[0004] The user manually types in a label and / or selects a label from a preloaded list (e.g., by navigating one or more drop-down menus). For body markers, the user selects an appropriate graphic for the body marker from a menu and manually places an icon over the graphic (e.g., using the selection buttons along with the arrow keys and / or trackball) indicating the location and orientation of the ultrasound probe. Both types of annotations can require significant time from the user, especially when many images need to be captured and annotated. For example, a typical abdominal exam requires 40-50 images.

[0005]

[0005] Some imaging systems offer a "smart exam" feature that automatically annotates ultrasound images with labels, body markers, and / or other annotations. However, a "smart exam" typically requires the user to acquire images in a particular order and may not provide all the images the user desires to acquire from the exam. Therefore, improved techniques are desired to reduce the time to annotate ultrasound images. Summary of the Invention [Problem to be solved by the invention]

[0006]

[0006] Significant time is spent by users of imaging systems assigning annotations, such as labels and body markers, to ultrasound images. Disclosed are systems, methods, and apparatus for automatically or semi-automatically applying labels and / or body markers as users acquire various images. This results in significant time savings for users and allows for tailoring to their experience. [Means for solving the problem]

[0007]

[0007] In accordance with at least one example of the present disclosure, an ultrasound imaging system is configured to annotate ultrasound images, the ultrasound imaging system including an ultrasound probe configured to acquire ultrasound signals, a non-transitory computer readable medium encoded with instructions, the non-transitory computer readable medium configured to store the ultrasound images, and at least one processor configured to communicate with the non-transitory computer readable medium to execute the instructions, which, when executed, cause the ultrasound imaging system to determine anatomical features present in the ultrasound image, determine at least one of an imaging plane, a position of the ultrasound probe, or an orientation of the ultrasound probe, determine annotations to be applied to the ultrasound image based at least in part on the determined anatomical features and at least one of the determined imaging plane, the determined position of the ultrasound probe, or the determined orientation of the ultrasound probe, and provide the ultrasound image and annotations to the non-transitory computer readable medium for storage.

[0008]

[0008] According to at least one example of the present disclosure, a method for annotating an ultrasound image includes receiving an ultrasound image acquired by an ultrasound probe; determining, with at least one processor, anatomical features present in the ultrasound image; determining, with at least one processor, at least one of an imaging plane, a position of the ultrasound probe, or an orientation of the ultrasound probe; determining, with the at least one processor, annotations to apply to the ultrasound image based at least in part on the determined anatomical features and at least one of the determined imaging plane, the determined position of the ultrasound probe, or the determined orientation of the ultrasound probe; and providing the ultrasound image and annotations to at least one of a non-transitory computer-readable medium for display or storage. [Brief description of the drawings]

[0009] [Figure 1] 1 is an example of a display from an ultrasound imaging system including annotations on an ultrasound image. [Diagram 2]

[0010] FIG. 1 is a block diagram of an ultrasound system in accordance with the principles of the present disclosure. [Diagram 3]

[0011] FIG. 2 is a block diagram providing an overview of data flow in an ultrasound imaging system according to an example of the present disclosure. [Figure 4]

[0012] 1 is a graphical representation of an example of a statistical analysis of one or more log files according to an example of the present disclosure. [Diagram 5]

[0013] 1 is a graphical representation of an example of a statistical analysis of one or more log files according to an example of the present disclosure. [Figure 6]

[0014] 1 is an illustration of a neural network applied to data according to an example of the present disclosure. [Figure 7]

[0015] 1 is an illustration of a cell of a long short-term memory model used to analyze data according to an example of the present disclosure. [Figure 8]

[0016] 1 is an illustration of a decision tree used to analyze data according to an example of the present disclosure. [Figure 9]

[0017] FIG. 1 is a block diagram of a process for training and deploying a neural network in accordance with the principles of the present disclosure. [Figure 10]

[0018] 1 is a flow chart of a method according to the principles of the present disclosure. [Figure 11]

[0019] 1 is a block diagram illustrating an exemplary processor in accordance with the principles of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010]

[0020] The following description of specific embodiments is merely exemplary in nature and is not intended to limit the invention or its application or uses in any manner. In the following detailed description of the embodiments of the system and method according to the present invention, reference is made to the accompanying drawings which form a part hereof, and in which are shown, by way of illustration, specific examples in which the described system and method are practiced. Although these embodiments have been described in sufficient detail to enable those skilled in the art to practice the apparatus, systems and methods disclosed herein, it should be understood that other embodiments may be used, and that structural and logical changes may be made without departing from the spirit and scope of the present disclosure. Moreover, for the sake of clarity, detailed descriptions of certain features will not be discussed when they are apparent to those skilled in the art, so as not to obscure the description of the apparatus, systems and methods of the present invention. Therefore, the following detailed description should not be taken in a limiting sense, and the scope of the system according to the present invention is defined only by the appended claims.

[0011]

[0021] According to examples of the present disclosure, an ultrasound imaging system can automatically annotate (e.g., apply annotations to) ultrasound images based on anatomical features identified in the ultrasound images from one or more sources of information, previously used annotations, previously acquired ultrasound images, usage data, and / or ultrasound probe tracking data.

[0012]

[0022] FIG. 2 illustrates a block diagram of an ultrasound imaging system 200 constructed in accordance with the principles of the present disclosure. The ultrasound imaging system 200 according to the present disclosure includes a transducer array 214 included in an ultrasound probe 212, such as an external or internal probe, such as a transvaginal ultrasound (TVUS) probe or a transesophageal echocardiogram (TEE) probe. The transducer array 214 is configured to transmit ultrasound signals (e.g., beams, waves) and receive echoes in response to the ultrasound signals. A variety of transducer arrays may be used, such as, for example, a linear array, a curved array, or a phased array. The transducer array 214 may include, for example, a two-dimensional array of transducer elements capable of scanning in both elevation and azimuth dimensions for 2D and / or 3D imaging (as shown). As is commonly known, the axial direction is the direction perpendicular to the face of the array (which fans out in the case of a curved array), the azimuth direction is commonly defined by the longitudinal dimension of the array, and the elevation direction is the direction transverse to the azimuth direction.

[0013]

[0023] Optionally, in some examples, the ultrasound probe 212 includes a tracking device 270. In some examples, the tracking device 270 includes an inertial measurement unit (IMU). The IMU includes an accelerometer, a gyroscope, a magnetometer, and / or a combination thereof. The IMU provides data regarding the velocity, acceleration, rotation, angular velocity, and / or orientation of the probe 212. In some examples, the tracking device 270 additionally or alternatively includes an electromagnetic tracking device. The electromagnetic tracking device independently provides position and / or orientation information of the probe 212 and / or operates in cooperation with a probe tracking system 272. The probe tracking system 272 transmits and / or receives signals to and from the electromagnetic tracking system 270 and provides information regarding the position and / or orientation of the probe 212 to the ultrasound imaging system 200, such as to the local memory 242. An example of a suitable probe tracking system 272 is the PercuNav system by Philips Healthcare, although in other examples, other tracking systems are also used. Data related to the velocity, acceleration, rotation, angular velocity, position, and / or orientation of the probe 212 provided by the tracking device 270 is collectively referred to as probe tracking data.

[0014]

[0024] In some embodiments, the transducer array 214 is coupled to a microbeamformer 116 that is disposed on the ultrasound probe 212 and controls the transmission and reception of signals by the transducer elements in the array 214. In some embodiments, the microbeamformer 216 controls the transmission and reception of signals by the active elements in the array 214 (e.g., a subset of the elements of the array that define an active aperture at any one time).

[0015]

[0025] In some embodiments, the microbeamformer 216 is coupled, for example by a probe cable or wirelessly, to a transmit / receive (T / R) switch 118 that switches between transmit and receive and protects the main beamformer 222 from high energy transmit signals. In some embodiments, such as portable ultrasound systems, the T / R switch 218 and other elements of the system may be included in the ultrasound probe 212 rather than in the ultrasound system base, which typically contains software and hardware components including circuitry for signal processing and image data generation, and executable instructions for providing a user interface (e.g., processing circuitry 250 and user interface 224).

[0016]

[0026] The transmission of ultrasound signals from the transducer array 214, under the control of the microbeamformer 216, is directed by a transmit controller 220, which is coupled to the T / R switch 218 and the main beamformer 222. The transmit controller 220 controls the direction in which the beam is steered. The beam may be steered straight away from the transducer array 214 (orthogonal to the array) or may be steered at a number of different angles for a wider field of view. The transmit controller 220 is coupled to a user interface 224 to receive input from user manipulation of user controls. The user interface 224 includes one or more input devices, such as a control panel 252, which may include one or more mechanical controls (e.g., buttons, encoders), touch-sensitive controls (e.g., track pad, touch screen, etc.), and / or other known input devices.

[0017]

[0027] In some embodiments, the partially beamformed signals generated by the microbeamformer 216 are coupled to a main beamformer 222, where the partially beamformed signals from the individual patches of transducer elements are combined into a fully beamformed signal. In some embodiments, the microbeamformer 216 is omitted and the transducer array 214 is controlled by the main beamformer 122, which performs all of the beamforming of the signals. In embodiments with or without the microbeamformer 216, the beamformed signals of the main beamformer 222 are coupled to processing circuitry 250, which includes one or more processors (e.g., signal processor 226, B-mode processor 228, Doppler processor 260, and one or more image generation and processing components 268) configured to generate ultrasound images from the beamformed signals (e.g., beamformed RF data).

[0018]

[0028] The signal processor 226 is configured to process the received beamformed RF data in various ways, such as bandpass filtering, decimation, I and Q component separation, and harmonic signal separation. The signal processor 226 can also perform additional signal enhancements, such as speckle reduction, signal combining, and noise removal. The processed signals (also referred to as I and Q components or IQ signals) are coupled to additional downstream signal processing circuitry for image generation. The I and Q signals are coupled to multiple signal paths within the system, each associated with a particular signal processing component configuration suitable for generating a different type of image data (e.g., B-mode image data, Doppler image data). For example, the system includes a B-mode signal path 258 that couples a signal from the signal processor 226 to a B-mode processor 228 to generate B-mode image data.

[0019]

[0029] The B-mode processor uses amplitude detection for imaging of structures within the body. The signals generated by the B-mode processor 228 are coupled to a scan converter 230 and / or a multiplanar reformatter 232. The scan converter 230 arranges the echo signals into a desired image format from the spatial relationship in which they were received. For example, the scan converter 230 arranges the echo signals into a two-dimensional (2D) sector format, or into a pyramid or other shaped three-dimensional (3D) format. The multiplanar reformatter 232 can convert echoes received from multiple points in a common plane in a volumetric region of the body into an ultrasound image of that plane (e.g., a B-mode image), as described, for example, in U.S. Pat. No. 6,443,896 (Detmer). In some embodiments, the scan converter 230 and the multiplanar reformatter 232 are implemented as one or more processors.

[0020]

[0030] The volume renderer 234 generates an image (also called a projection, render, or rendering) of the 3D data set as viewed from a given reference point, for example as described in U.S. Patent No. 6,530,885 (Entrekin et al.). In some embodiments, the volume renderer 234 is implemented as one or more processors. The volume renderer 234 can generate renderings, such as positive or negative renderings, by any known or future known technique, such as surface rendering and maximum intensity rendering.

[0021]

[0031] In some embodiments, the system may include a Doppler signal path 262 that couples the output from the signal processor 226 to a Doppler processor 260. The Doppler processor 260 is configured to estimate the Doppler shift and generate Doppler image data. The Doppler image data includes color data that is overlaid with B-mode (i.e., grayscale) image data for display. The Doppler processor 260 is configured to filter out undesired signals (i.e., noise or clutter associated with stationary tissue), for example, using a wall filter. The Doppler processor 260 is further configured to estimate velocity and power according to known techniques. For example, the Doppler processor may include a Doppler estimator, such as an autocorrelator, where the velocity (Doppler frequency) estimate is based on the argument of a lag-1 autocorrelation function and the Doppler power estimate is based on the magnitude of a lag-0 autocorrelation function. Motion can also be estimated by known phase domain (e.g., parametric frequency estimators such as MUSIC or ESPRIT) or time domain (e.g., cross-correlation) signal processing techniques. Instead of or in addition to the velocity estimator, other estimators related to the time or spatial distribution of the velocity may be used, such as acceleration estimators or estimators of the temporal and / or spatial velocity derivatives. In some embodiments, the velocity and / or power estimates may be subjected to further post-processing such as thresholding to further reduce noise, segmentation, filling and smoothing. The velocity and / or power estimates are then mapped to a desired display color range according to a color map. The color data, also referred to as Doppler image data, is then coupled to a scan converter 230, where the Doppler image data is converted to a desired image format and overlaid on a B-mode image of the tissue structure to form a color Doppler or power Doppler image. In some examples, the power estimate (e.g., lag 0 autocorrelation information) is used to mask or segment the flow in the color Doppler (e.g., velocity estimate) before overlaying the color Doppler image on top of the B-mode image.

[0022]

[0032] Output from the scan converter 230, multiplanar reformatter 232, and / or volume renderer 234 are coupled to an image processor 236 for further enhancement, buffering, and temporary storage before being displayed on an image display 238. A graphics processor 240 generates graphic overlays that are displayed with the images. These graphic overlays may include standard identifying information, such as, for example, the patient's name, date and time of the image, imaging parameters, etc. For these purposes, the graphics processor 240 is configured to receive input, such as a typed patient name or other annotations (e.g., labels, body markers), from a user interface 224. The user interface 224 is also coupled to the multiplanar reformatter 232 for selection and control of the display of multiple multiplanar reconstructed (MPR) images.

[0023]

[0033] The ultrasound imaging system 200 includes a local memory 242. The local memory 242 is implemented as any suitable non-transitory computer-readable medium (e.g., flash drive, disk drive). The local memory 242 stores data generated by the system 200 including ultrasound images, executable instructions, imaging parameters, log files including usage data, training data sets, or any other information required for the operation of the system 200. In some examples, the local memory 242 includes multiple memories, which may be of the same or different types. For example, the local memory 242 may include a dynamic random access memory (DRAM) and a flash memory.

[0024]

[0034] As described above, the system 200 includes a user interface 224. The user interface 224 includes a display 238 and a control panel 252. The display 238 includes a display device implemented using various known display technologies, such as LCD, LED, OLED, or plasma display technology. In some embodiments, the display 238 includes multiple displays. The control panel 252 is configured to receive user input (e.g., type of exam, imaging parameters). The control panel 252 includes one or more hard controls (e.g., buttons, knobs, dials, encoders, mouse, trackball, etc.). In some embodiments, the control panel 252 additionally or alternatively includes soft controls (e.g., GUI control elements, or simply GUI controls) provided on a touch-sensitive display. In some embodiments, the display 238 is a touch-sensitive display that includes one or more soft controls of the control panel 252.

[0025]

[0035] In some embodiments, the various components shown in FIG. 2 are combined. For example, the image processor 236 and the graphics processor 240 are implemented as a single processor. In some embodiments, the various components shown in FIG. 2 are implemented as separate components. For example, the signal processor 226 can be implemented as multiple separate signal processors for each imaging mode (e.g., B-mode, Doppler). In other examples, the image processor 236 is implemented as multiple separate processors for different tasks and / or for parallel processing of one and the same task. In some embodiments, one or more of the various processors shown in FIG. 2 are implemented by a general-purpose processor and / or by a microprocessor configured to perform a specified task. In some examples, the processors are configured to provide instructions for a task from a non-transitory computer-readable medium (e.g., from the local memory 242). These instructions are then executed by the processor. In some embodiments, one or more of the various processors are implemented as application-specific circuitry. In some embodiments, one or more of the various processors (eg, image processor 236) are implemented with one or more graphical processing units (GPUs).

[0026]

[0036] According to examples of the present disclosure, one or more processors of system 200 , such as image processor 236 and / or graphics processor 240 , automatically or semi-automatically annotate ultrasound images acquired by system 200 .

[0027]

[0037] In some examples, the one or more processors include any one or more machine learning, artificial intelligence (AI) algorithms, and / or neural networks (collectively, AI models) that are trained to annotate ultrasound images. In some examples, the one or more processors include one or more of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), an autoencoder neural network, and / or a single-shot detector, etc. The AI ​​models are implemented as hardware components (e.g., neurons of the neural network are represented by physical components) and / or as software components (e.g., neurons and pathways are implemented as software applications). Neural networks implemented according to the present disclosure use various topologies and learning algorithms to train the neural network to produce desired outputs. For example, a software-based neural network may be implemented using a processor (e.g., a single or multi-core CPU, a single GPU or a GPU cluster, or multiple processors arranged for parallel processing) configured to execute instructions stored on a computer-readable medium that, when executed, cause the processor to perform a trained algorithm. In some examples, one or more processors perform the AI ​​in combination with other image processing or data analysis methods (e.g., segmentation, histogram analysis, statistical analysis).

[0028]

[0038] In various examples, the AI ​​model is trained using any of a variety of learning techniques now known or later developed to obtain a neural network (e.g., a trained algorithm, or a hardware-based node system) configured to analyze input data having the form of ultrasound images, usage data, probe tracking data, user inputs, measurements, and / or statistics. In some embodiments, the AI ​​is trained statically. That is, the AI ​​model is trained with a data set and deployed in system 200 and implemented by one or more processors. In some embodiments, the AI ​​model is trained dynamically. In these examples, the AI ​​model is trained with an initial data set and deployed in system 200. However, the AI ​​model is trained and modified based on ultrasound images acquired by system 200 even after the AI ​​model is deployed in the system and implemented by one or more processors.

[0029]

[0039] In some embodiments, the ultrasound imaging system 200 receives and stores usage data in a computer readable medium, such as the local memory 242. Examples of usage data include, but are not limited to, annotations added to ultrasound images from a cine buffer (e.g., storage) selected for acquisition, keystrokes, button presses, other hard control operations (e.g., turning dials, toggling switches), screen touches, other soft control operations (e.g., swiping, pinching), menu selection and navigation, and voice commands. In some examples, additional usage data may be received, such as the geographic location of the ultrasound system, the type of ultrasound probe used (e.g., type, make, model), a unique user identifier, the type of exam, and / or what object is currently being imaged by the ultrasound imaging system. In some examples, the usage data may be provided by a user via a user interface, such as the user interface 224, by a processor, such as the image processor 236 and / or the graphics processor 240, by the ultrasound probe (e.g., ultrasound probe 212), and / or may be preprogrammed and stored in the ultrasound imaging system (e.g., the local memory 242).

[0030]

[0040] In some examples, some or all of the usage data is written and stored in a computer-readable file, such as a log file, for later retrieval and analysis. In some examples, the log file stores a record of some or all of the interactions between the user and the ultrasound imaging system. The log file includes time and / or sequence data such that the time and / or sequence of different interactions the user had with the ultrasound imaging system can be determined. The time data includes a timestamp associated with each interaction (e.g., each keystroke, each annotation on an ultrasound image). In some examples, the log file stores interactions as a list in the order in which they occurred such that the sequence of interactions can be determined even if timestamps are not stored in the log file. In some examples, the log file indicates the particular user associated with the interactions recorded in the log file. For example, if a user logs into the ultrasound imaging system using a unique identifier (e.g., username, password), the unique identifier is stored in the log file. The log file is a text file, a spreadsheet, a database, and / or any other suitable file or data structure that can be analyzed by one or more processors. In some examples, one or more processors of the ultrasound imaging system collect usage data and write the usage data to one or more log files, which are stored in a computer-readable medium. In some examples, the log files and / or other usage data are received by the imaging system from one or more other imaging systems. The log files and / or other usage data are stored in local memory. The log files and / or other usage data are received by any suitable method, including wireless (e.g., BlueTooth, WiFi) and wired (e.g., Ethernet cable, USB device) methods. In some examples, usage data from one or more users and one or more imaging systems is used to automatically and / or semi-automatically annotate ultrasound images.

[0031]

[0041] FIG. 3 is a block diagram providing an overview of data flow in an ultrasound imaging system according to an example of the present disclosure. The anatomical awareness AI model 306 is implemented by one or more processors, such as the image processor 236 of the system 200. The anatomical awareness model 306 receives an ultrasound image 300. The ultrasound image 300 has been acquired by a probe, such as the probe 212. In some examples, the anatomical awareness AI model 306 is trained to identify one or more anatomical features in the ultrasound image 300. The anatomical features include organs, sub-regions of an organ, and / or features of an organ. In some examples, the anatomical awareness AI model 306 is trained to identify an imaging plane from which the ultrasound image 300 was acquired. In some examples, the anatomical awareness AI model 306 is trained to identify a position and / or orientation of an ultrasound probe when the ultrasound image 300 was acquired. In some examples, the anatomical awareness AI model 306 continuously attempts to recognize anatomical features in the ultrasound image 300 as a user scans with the probe. In some examples, the anatomical recognition AI model 306 waits until the user freezes the image.

[0032]

[0042] Optionally, the anatomical awareness AI model 306 also receives probe tracking data 302. The probe tracking data 302 is provided, at least in part, by a probe tracking device, such as the probe tracking device 270, and / or by a probe tracking system, such as the probe tracking system 272. The anatomical awareness AI model 306 analyzes the probe tracking data 302 to assist in determining anatomical features, image planes, ultrasound probe position, and / or ultrasound probe orientation.

[0033]

[0043] Optionally, the anatomical awareness AI model 306 also receives usage data 304. The usage data 304 is provided by a user interface, such as user interface 224, and / or a log file. The anatomical awareness AI model 306 analyzes the usage data 304 to assist in determining anatomical features, image planes, ultrasound probe position, and / or ultrasound probe orientation. For example, the usage data 304 includes previously acquired ultrasound images and / or annotations applied to previously acquired ultrasound images.

[0034]

[0044] The anatomical features, image plane, ultrasound probe location, and / or ultrasound probe orientation determined by the anatomical awareness AI model 306 are provided to an image annotation AI model 308. The image annotation AI model 308 is trained to apply one or more annotations 310 to the image 300 based at least in part on an analysis of the output of the anatomical awareness AI model 306. In the example shown in FIG 3, the image annotation AI model 308 provides the label 310 (trans aorta distal) to the image 300.

[0035]

[0045] Optionally, in some examples, the image annotation AI model 308 receives usage data 304. The image annotation AI model 308 analyzes the usage data 304 to help determine annotations to apply to the image 300. For example, the image annotation AI model 308 analyzes the usage data 304 to determine whether the user prefers to use "long" or "sagittal" in the annotation. In other examples, the image annotation AI model 308 analyzes the usage data 304 to determine whether the user prefers to use body markers, labels, and / or both as annotations. In further examples, the image annotation AI model 308 analyzes previously applied annotations included in the usage data 304 to determine annotations to apply to the current image.

[0036]

[0046] In some examples, the anatomical recognition AI model 306 performs image recognition and the image annotation AI model 308 applies annotations automatically without any user intervention. In some examples, the AI ​​models 306, 308 perform their respective tasks semi-automatically. In these examples, a user acquires an initial ultrasound image and manually applies annotations (e.g., labels, body markers, combinations thereof). The AI ​​models 306, 308 use the user-annotated images as "seeds" for making their respective decisions.

[0037]

[0047] In some examples, when the anatomical recognition AI model 306 makes a decision while the user is scanning, the image annotation AI model 308 provides annotations 310 to a display, such as display 238, during the scan. Thus, the annotations 310 change as the user scans to different positions and / or orientations. However, some users may find the changing annotations 310 uncomfortable. Thus, in some examples, the annotations 310 are provided when the probe tracking data 302 indicates that the probe is stationary. In other examples, the annotations 310 are provided when the user freezes the current image.

[0038]

[0048] Once the image 300 and annotations 310 are provided on the display, the user accepts the annotations 310 by saving the annotated image 300 unchanged. If the user believes the annotations 310 are incorrect and / or prefers a different style of annotation 310 (atrium vs. atrial appendage, superior vs. upper), the user can remove and / or modify the annotations 310 via a user interface before saving the image 300. In some examples, whether the user accepts the annotations 310 provided by 308 or modifies the annotations 310 is stored by the ultrasound imaging system and used to train the image annotation AI model 308 and / or the anatomical recognition AI model 306.

[0039]

[0049] 3 shows two separate AI models, in some examples, the anatomical recognition AI model 306 and the image annotation AI model 308 are implemented as a single AI model that performs the tasks of both AI models 306, 308. In some examples, the anatomical recognition AI model 306 and the image annotation AI model 308 implement a combination of AI models and analytical techniques not traditionally thought of as AI models (e.g., statistical analysis) to make decisions.

[0040]

[0050] In some examples, the usage data (e.g., usage data stored in one or more log files) is analyzed by statistical methods. A graphical representation of an example of a statistical analysis of one or more log files according to an example of the present disclosure is shown in FIG. 4. A processor 400 of an ultrasound imaging system, such as ultrasound imaging system 200, receives one or more log files 402 for analysis. The processor 400, in some examples, is implemented by the image processor 236 and / or the graphics processor 240. In some examples, the processor 400 implements one or more models, such as AI model 306 and / or AI model 308. The processor 400 analyzes the usage data in the log files 402 to calculate various statistics related to user input (e.g., annotations, acquired ultrasound images) and provide one or more outputs 404. 4, the processor 400 determines the total number of times that one or more annotations (e.g., annotation A, annotation B, annotation C) were selected (e.g., selected on a control panel and / or menu) and / or accepted by one or more users (e.g., the ultrasound imaging system automatically or semi-automatically applied the annotation and the annotation was not changed by the user) and the percentage likelihood that each of the one or more annotations will be selected and / or accepted. In some examples, the percentage likelihood is based on the total number of times a particular annotation was selected divided by the total number of annotation selections.

[0041]

[0051] In some examples, the output 404 of the processor 400 is used to determine a user's preferred language and / or graphics for annotation. For example, the output 404 is used to determine whether a user prefers to use "RT" or "RIGHT" as a label to annotate an image of the right kidney.

[0042]

[0052] A graphical representation of another example of statistical analysis of one or more log files according to an example of the present disclosure is shown in FIG. 5. A processor 500 of an ultrasound imaging system, such as ultrasound imaging system 200, receives one or more log files 502 for analysis. The processor 500, in some examples, is implemented by the image processor 236 and / or the graphics processor 240. In some examples, the processor 500 implements one or more models, such as AI model 306 and / or AI model 308. The processor 500 analyzes the usage data in the log files 502 to calculate various statistics related to annotations selected and / or accepted by one or more users and provide one or more outputs 504, 506. As shown in FIG. 5, the processor 500 analyzes the log files to determine one or more sequences of annotation selection / acceptance. The processor 500 may use a moving window to search the sequence, searching for a particular command and / or annotation selection that indicates the start of the sequence (e.g., "freeze", "inspection type", etc.), and / or other methods (e.g., the sequence ends when the time interval between annotation selections exceeds a maximum duration). For one or more annotation selections that start a sequence, the processor 500 calculates the percentage probability of the next annotation being selected. For example, as shown in output 504, when annotation A is applied at the start of a sequence, the processor 500 calculates the probability (e.g., percentage probability) that one or more other annotations (e.g., annotations B, C, etc.) will be selected / accepted next in the sequence. As shown in output 504, the processor 500 calculates the probability that one or more annotations (e.g., annotation D, annotation E, etc.) will be selected after one or more of the other controls selected after annotation A. This calculation of probabilities continues for any desired length of the sequence.

[0043]

[0053] Based on the output 504, the processor 500 calculates the most likely sequence of annotations selected and accepted by the user. As shown in output 506, it is determined that after annotation A is applied by the user to the ultrasound image, annotation B has the highest probability of being selected by the user, and after annotation B is selected by the user, annotation C has the highest probability of being applied by the user.

[0044]

[0054] In some examples, the output 506 of the processor 500 is used to determine one or more annotations to be applied to the ultrasound image based at least in part on previously applied annotations. In some applications, this allows automatic and / or semi-automatic annotations to be tailored to clinical and / or individual user specific protocols.

[0045]

[0055] Analysis of the log files, including the examples of statistical analysis described with reference to Figures 4 and 5, can occur as the usage data is received and recorded in the log file (e.g., live capture) and / or the analysis can occur at a later time (e.g., pause in workflow, end of exam, user logs off). Although statistical analysis of the log files has been described, in some examples, one or more processors of the ultrasound imaging system (e.g., image processor 236, graphics processor 240) implement one or more trained AI models to analyze the usage data, whether as a log file or in some other format (e.g., live capture before being recorded in the log file).

[0046]

[0056] Examples of AI models used to analyze the usage data and / or ultrasound images include, but are not limited to, decision trees, convolutional neural networks, and long short-term memory (LSTM) networks. In some examples, probe tracking data, such as probe tracking data 302, is also analyzed by one or more AI models. In some examples, the use of one or more AI models allows for faster and / or more accurate determination of anatomical features in the ultrasound images, imaging planes, probe positions, probe orientations, and / or annotations applied to the ultrasound images as compared to non-AI model techniques.

[0047]

[0057] FIG. 6 is an illustration of a neural network used to analyze data according to examples of the present disclosure. In some examples, the neural network 600 is implemented by one or more processors (e.g., image processor 236, graphics processor 240) of an ultrasound imaging system, such as ultrasound imaging system 200. In some examples, the neural network 600 is included in AI model 306 and / or AI model 08. In some examples, the neural network 600 is a convolutional network with single and / or multi-dimensional layers. The neural network 600 includes one or more input nodes 602. In some examples, the input nodes 602 are organized as layers of the neural network 600. The input nodes 602 are coupled by weights 604 to one or more layers 608 of hidden units 606. In some examples, the hidden units 606 perform operations on one or more inputs from the input nodes 602 based at least in part on the associated weights 604. In some examples, hidden unit 606 is coupled to one or more layers 614 of hidden units 612 by weights 610. Hidden unit 612 performs an action on one or more outputs from hidden unit 606 based at least in part on weights 610. The output of hidden unit 612 is provided to output node 616 to provide an output (e.g., an inference, decision, prediction) of neural network 600. Although one output node 616 is shown in FIG. 6, in some examples, the neural network has multiple output nodes 616. In some examples, the output is accompanied by a confidence level. The confidence level is a value from 0 to 1 inclusive, where a confidence level of 0 indicates that neural network 600 has no confidence that the output is correct, and a confidence level of 1 indicates that the neural network has 100% confidence that the output is correct.

[0048]

[0058] In some examples, inputs to the neural network 600 provided at one or more input nodes 602 include log files, live capture usage data, probe tracking data, and / or images acquired by the ultrasound probe. In some examples, outputs provided at output node 616 include predictions of the next annotations to be applied to the image, predictions of annotations likely to be used by a particular user, annotations likely to be used during a particular exam type, and / or annotations likely to be used when a particular anatomical feature is being imaged. In some examples, outputs provided at output 616 include a determination of one or more anatomical features (e.g., an organ, a subregion of an organ, and / or an organ feature) in the ultrasound image, an imaging plane from which the ultrasound image was acquired, and a position and / or orientation of the ultrasound probe when the ultrasound image was acquired.

[0049]

[0059] The output of the neural network 600 is used by the ultrasound imaging system to automatically or semi-automatically apply annotations to ultrasound images.

[0050]

[0060] FIG. 7 is an illustration of a long short-term memory (LSTM) cell used to analyze data according to examples of the present disclosure. In some examples, the LSTM model is implemented by one or more processors (e.g., image processor 236, graphics processor 240) of an ultrasound imaging system such as ultrasound imaging system 200. The LSTM model is a type of recurrent neural network that can learn long-term dependencies. Thus, the LSTM model is suitable for analyzing and predicting sequences, such as sequences of applied annotations, acquired images, and / or ultrasound probe movements. The LSTM model typically includes multiple cells that are interconnected. The number of cells is based, at least in part, on the length of the sequence analyzed by the LSTM. For simplicity, only one cell 700 is shown in FIG. 7. In some examples, the LSTM including the cell 700 is included in the AI ​​model 306 and / or the AI ​​model 308.

[0051]

[0061] The variable C moving across the top of the cell 700 is the cell state. The previous LSTM cell state C t-1 is provided as an input to cell 700. Data can be selectively added to or removed from the cell state by cell 700. The addition or removal of data is controlled by three "gates", each of which comprises a separate neural network layer. The modified or unmodified state of cell 700 is passed by cell 700 to the next LSTM cell, C t Provided as.

[0052]

[0062] The variable h moving across the bottom of cell 700 is the hidden state vector of the LSTM model. t-1 is provided as an input to cell 700. The hidden state vector h t-1 is the current input x to the LSTM model provided in cell 700. t The hidden state vector may also be modified by the cell state 700C. t The modified hidden state vector 700 of the cell 700 can be modified based on the output h t The output h t is provided as the hidden state vector to the next LSTM cell and / or as the output of the LSTM model.

[0053]

[0063] Turning now to the internal functioning of cell 700, a first gate (e.g., a forget gate) for controlling the state of cell C includes a first layer 702. In some examples, this first layer is a sigmoid layer. The sigmoid layer stores the hidden state vector h t-1 and the current input x t The first layer 702 receives the concatenation of the output f t However, this output f t contains weights that indicate which data from the previous cell state should be "forgotten" and which data from the previous cell state should be "remembered" by the cell 700.t-1 In order to remove any data determined to be forgotten by the first layer 702, f t is multiplied.

[0054]

[0064] The second gate (e.g., an input gate) includes a second layer 706 and a third layer 710. Both the second layer 706 and the third layer 710 input a hidden state vector h t-1 and the current input x t In some examples, the second layer 706 is a sigmoid function. The second layer 706 outputs i t The third layer 710, in some examples, includes a tanh function. The third layer 710 provides t-1 and x t A vector containing all possible data that can be added to the cell state from

number

[0055]

[0065] The third gate (e.g., an output gate) includes a fourth layer 714. In some examples, the fourth layer 714 is a sigmoid function. The fourth layer 714 outputs a hidden state vector h t-1 and the current input x t and cell state C t Which data is the hidden state vector h of cell 700? t The output o includes weights that indicate which t Cell state C tThe data is converted to a vector by the tanh function at operation 716, and then converted to a vector by operation 718. t is multiplied to obtain the hidden state vector / output vector h t In some cases, the output vector h t is accompanied by a confidence value, similar to the output of a convolutional neural network, such as that described with reference to FIG. 6.

[0056]

[0066] As depicted in FIG. 7, cell 700 is the “middle” cell. That is, cell 700 receives the input C t-1 and t-1 and output C t and t If cell 700 is the first cell in the LSTM, it provides the input x t If cell 700 is the last cell in the LSTM, it only receives the output h t and C. t and are not provided to other cells.

[0057]

[0067] In some examples where a processor in an ultrasound imaging system (e.g., image processor 236, graphics processor 240) implements an LSTM model, the current input x t contains data related to the ultrasound images, annotations applied to the ultrasound images, and / or probe tracking data. t-1 The cell state C includes data related to the prior prediction, such as annotations, anatomical features in the ultrasound image, the position and / or orientation of the ultrasound probe, and / or the image plane. t-1 contains data related to previous annotations selected and / or accepted by a user. In some examples, the output of the LSTM model, h t is used by this processor or another processor in the ultrasound imaging system to apply annotations to the ultrasound image.

[0058]

[0068] 8 is an illustration of a decision tree used to analyze data according to examples of the present disclosure. In some examples, decision tree 800 is implemented by one or more processors (e.g., image processor 236, graphics processor 240) of an ultrasound imaging system, such as ultrasound imaging system 200. In some examples, decision tree 800 is included in AI model 306 and / or AI model 308.

[0059]

[0069] The decision tree 800 includes multiple layers 804, 806, 808 of tests to make decisions regarding an input 802. The tests applied at a particular layer are based at least in part on decisions made in previous layers. In the example shown in FIG. 8, the decision tree 800 is implemented to analyze ultrasound image data, and ultrasound images are provided as input 802. However, in other examples, the decision tree 800 is implemented to analyze other types of data (e.g., usage data, probe tracking data) or additional types of data. For example, the input 802 includes both ultrasound images and probe tracking data.

[0060]

[0070] In the first decision layer 804, an organ is identified in the ultrasound image provided as input 802. In some examples, the first decision layer 804 includes an AI model, such as neural network 600, to determine the organ in the image. In the second decision layer 806, one or more additional decisions are made depending on which organ was identified in the decision layer 804. For example, if the liver was identified, in the second decision layer 806, a decision is made regarding whether the ultrasound image was acquired along a long (e.g., sagittal) or transverse plane of the liver, whether the image was acquired from the left to the right side of the liver, whether the inferior vena cava or aorta is visible in the image, and / or whether portal hypertension or variceal bleeding is present. In some examples, the one or more decisions are made using one or more AI models (e.g., neural networks, LSTMs). In some examples, the same data (e.g., ultrasound images) used in the first decision layer 804 is used. In some examples, different and / or additional data is used to make the decision in the second decision layer 806. For example, probe tracking data is used in addition to the ultrasound image.

[0061]

[0071] Depending on the decisions made in the second decision layer 806, additional decisions are made in the third decision layer 808. For example, once the long or transverse plane is determined, it is determined whether it is the superior, middle, or inferior region of the liver, or whether it is the middle, lateral, or intermediate region of the liver. Similar to the first and second decision layers 804, 806, the third decision layer 808 uses one or more AI models to make one or more of the decisions.

[0062]

[0072] Once all the decisions have been made, one or more of the decisions of one or more of the decision layers 804, 806, 808 are provided as output. In some examples, this output is used to determine annotations to apply to the ultrasound image. In the example provided in FIG. 8, three decision layers are shown, but the decision tree 800 may include more or fewer decision layers in other examples.

[0063]

[0073] As described herein, an AI model (e.g., AI model 306, AI model 308, neural network 600, LSTM including cells 700, and decision tree 800) provides a confidence level associated with one or more outputs. In some examples, a processor (e.g., image processor 236, graphics processor 240, etc.) provides annotations to the ultrasound image only if the confidence level associated with the output is equal to or exceeds a threshold (e.g., greater than 50%, greater than 70%, greater than 90%, etc.). In some examples, if the confidence level is below the threshold, the processor cannot apply the annotations to the ultrasound traces. In some examples, this means doing nothing and / or prompting the user to manually provide annotations.

[0064]

[0074] Although convolutional neural networks, LSTM models, and decision trees have been described herein, these AI models are provided merely as examples, and the principles of the present disclosure are not limited to these particular models. Additionally, in some examples, statistical analysis techniques, such as those described with reference to Figures 4 and 5, are used in combination with one or more of the AI ​​models to analyze the data. In some examples, the AI ​​models and / or statistical analysis techniques are implemented, at least in part, by one or more processors executing computer-readable instructions. The computer-readable instructions are provided to the one or more processors by a non-transitory computer-readable memory, such as the local memory 242.

[0065]

[0075] FIG. 9 illustrates a block diagram of a process for training and deploying a neural network according to the principles of the present disclosure. The process illustrated in FIG. 9 is used to train an AI model implemented by a medical imaging system, such as the AI ​​models 306, 308 illustrated in FIG. 3. Phase 1, the left side of FIG. 9, illustrates training the A model. To train an AI model, a training set containing multiple examples of input data and output classifications is provided to the AI ​​model's training algorithm (e.g., the AlexNet training algorithm as described in Krizhevsky, A., Sutskever, I., and Hinton, GE, "ImageNet Classification with Deep Convolutional Neural Networks," NIPS 2012 or supplements thereto). Training includes selecting a starting architecture 912 and preparing training data 914. The starting architecture 912 is a blank architecture (e.g., an architecture with defined layers and node arrangements but no previously trained weights) or a partially trained model, such as an Inception network, which is further adapted for classification of ultrasound images, tracking data, and / or usage data. The starting architecture 912 (e.g., blank weights) and training data 914 are provided to a training engine 910 (e.g., an ADAM optimizer) to train the model. Once a sufficient number of iterations have been performed (e.g., when the model operates consistently within an acceptable error), the model 920 is said to be trained and ready for deployment, which is illustrated in the middle of FIG. 9 as Phase 2. On the right side of FIG. 9, Phase 3, the trained model 920 is applied (via an inference engine 930) to analyze new data 932, which is data that was not provided to the model during the initial training (in Phase 1).For example, new data 932 includes unknown images, such as live ultrasound images acquired during a patient scan, and / or annotations manually applied to the unknown images by a user. The trained model 920 implemented via engine 930 is used to classify the unknown images according to the training of the model 920 and provide output 934 (e.g., anatomical features, image plane, probe position and / or orientation, annotations). The output 934 is then used by the system for further processes 940 (e.g., applying annotations, generating annotated images). In some examples where the trained model is dynamically trained, additional data, shown as field training 938, is provided to the inference engine 930. The additional data includes new data 932, data indicating whether the user accepted or modified the annotations provided, and / or other data.

[0066]

[0076] In embodiments in which the trained model 920 is used to implement a neural network executed by a processor such as the image processor 236 and / or the graphics processor 240, the starting architecture is a convolutional neural network or a deep convolutional neural network architecture. The training data 914 includes a plurality (hundreds, often thousands or even more) of annotated images, associated usage data, and / or associated probe tracking data.

[0067]

[0077] FIG. 10 is a flowchart of a method according to an example of the present disclosure. Method 1000 is a method for automatically or semi-automatically annotating images. In some examples, method 1000 is performed by an ultrasound imaging system, such as ultrasound imaging system 200. In some examples, all or a portion of method 1000 is performed by one or more processors, such as image processor 236 and / or graphics processor 240. In some examples, one or more processors implement one or more AI models, such as those shown in FIGS. 3 and 6-8, to perform part or all of method 1000. In some examples, one or more processors implement one or more statistical analysis techniques, such as those shown in FIGS. 4 and 5, to perform part or all of method 1000.

[0068]

[0078] One or more processors of the ultrasound imaging system receive an ultrasound image acquired by an ultrasound probe, as indicated by block 1002. Based at least in part on the ultrasound image, the one or more processors determine anatomical features present in the ultrasound image, as indicated by block 1004. In some examples, the one or more processors further determine at least one of an imaging plane, a position of the ultrasound probe, or an orientation of the ultrasound probe, as indicated by block 1006. The one or more processors determine annotations to be applied to the ultrasound image, as indicated by block 1008. The determined annotations are based at least in part on the determined anatomical features and at least one of the determined imaging plane, the determined position of the ultrasound probe, or the determined orientation of the ultrasound probe. The one or more processors provide the ultrasound image and annotations to at least one of a non-transitory computer-readable medium for display or storage, as indicated by block 1010.

[0069]

[0079] In some examples, the one or more processors further receive probe tracking data and / or usage data to make the decisions at blocks 1004, 1006, and / or 1008. In some examples, one or more of these decisions are made by one or more AI models and / or statistical analysis methods.

[0070]

[0080] 11 is a block diagram illustrating an exemplary processor 1100 according to the principles of the present disclosure. The processor 1100 may be used to implement one or more of the processors and / or controllers described herein, such as, for example, the image processor 236 shown in FIG. 2 and / or any other processors shown in FIG. 2. The processor 1100 may be any suitable type of processor, including, but not limited to, a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable array (FPGA) programmed to form a processor, a graphical processing unit (GPU), an application specific circuit (ASIC) designed to form a processor, or a combination thereof.

[0071]

[0081] The processor 1100 includes one or more cores 1102. The cores 1102 include one or more arithmetic logic units (ALUs) 1104. In some embodiments, the cores 1102 include a floating point logic unit (FPLU) 1106 and / or a digital signal processing unit (DSPU) 1108 in addition to or instead of the ALUs 1104.

[0072]

[0082] The processor 1100 includes one or more registers 1112 communicatively coupled to the core 202. The registers 1112 are implemented using dedicated logic gate circuits (e.g., flip-flops) and / or any memory technology. In some embodiments, the registers 1112 are implemented using static memory. The registers provide data, instructions, and addresses to the core 1102.

[0073]

[0083] In some embodiments, the processor 1100 includes one or more levels of cache memory 1110 communicatively coupled to the core 1102. The cache memory 1110 provides computer-readable instructions to the core 1102 for execution. The cache memory 1110 provides data for processing by the core 1102. In some embodiments, the computer-readable instructions are provided to the cache memory 1110 by a local memory, such as a local memory attached to an external bus 1116. The cache memory 1110 is implemented using any suitable cache memory type, for example, metal-oxide-semiconductor (MOS) memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), and / or any other suitable memory technology. A computer-readable medium, such as those described above, stores instructions that, when executed by the processor, deploy the techniques disclosed herein.

[0074]

[0084] The processor 1100 includes a controller 1114 that controls input to the processor 1100 from other processors and / or components in the system (e.g., the control panel 252 and the scan converter 230 shown in FIG. 2) and / or output from the processor 1100 to other processors and / or components in the system (e.g., the display 238 and the volume renderer 234 shown in FIG. 2). The controller 1114 controls the data paths in the ALU 1104, the FPLU 1106 and / or the DSPU 1108. The controller 1114 is implemented as one or more state machines, data paths and / or dedicated control logic. The gates of the controller 1114 are implemented as stand-alone gates, FPGAs, ASICs, or any other suitable technology.

[0075]

[0085] The registers 1112 and cache memory 1110 communicate with the controller 1114 and the core 1102 via internal connections 1120A, 1120B, 1120C and 1120D. The internal connections may be implemented as a bus, a multiplexer, a crossbar switch, and / or any other suitable connection technology.

[0076]

[0086] Input and output for the processor 1100 are provided via a bus 1116, which may include one or more conductive lines. The bus 1116 is communicatively coupled to one or more components of the processor 1100, such as the controller 1114, the cache 1110, and / or the registers 1112. The bus 1116 is coupled to one or more components of the system, such as the display 238 and the control panel 252 described above.

[0077]

[0087] The bus 1116 is coupled to one or more external memories. The external memory includes a read-only memory (ROM) 1132. The ROM 1132 may be a masked ROM, an electronically programmable read-only memory (EPROM), or any other suitable technology. The external memory includes a random access memory (RAM) 1133. The RAM 1133 may be a static RAM, a battery backed static RAM, a dynamic RAM (DRAM), or any other suitable technology. The external memory includes an electrically erasable programmable read-only memory (EEPROM) 1135. The external memory includes a flash memory 1134. The external memory includes a magnetic storage device, such as a disk 1136. In some embodiments, the external memory is included in a system, such as the ultrasound imaging system 200 shown in FIG. 2, such as, for example, the local memory 242.

[0078]

[0088] The systems, methods, and devices disclosed herein automatically and / or semi-automatically apply annotations, such as labels and / or body markers, to ultrasound images, which in some embodiments reduces examination time by reducing the time required for a user to manually apply annotations.

[0079]

[0089] Although the examples described herein discuss the processing of ultrasound image data, it will be understood that the principles of the present disclosure are not limited to ultrasound but may also be applied to image data from other modalities, such as magnetic resonance imaging and computed tomography.

[0080]

[0090] In various embodiments in which the components, systems, and / or methods are implemented using a programmable device, such as a computer-based system or programmable logic circuit, it should be understood that the above-described systems and methods can be implemented using any of a variety of known or later developed programming languages, such as "C", "C++", "C#", "JAVA", "Python", etc. Accordingly, various storage media, such as magnetic computer disks, optical disks, electronic memory, etc., containing information capable of instructing a device, such as a computer, can be provided to implement the above-described systems and / or methods. Once an appropriate device has access to the information and programs contained in the storage medium, the storage medium can provide the information and programs to the device, thus enabling the device to perform the functions of the systems and / or methods described herein. For example, if a computer is provided with a computer disk containing appropriate material, such as source files, object files, executable files, etc., the computer can receive the information and appropriately configure itself to perform the various functions and perform the functions of the various systems and methods outlined in the figures and flow charts above. That is, the computer receives various pieces of information relating to different elements of the systems and / or methods described above from the disk and performs the individual systems and / or methods, coordinating the functions of the individual systems and / or methods described above.

[0081]

[0091] In view of this disclosure, it should be noted that the various methods and devices described herein can be implemented as hardware, software, and / or firmware. Furthermore, the various methods and parameters are included merely as examples and not in any limiting sense. In view of this disclosure, one skilled in the art can implement the present teachings in determining his or her own techniques and the equipment required to effect these techniques while remaining within the scope of the present invention. One or more functions of the processors described herein may be incorporated into fewer or a single processing unit (e.g., a CPU) or may be implemented using an application specific integrated circuit (ASIC) or a general purpose processing circuit that is programmed in response to executable instructions to perform the functions described herein.

[0082]

[0092] Although the present system has been described with particular reference to an ultrasound imaging system, it is envisioned that the present system is extendable to other medical imaging systems in which one or more images are systematically acquired. Thus, the present system may be used to acquire and / or record image information relating to, but not limited to, the kidneys, testes, breasts, ovaries, uterus, thyroid, liver, lungs, musculoskeletal system, spleen, heart, arteries and vascular system, as well as for other imaging applications relating to ultrasound-guided interventions. Additionally, the present system includes one or more programs for use with conventional imaging systems to provide the features and advantages of the present system. Certain additional advantages and features of the present disclosure will become apparent to those skilled in the art upon review of the present disclosure, or may be experienced by those employing the novel systems and methods of the present disclosure. Another advantage of the present systems and methods is that conventional medical imaging systems can be easily upgraded to incorporate the features and advantages of the present systems, devices, and methods.

[0083]

[0093] Of course, it should be understood that any one of the examples, embodiments, or processes described herein may be combined with one or more other examples, embodiments, and / or processes, or may be separated and / or performed among separate devices or device portions according to the present systems, devices, and methods.

[0084]

[0094] Finally, the foregoing discussion is intended to be merely illustrative of the present system and method, and should not be construed as limiting the appended claims to any particular embodiment or group of embodiments. Thus, while the present system has been described in particular detail with reference to exemplary embodiments, it should also be recognized that numerous modifications and alternative embodiments may be devised by those skilled in the art without departing from the broader intended spirit and scope of the present system as set forth in the following claims. Accordingly, the specification and drawings are to be regarded as illustrative, and are not intended to limit the scope of the appended claims.

Claims

1. An ultrasonic imaging system configured to annotate an ultrasonic image, the ultrasonic imaging system comprising: an ultrasonic probe that acquires an ultrasonic signal; a processing circuit that generates an ultrasonic image from the ultrasonic signal; a non-transitory computer-readable medium in which instructions are encoded, the non-transitory computer-readable medium storing the ultrasonic image; at least one processor that communicates with the non-transitory computer-readable medium and executes the instructions; wherein when the instructions are executed, the ultrasonic imaging system is caused to: determine anatomical features present in the ultrasonic image; determine at least one of an imaging plane, a position of the ultrasonic probe, or an orientation of the ultrasonic probe; determine an annotation to be applied to the ultrasonic image based at least in part on the determined anatomical features and at least one of the determined imaging plane, the determined position of the ultrasonic probe, or the determined orientation of the ultrasonic probe.

2. The ultrasonic imaging system according to claim 1, further comprising a display that displays the ultrasonic image, wherein the at least one processor further provides the annotation for display on the ultrasonic image.

3. The ultrasonic imaging system according to claim 2, further comprising a user interface that enables a user to change the annotation provided by the at least one processor.

4. The ultrasonic imaging system according to claim 1, further comprising a user interface that receives an input from a user, the input being stored as usage data in the non-transitory computer-readable medium, and the annotation being determined based on the usage data.

5. The ultrasonic imaging system according to claim 4, wherein the annotation is determined based on a statistical analysis of the usage data.

6. The instructions, in response to execution on the processor, cause the processor to: in response to a determination that the ultrasonic probe is moving, cause at least one of (i) removal of the annotation from the display and (ii) cessation of providing the annotation to the display. In response to a determination that the ultrasonic probe is stationary, causing at least one of (i) a return of the annotation to the display and (ii) a re - provision to the display, the ultrasonic imaging system according to claim 2.

7. In response to a determination that the ultrasonic probe is at at least one of a second determined orientation or a second determined position with respect to at least one of the determined anatomical feature or an identification of a second determined anatomical feature, the provision of the annotation changes, the ultrasonic imaging system according to claim 2.

8. The at least one processor implements one or more artificial intelligence (AI) models to determine at least one of the annotation, the anatomical feature, the imaging plane, the position of the ultrasonic probe, or the orientation of the ultrasonic probe, the ultrasonic imaging system according to claim 1.

9. The one or more AI models include a neural network, the ultrasonic imaging system according to claim 8.

10. The one or more AI models include a decision tree, the ultrasonic imaging system according to claim 8.

11. A process - tracking device coupled to the ultrasonic probe, the ultrasonic imaging system according to claim 1, further comprising the process - tracking device that provides process - tracking data to the at least one processor to determine at least one of the anatomical feature, the imaging plane, the position of the ultrasonic probe, or the orientation of the ultrasonic probe.

12. The process - tracking device is an electromagnetic probe - tracking device, the ultrasonic imaging system according to claim 11.

13. The processing circuit is included in the ultrasonic probe, the ultrasonic imaging system according to claim 1.

14. The annotation includes graphical body markers, the ultrasonic imaging system according to claim 1.

15. A method for annotating an ultrasonic image, the method comprising: Receiving an ultrasonic image acquired by an ultrasonic probe coupled to a processing circuit; Using at least one processor to determine an anatomical feature present in the ultrasonic image; Using at least one processor to determine at least one of an imaging plane, the position of the ultrasonic probe, or the orientation of the ultrasonic probe; Using the at least one processor, determining an annotation to apply to the ultrasound image based at least in part on the determined anatomical feature and at least one of the determined imaging plane, the determined position of the ultrasound probe, or the determined orientation of the ultrasound probe; A method having the above. **Claim 16** The method according to claim 15, further comprising receiving probe tracking data from a probe tracking device coupled to the ultrasound probe, wherein determining the annotation is based on the probe tracking data. **Claim 17** The method according to claim 15, wherein determining at least one of the annotation, the anatomical feature, the imaging plane, the position of the ultrasound probe, or the orientation of the ultrasound probe is performed by one or more artificial intelligence (AI) models. **Claim 18** The method according to claim 17, wherein the one or more AI models include at least one of a neural network, a short-term memory model, or a decision tree. **Claim 19** The method according to claim 15, further comprising receiving usage data from at least one of a user interface or a non-transitory computer-readable medium, wherein determining the annotation is based on the usage data. **Claim 20** A computer-readable medium for annotating an ultrasound, which, when executed on a processor, causes the processor to: receive an ultrasound image acquired by an ultrasound probe coupled to a processing circuit; using at least one processor, determine an anatomical feature present in the ultrasound image; using the at least one processor, determine at least one of an imaging plane, a position of the ultrasound probe, or an orientation of the ultrasound probe; using the at least one processor, determine an annotation to apply to the ultrasound image based at least in part on the determined anatomical feature and at least one of the determined imaging plane, the determined position of the ultrasound probe, or the determined orientation of the ultrasound probe, the computer-readable medium comprising instructions.

Images