Method and system for enabling context-aware ultrasound scanning

Abstract

The present invention is titled "Methods and Systems for Enabling Context-Aware Ultrasound Scanning". The present invention provides various methods and systems for generating context-aware graphics of medical scan images. In one example, the context-aware graphics include relative size and relative position annotations regarding one or more internal anatomical features in the scan image to enable a user to determine a current scan plane and also guide the user to a target scan plane.

Description

Technical Field

[0001] Embodiments of the subject matter disclosed herein relate to medical imaging modalities and, more particularly, to systems and methods for ultrasound imaging. Background Art

[0002] Medical imaging systems are commonly used to detect internal features of a subject, such as a patient, and obtain internal physiological information. For example, a medical imaging system can be used to obtain scanned images of a subject's bone features, brain, heart, lungs, and various other features. Medical imaging systems can include ultrasound systems, magnetic resonance imaging (MRI) systems, computed tomography (CT) systems, x-ray systems, and various other imaging modalities. Scanned images can show several anatomical regions, one or more of which can be evaluated during a medical scan. Summary of the Invention

[0003] In one embodiment, a method for a medical imaging processor includes: acquiring a medical scan image; identifying one or more internal features in the medical scan image; generating a context-aware graphic based on relative sizes and relative positions of the one or more internal features; and displaying the context-aware graphic on a display portion of a user interface communicatively coupled to the medical imaging processor; wherein the context-aware graphic includes relative position annotations and relative size annotations for each of the one or more internal features identified in the medical scan image.

[0004] In this way, the context-aware graphic of the scanned image generated during a medical scan provides internal feature information and the relationships between the internal features visible in the scanned image. Thus, the context-aware graphic helps a user to locate the scan plane within the human anatomy.

[0005] It should be understood that the above summary is provided to introduce in a simplified form a series of concepts that are further described in the detailed description. This is not meant to identify key or essential features of the claimed subject matter, the scope of which is uniquely defined by the claims that follow the detailed description. Moreover, the claimed subject matter is not limited to embodiments that solve any disadvantages noted above or in any part of this disclosure. Brief Description of the Drawings

[0006] The present invention will be better understood from the following description of non-limiting embodiments, read in conjunction with the accompanying drawings, in which:

[0007] Figure 1 A block diagram of an exemplary embodiment of an ultrasound system is shown;

[0008] Figure 2is a schematic diagram showing a system for segmenting a scanned image and generating a context-aware graphic of the scanned image according to an exemplary embodiment;

[0009] Figure 3 is a schematic diagram showing the layout of an exemplary deep learning network that can be used to segment a scanned image and identify one or more internal features in the scanned image according to an exemplary embodiment;

[0010] Figure 4 shows a schematic flowchart according to an exemplary embodiment, which shows a method for displaying a scanned image;

[0011] Figure 5 shows a flowchart according to an exemplary embodiment, which shows a method for generating a context-aware graphic of a scanned image;

[0012] Figure 6 shows a flowchart according to an exemplary embodiment, which shows a method for generating a context-aware graphic in real time during an ultrasound scan;

[0013] Figure 7 shows a flowchart according to an exemplary embodiment, which shows a method for generating one or more desired transformation graphics during an ultrasound scan to achieve a target scan plane;

[0014] Figure 8A shows an exemplary set of two ultrasound scan images that do not have the identified internal features and do not have the represented context-aware graphic.

[0015] Figure 8B shows an exemplary set of ultrasound scan images of 8A that have internal feature annotations but do not have a context-aware graphic.

[0016] Figure 8C shows an exemplary set of ultrasound scan images of 8B that include a context-aware graphic.

[0017] Figure 9 shows an exemplary initial context-aware graphic and a target context-aware graphic for an initial scan image and a target scan image respectively according to an exemplary embodiment, and one or more desired transformation graphics to obtain the target scan image;

[0018] Figure 10 shows exemplary graphics according to an exemplary embodiment, which show the visual change in the size and positioning of a first internal feature relative to a second internal feature and the corresponding correlation with the scan direction during an ultrasound scan; and

[0019] Figure 11A flowchart according to an exemplary embodiment is shown, which shows an exemplary method for warning a user of a probe movement direction during an ultrasound scan based on context-aware graphics of a current scan image and a target scan image. DETAILED DESCRIPTION

[0020] The following description relates to various embodiments for enabling context awareness of one or more internal features (such as anatomical features) in a medical scan image. For example, during an ultrasound scan, the scan image displayed to the user may show one or more internal anatomical features. The user may adjust the ultrasound probe based on the displayed image to reach a desired scan plane (alternatively referred to herein as a target scan plane) to evaluate the internal features. To reach the target scan plane, the user may need to interpret the displayed scan image to identify the visible internal features in the current scan plane, visualize the current scan plane within the larger human anatomy, and determine the direction to move the probe.

[0021] Previous methods have employed artificial intelligence-based machine learning methods to learn the appearance of various anatomical features in the scan image and display a segmentation graphic that identifies the visible internal features in the ultrasound scan image. However, the segmentation graphic only provides the identification of the internal features. The operator may not be able to visualize the scan plane relative to the entire human anatomy and may thus move the probe in a direction away from the target scan plane. Therefore, it may take longer to reach the target scan plane. In addition, the operator may not be able to correctly identify the target scan plane even when imaging at the target scan plane or may not be able to reach the target scan plane during the scan. Therefore, even though the segmentation graphic provides the identification information of the visible anatomical features in the scan image, the evaluation quality of the internal features may be reduced.

[0022] Accordingly, to improve the quality of ultrasound imaging and reduce the time taken to perform an ultrasound scan, it is possible to perform an ultrasound scan in a context-aware mode, in which a context-aware graphic of the ultrasound scan image on a display can be generated and presented to a user. In the context-aware graphic, the relative sizes and relative positions of visible internal features (i.e., internal features visible in the scan image) are annotated. Specifically, each internal feature is represented by a node (e.g., a circular node), where the node size (e.g., area or diameter in the case of a circular node) depicts the relative size of the internal feature, and the position of the node on the graphic depicts the relative position of the internal feature on the scan image. Thus, the context-aware graphic can include one or more nodes, where each node indicates an internal feature in the scan image, and the overall shape of the context-aware graphic depicts the relative positions of the internal features. Additionally, the context-aware graphic can include line segments depicting the edges of the graphic, where each line segment connects at least two nodes. These line segments can indicate one or more of the relative distances and relative angular positions between the nodes they connect, and thus indicate one or more of the relative distances and angular positions between the internal features represented by the connected nodes. For example, the length of each line segment can be based on the relative distance between the two nodes, and thus indicate the relative distance between the corresponding internal features represented by the two nodes.

[0023] In this way, the context-aware graphic can provide an operator with information about the relative sizes and relative positions of one or more internal features in the scan image. Additionally, the context-aware graphic can be generated in real time as the imaging system acquires and processes the scan image, which enables the operator to more clearly identify and visualize changes in the relative sizes and relative positions of the internal features during the scan.

[0024] Furthermore, during an ultrasound scan, when the probe is moved from a current scan plane to a target scan plane, the shape of the context-aware graphic changes based on the relative sizes, relative positions, and visibility of the internal features in the scan direction (e.g., the graphic shape can change from a triangle to a line), which can provide an indication to the user as to whether the probe is moving towards the target scan plane. In this way, the context-aware graphic provides visual feedback to the operator. The visual feedback provided by the context-aware graphic can be used independently of other feedback systems (such as haptic feedback). Accordingly, the context-aware graphic can reduce the need for additional sensors on the probe, which in turn can reduce the bulkiness of the system and lower the manufacturing, system maintenance, and diagnostic costs.

[0025] Moreover, since the context-aware graphic enables the operator to more clearly interpret changes in the relative sizes and positions of internal features in a real-time scan image during the scan, the efficiency and accuracy of the scan are improved.

[0026] Further, the context-aware graphic can be used to guide and direct novice users in interpreting the scanned image, determine whether the user is moving towards the target scan plane, and correctly identify the target scan plane with increased efficiency and precision without using sensors coupled to the probe or the patient.

[0027] Figure 1 An exemplary ultrasound system is shown, which includes an ultrasound probe, a display device, and an imaging processing system. Via the ultrasound probe, an ultrasound image can be acquired and displayed on the display device. The displayed image can be annotated with annotations generated using a segmentation model that can identify anatomical features in the ultrasound image. The segmentation model can be stored on and executed by the image processing system, as Figure 2 shown. In some examples, the segmentation model can be a convolutional neural network (CNN), such as Figure 3 the CNN shown. A context-aware graphic of the scanned image can be generated based on one or more attributes separated from the output of the segmentation model. An Figure 4 exemplary method for performing a medical scan (such as an ultrasound scan) in a context-aware mode is shown at Figure 5 and an exemplary method for generating a context-aware graphic is shown in Figure 8A . The context-aware graphic can be a graphical representation of the relative sizes and relative positions of one or more internal features visible in the scanned image. An Figure 8B exemplary initial scan image and target scan image are shown at Figure 8C . Exemplary segmentation graphics of the initial scan image and the target scan image, including exemplary annotations of one or more internal features superimposed on the respective initial scan image and target scan image, are shown at Figure 9 . An exemplary context-aware graphic generated based on the relative position and relative size attributes of the internal features in the initial scan image and the target scan image is shown as an overlay on the corresponding displayed scan image at Figure 6 . Further, Figure 7As shown. Additionally, based on the context-aware graphics of the current scan image and the target scan image, the user can be guided in real time in the desired direction towards the target scan plane to reach the target scan image, and the user can be alerted when moving away from the target scan plane, as Figure 11 shown. Thus, the position of the current scan plane and the desired probe movement towards the target scan plane can be determined and displayed to the user based on the context-aware graphics of the initial scan image and the target scan image, and these context-aware graphics indicate changes in one or more of the relative dimensions and relative positioning of the internal features. Figure 10 Exemplary graphics are shown at Figure 10 , which illustrate the correlation between the relative dimensions, relative positions of one or more exemplary anatomical features, and the orientation of the scan plane relative to the larger human anatomy.

[0028] It will be appreciated that although the generation of context-aware graphics is shown herein with respect to the Figure 1 ultrasound imaging system shown, context-aware graphics can be generated for any medical scan image such as MRI scan images, CT scan images, SPECT images, X-ray images, etc. For example, a medical scan image can be acquired by a processor of a medical imaging system. After acquiring the medical scan image, one or more internal features can be identified via, for example, a segmentation artificial intelligence model, and corresponding segmentation outputs and / or graphics can be generated. By using the segmentation output, one or more attributes of the medical scan image, including size and position attributes, can be separated via, for example, a data abstraction scheme. The separated attributes can be used to determine the relative dimensions and relative positions of each internal feature identified in the medical scan image. The relative dimension and relative position information can be applied to a context-aware graphics generation model to output a context-generated graphic of the medical scan image, where the context-generated graphic includes one or more nodes representing each internal feature and the overall shape of the context-aware graphic, including nodes representing the relative positions of each internal feature and one or more line segments.

[0029] Figure 1 is a schematic diagram of an ultrasound imaging system 100 according to an embodiment of the present invention. The ultrasound imaging system 100 includes a transmit beamformer 101 and a transmitter 102, which drive elements 104 within a transducer array or probe 106 to transmit pulsed ultrasound signals into the body (not shown). According to one embodiment, the transducer array 106 can be a one-dimensional transducer array probe. However, in some embodiments, the transducer array 106 can be a two-dimensional matrix transducer array probe. Still referring to Figure 1, The pulsed ultrasound signal is backscattered from in-vivo features such as blood cells or muscle tissue to generate an echo that returns to element 104. The echo is converted by element 104 into an electrical signal or ultrasound data, and the electrical signal is received by receiver 108. The electrical signal representing the received echo passes through receive beamformer 110 that outputs the ultrasound data. According to some embodiments, probe 106 may include electronic circuitry to perform all or a portion of transmit beamforming and / or receive beamforming. For example, all or a portion of transmit beamformer 101, transmitter 102, receiver 108, and receive beamformer 110 may be located within probe 106. In the present disclosure, the term “scan” or “scanning” may also be used to refer to the process of acquiring data by transmitting and receiving ultrasound signals. In the present disclosure, the term “data” may be used to refer to one or more data sets acquired with an ultrasound imaging system.

[0030] User interface 115 may be used to control the operation of ultrasound imaging system 100, including for controlling the input of patient data, for changing scan or display parameters, etc. User interface 115 may include one or more of the following: a rotator, a mouse, a keyboard, a trackball, hard keys linked to specific actions, soft keys configurable to control different functions, and a graphical user interface displayed on display device 118.

[0031] The ultrasound imaging system 100 further includes a processor 116 that controls the transmit beamformer 101, the transmitter 102, the receiver 108, and the receive beamformer 110. The processor 116 communicates electronically with the probe 106. For the purposes of this disclosure, the term "electronically communicate" can be defined to include both wired and wireless communication. The processor 116 can control the probe 106 to acquire data. The processor 116 controls which of the elements 104 are active and the shape of the beam transmitted from the probe 106. The processor 116 also communicates electronically with the display device 118, and the processor 116 can process the data into an image for display on the display device 118. According to one embodiment, the processor 116 can include a central processing unit (CPU). According to other embodiments, the processor 116 can include other electronic components capable of performing processing functions, such as a digital signal processor, a field programmable gate array (FPGA), or a graphics board. In some embodiments, the processor 116 can be configured as a graphics processing unit with parallel processing capabilities. According to other embodiments, the processor 116 can include multiple electronic components capable of performing processing functions. For example, the processor 116 can include two or more electronic components selected from a list of electronic components that includes: a central processing unit, a digital signal processor, a field programmable gate array, and a graphics board. According to another embodiment, the processor 116 can further include a complex demodulator (not shown) that demodulates the RF data and generates raw data. In another embodiment, the demodulation can be performed earlier in the processing chain. The processor 116 is adapted to perform one or more processing operations according to a plurality of selectable ultrasound modalities on the data. As the echo signals are received, the data can be processed in real time during a scan session. For the purposes of this disclosure, the term "real time" is defined to include a process that is performed without any intentional delay. For example, embodiments are capable of acquiring images at a real-time rate of 7 to 20 volumes per second. The ultrasound imaging system 100 is capable of acquiring 2D data for one or more planes at a significantly faster rate. However, it should be understood that the real-time volume rate can depend on the length of time taken to acquire each volume of data for display. Therefore, when acquiring relatively large volumes of data, the real-time volume rate may be slower. Thus, some embodiments can have a real-time volume rate significantly faster than 20 volumes per second, while other embodiments can have a real-time volume rate slower than 7 volumes per second. The data can be temporarily stored in a buffer (not shown) during a scan session and processed in a less real-time manner in real-time or offline operations. Some embodiments of the present invention can include multiple processors (not shown) to handle the processing tasks handled by the processor 116 according to the exemplary embodiments described above. For example, a first processor can be used to demodulate and decimate the RF signals, while a second processor can be used to further process the data before displaying the image. It should be understood that other embodiments can use different processor arrangements.

[0032] The ultrasound imaging system 100 can continuously acquire data at a volume rate of, for example, 10 Hz to 30 Hz. Images generated from the data can be refreshed at a similar frame rate. Other embodiments can acquire and display data at different rates. For example, depending on the size of the volume and the intended application, some embodiments can acquire data at a volume rate less than 10 Hz or greater than 30 Hz. A memory 120 is included for storing volumes of the processed acquired data. In one exemplary embodiment, the memory 120 has sufficient capacity to store at least several seconds' worth of ultrasound data volumes. The data volumes are stored in a manner that facilitates retrieval according to their acquisition order or time. The memory 120 can include any known data storage medium.

[0033] Optionally, contrast agents can be utilized to implement embodiments of the present invention. When using ultrasound contrast agents including microbubbles, contrast imaging generates enhanced images of in vivo anatomical features and blood flow. After acquiring data using a contrast agent, image analysis includes separating harmonic and linear components, enhancing the harmonic components, and generating an ultrasound image by utilizing the enhanced harmonic components. Appropriate filters are used to perform the separation of harmonic components from the received signals. Ultrasound imaging using contrast agents is well known to those skilled in the art and will not be described in detail herein.

[0034] In various embodiments of the present invention, the processor 116 can process data through other or different mode-related modules (e.g., B-mode, color Doppler, M-mode, color M-mode, spectral Doppler, elastography, TVI, strain, strain rate, etc.) to form 2D or 3D data. For example, one or more modules can generate B-mode, color Doppler, M-mode, color M-mode, spectral Doppler, elastography, TVI, strain, strain rate, and combinations thereof, etc. Image lines and / or volumes are stored, and timing information indicating the time of data acquisition in the memory can be recorded. These modules can include, for example, a scan conversion module for performing scan conversion operations to convert the image volume from beam space coordinates to display space coordinates. A video processor module can be provided, which reads the image volume from the memory and displays the image in real time during a procedure on a patient. The video processor module can store the image in an image memory, and the display device 118 reads and displays the image from the image memory.

[0035] In various embodiments of the present invention, one or more components of the ultrasound imaging system 100 may be included in a portable handheld ultrasound imaging device. For example, the display 118 and the user interface 115 may be integrated into the outer surface of the handheld ultrasound imaging device, which may further include a processor 116 and a memory 120. The probe 106 may include a handheld probe that communicates electronically with the handheld ultrasound imaging device to collect raw ultrasound data. The transmit beamformer 101, the transmitter 102, the receiver 108, and the receive beamformer 110 may be included in the same or different parts of the ultrasound imaging system 100. For example, the transmit beamformer 101, the transmitter 102, the receiver 108, and the receive beamformer 110 may be included in the handheld ultrasound imaging device, the probe, and combinations thereof.

[0036] After performing a two-dimensional ultrasound scan, a data block containing scan lines and their samples is generated. After applying a backend filter, a process called scan conversion is performed to transform the two-dimensional data block into a displayable bitmap image with additional scan information such as depth, angle of each scan line, etc. During scan conversion, interpolation techniques are applied to fill in the missing holes (i.e., pixels) in the resulting image. These missing pixels occur because each element of the two-dimensional block is typically supposed to cover many pixels in the resulting image. For example, in current ultrasound imaging systems, bicubic interpolation is applied, which utilizes adjacent elements of the two-dimensional block. Thus, if the two-dimensional block is relatively small compared to the size of the bitmap image, the scan-converted image will include regions of poor or low resolution, especially for deeper regions.

[0037] The ultrasound images acquired by the ultrasound imaging system 100 may be further processed. In some embodiments, the ultrasound images generated by the ultrasound imaging system 100 may be transmitted to an image processing system, where in some embodiments, the ultrasound images may be segmented using a machine learning model trained with the ultrasound images and corresponding ground truth outputs. As used herein, a ground truth output refers to the expected or "correct" output based on a given input into the machine learning model. For example, if a machine learning model is being trained to classify images of cats, the ground truth output of the model when fed an image of a cat is the label "cat".

[0038] Although described herein as separate systems, it should be understood that in some embodiments, the ultrasound imaging system 100 includes an image processing system. In other embodiments, the ultrasound imaging system 100 and the image processing system may include separate devices. In some embodiments, the images generated by the ultrasound imaging system 100 may be used as a training data set for training one or more machine learning models, where, as described below, the machine learning models may be used to perform one or more steps of ultrasound image processing.

[0039] See Figure 2 Figure 2 , which shows an image processing system 202 according to an exemplary embodiment. In some embodiments, the image processing system 202 is incorporated into an ultrasound imaging system 100. In some embodiments, at least a portion of the image processing system 202 is disposed at a device (e.g., an edge device, a server, etc.) communicatively coupled to the ultrasound imaging system via a wired connection and / or a wireless connection. In some embodiments, at least a portion of the image processing system 202 is disposed at a separate device (e.g., a workstation) that can receive images / maps from the ultrasound imaging system or from a storage device storing images / data generated by the ultrasound imaging system. The image processing system 202 can be operably / communicatively coupled to a user input device 214 and a display device 216.

[0040] The image processing system 202 includes a processor 204 configured to execute machine-readable instructions stored in a non-transitory memory 206. The processor 204 can be single-core or multi-core, and the program executed thereon can be configured for parallel or distributed processing. In some embodiments, the processor 204 can optionally include separate components distributed across two or more devices, which can be remotely located and / or configured for coordinated processing. In some embodiments, one or more aspects of the processor 204 can be virtualized and executed by a remotely accessible networked computing device configured in a cloud computing configuration.

[0041] The non-transitory memory 206 can store a segmentation module 208, a context-aware module 209, and ultrasound image data 212. The segmentation module 208 can include one or more machine learning models (such as deep learning networks), the one or more machine learning models including a plurality of weights and biases, activation functions, loss functions, gradient descent algorithms, and instructions for implementing the one or more deep neural networks to process input ultrasound images. For example, the segmentation module 208 can store instructions for implementing a neural network (such as Figure 3 the convolutional neural network (CNN) 300 shown). The segmentation module 208 can include trained and / or untrained neural networks and can also include training routines or parameters (e.g., weights and biases) associated with one or more neural network models stored therein.

[0042] The image processing system 202 is communicatively coupled to a training module 210 that includes instructions for training one or more machine learning models stored in a segmentation module 208. The training module 210 can include instructions that, when executed by a processor, cause the processor to perform one or more of the steps of method 600, which is discussed in detail below. In one example, the training module 210 includes instructions for receiving a training data set from ultrasound image data 212, the training data set including a collection of ultrasound images, associated ground truth labels / images, and associated model outputs for training one or more of the machine learning models stored in the segmentation and tracking module 208. The training module 210 can receive ultrasound images, associated ground truth labels / images, and associated model outputs for training one or more machine learning models from sources other than the ultrasound image data 212, such as other image processing systems, the cloud, etc. In some embodiments, one or more aspects of the training module 210 can include a remotely accessible network storage device configured in a cloud computing configuration. The non-transitory memory 206 can also store the ultrasound image data 212, such as Figure 1 ultrasound images captured by an

[0043] ultrasound imaging system. For example, the ultrasound image data 212 can store ultrasound images, ground truth outputs, iterations of machine learning model outputs, and other types of ultrasound image data. In some embodiments, the ultrasound image data 212 can store the ultrasound images and ground truth outputs in an ordered format such that each ultrasound image is associated with one or more corresponding ground truth outputs.

[0044] In some embodiments, the non-transitory memory 206 can include components disposed on two or more devices, which can be remotely located and / or configured for coordinated processing. In some embodiments, one or more aspects of the non-transitory memory 206 can include remotely accessible networked storage devices configured in a cloud computing configuration.

[0045] The user input device 216 can include one or more of a touch screen, a keyboard, a mouse, a touchpad, a motion sensing camera, or other devices configured to enable a user to interact with and manipulate data within the image processing system 31. In one example, the user input device 216 can enable a user to select an ultrasound image for training a machine learning model or for further processing using the trained machine learning model.

[0046] The display device 214 can include one or more display devices utilizing almost any type of technology. In some embodiments, the display device 214 can include a computer monitor and can display ultrasound images. The display device 214 can be incorporated in a common housing with the processor 204, the non-transitory memory 206, and / or the user input device 216, or can be a peripheral display device, and can include a monitor, a touch screen, a projector, or other display devices known in the art, which can enable a user to view ultrasound images generated by the ultrasound imaging system, and to view segmentation annotations and context-aware graphical annotations of the ultrasound scan images, and / or to interact with various data stored in the non-transitory memory 206.

[0047] It should be understood that Figure 2 the illustrated image processing system 202 is for illustration and not limitation. Another suitable image processing system can include more, fewer, or different components.

[0048] Turning to Figure 3 , the architecture of an exemplary convolutional neural network (CNN) 300 is shown. The CNN 300 represents an example of a machine learning model that can be used to determine anatomical segmentation parameters, which can then be used to generate context-aware graphics of ultrasound scan images. The CNN 300 includes a U-net architecture, which can be divided into an autoencoder part (the down part, elements 302b - 330) and an auto-decoder part (the up part, elements 332 - 356a). The CNN 300 is configured to receive an ultrasound image including a plurality of pixels / voxels and map the input ultrasound scan image to generate an anatomical segmentation of the scan image. The CNN 300 includes a series of mappings that start from the input image patch 302b that can be received by the input layer, pass through a plurality of feature maps, and finally map to the output layer 356a.

[0049] Various components including the CNN 300 are labeled in legend 358. As indicated in legend 358, the CNN 300 includes a plurality of feature maps (and / or replicated feature maps), where each feature map may receive input from an external file or a previous feature map and may transform / map the received input into an output to generate the next feature map. Each feature map may include a plurality of neurons, where in some embodiments, each neuron may receive input from a subset of neurons in the previous layer / feature map and may compute a single output based on the received input, where the output may be propagated to a subset of neurons in the next layer / feature map. Feature maps may be described using spatial dimensions such as length, width, and depth, where the dimensions refer to the number of neurons included in the feature map (e.g., how many neurons long, wide, and deep a given feature map is).

[0050] In some embodiments, neurons in a feature map may compute an output by performing a dot product of the received input using a learned set of weights (each learned set of weights may be referred to herein as a filter), where each received input has a unique corresponding learned weight, where the learned weight is learned during training of the CNN.

[0051] The transformation / mapping performed by each feature map is indicated by an arrow, where each type of arrow corresponds to a different transformation, as indicated in legend 358. A solid black arrow pointing to the right indicates a 3×3 convolution with a stride of 1, where the output of a 3×3 grid from the feature channels of the immediately previous feature map is mapped to a single feature channel of the current feature map. Each 3×3 convolution may be followed by an activation function, where in one embodiment, the activation function includes a rectified linear unit (ReLU).

[0052] A hollow arrow pointing down indicates 2×2 max pooling, where the maximum value from a 2×2 grid of a feature channel is propagated from the immediately previous feature map to a single feature channel of the current feature map, resulting in a 4-fold reduction in the spatial resolution of the immediately previous feature map.

[0053] A hollow arrow pointing up indicates 2×2 transposed convolution, which includes mapping the output of a single feature channel from the immediately previous feature map to a 2×2 grid of feature channels in the current feature map, thereby increasing the spatial resolution of the immediately previous feature map by 4-fold.

[0054] The right-pointing dashed arrow indicates copying and cropping the feature map to cascade with another feature map that appears later. Cropping enables the dimensions of the copied feature map to match the dimensions of the feature channels to which the copied feature map is to be cascaded. It should be understood that when the size of the first feature map being copied is equal to the size of the second feature map to be cascaded with the first feature map, cropping may not be performed.

[0055] The right-pointing arrow with a hollow elongated triangular head indicates a 1×1 convolution, where each feature channel in the immediately previous feature map is mapped to a single feature channel in the current feature map, or in other words, where a one-to-one mapping of feature channels occurs between the immediately previous feature map and the current feature map.

[0056] The right-pointing arrow with a bow-shaped hollow head indicates a batch normalization operation, where the distribution of activations of the input feature map is normalized. The right-pointing arrow with a short hollow triangular head indicates a dropout operation, where random or pseudo-random dropout of input neurons (and their inputs and outputs) occurs during training.

[0057] In addition to the operations indicated by the arrows within the legend 358, the CNN 300 also includes feature maps represented by solid-filled rectangles in Figure 3 where the feature map includes a height (the length from top to bottom as shown, which corresponds to the y spatial dimension in the x-y plane), a width ( Figure 3 not shown in, assumed to have the same magnitude as the height, and corresponding to the x spatial dimension in the x-y plane), and a depth (the length from left to right as shown in Figure 3 which corresponds to the number of features within each feature channel). Similarly, the CNN 300 includes copied and cropped feature maps represented by hollow (unfilled) rectangles in Figure 3 where the copied feature map includes a height (the length from top to bottom as shown, which corresponds to the y spatial dimension in the x-y plane), a width ( Figure 3 not shown in, assumed to have the same magnitude as the height, corresponding to the x spatial dimension in the x-y plane), and a depth (the length from left to right as shown in Figure 3 which corresponds to the number of features within each feature channel). Figure 3 not shown in, assumed to have the same magnitude as the height, corresponding to the x spatial dimension in the x-y plane), and a depth (the length from left to right as shown in Figure 3 which corresponds to the number of features within each feature channel).

[0058] Starting from the input image tile 302b (also referred to herein as the input layer), data corresponding to the ultrasound image can be input and mapped to the first feature set. In some embodiments, the input data is preprocessed (e.g., normalized) before being processed by the neural network. The weights / parameters of each layer of the CNN 300 can be learned during the training process, where a matching pair of input and expected output (ground truth output) is fed into the CNN 300. The parameters can be adjusted based on a gradient descent algorithm or other algorithms until the output of the CNN 300 matches the expected output (ground truth output) within a threshold accuracy.

[0059] As indicated by the solid black right - pointing arrow immediately to the right of the input image tile 302b, a 3×3 convolution of the feature channels of the input image tile 302b is performed to produce the feature map 304. As discussed above, the 3×3 convolution includes mapping the input from a 3×3 grid of feature channels to a single feature channel of the current feature map using learned weights, where the learned weights are referred to as convolution filters. Each 3×3 convolution in the CNN architecture 300 can include a subsequent activation function, which in one embodiment includes passing the output of each 3×3 convolution through ReLU. In some embodiments, activation functions other than ReLU can be employed, such as Softplus (also known as SmoothReLU), leaky ReLU, noisy ReLU, exponential linear unit (ELU), Tanh, Gaussian, Sinc, Bent identity, logistic function, and other activation functions known in the field of machine learning.

[0060] The output layer 356a can include an output layer of neurons, where each neuron can correspond to a pixel of the segmented ultrasound image, and where the output of each neuron can correspond to a predicted anatomical feature (or lack thereof) at a given location in the input ultrasound scan image. For example, the output of a neuron can indicate whether the corresponding pixel of the segmented ultrasound image is part of a blood vessel, nerve, bone, artery, etc., or part of an unrecognized feature. Thus, the output layer 356a can include segmentation information of the input ultrasound scan image, where the segmentation information includes the identification, relative position, and relative size of one or more internal anatomical features in the ultrasound scan image.

[0061] The output layer 356a can be input into a context - aware module, such as Figure 2The context awareness module 209 at [location]. The context awareness module may include instructions to receive the output layer 356a from the segmentation module and abstract the segmentation information, which includes anatomical feature recognition information in the input ultrasound scan image, location information of each recognized anatomical feature, and size information of each recognized anatomical feature. The location information includes relative location information of each recognized anatomical feature relative to each other anatomical feature recognized in the scan image and relative to the larger human anatomical structure. The size information includes relative size information of each recognized anatomical feature relative to each other anatomical feature recognized in the scan image (based on visibility in the scan image). The size information may also include relative size information relative to the larger human anatomical structure.

[0062] The segmentation of the relative size and relative location information of the anatomical features in the ultrasound scan image may consider the lateral, axial, and temporal resolutions of the ultrasound scan image. Then, the segmentation information abstracted from the output layer (including anatomical feature recognition data, relative location data, and relative size data) may be used to generate a context awareness graph that depicts a context awareness relationship showing the relative size and relative location of the anatomical features visible in the scan image.

[0063] In one exemplary embodiment, a context awareness graph may be generated in real time for each scan image, and the context awareness graph may be displayed as an overlay with the corresponding scan image. Additionally, in another exemplary embodiment, a desired context awareness graph may be generated and displayed to the user, which shows the desired change in the current context awareness graph when the user moves from the current scan plane to the target scan plane. In this way, a context awareness graph may be generated in real time for the ultrasound scan image, and in addition to the recognition information of each anatomical feature visible in the ultrasound scan image, the context awareness graph may also depict relative size and relative location information. The real-time relative size and relative location information may be used to guide the user from the current scan plane to the target scan plane during an ultrasound examination. Thus, the dependence on sensor-based feedback systems (such as haptic feedback) is reduced, and the user may be visually guided to the target scan plane. Furthermore, the context awareness graph allows the user to determine whether the scan is progressing in the desired direction, thus reducing the time required to reach the target scan plane and improving the overall efficiency of the ultrasound evaluation process.

[0064] It should be understood that the present disclosure includes neural network architectures that include one or more regularization layers, which include batch normalization layers, dropout layers, Gaussian noise layers, and other regularization layers known in the field of machine learning that can be used during training to mitigate overfitting and improve training efficiency while reducing training time. The regularization layers are used during CNN training and are deactivated or removed during the post-training implementation of the CNN. These layers can be scattered among Figure 3 the layers / feature maps shown, or can replace one or more of the layers / feature maps shown.

[0065] It should be understood that Figure 3 the architecture and configuration of the CNN 300 shown are for illustration and not limitation. Any suitable neural network such as ResNet, recurrent neural network, generalized regression neural network (GRNN), etc. can be used. One or more specific embodiments of the present disclosure have been described above to provide a thorough understanding. Those skilled in the art will understand that the specific details described in the embodiments can be modified during implementation without departing from the essence of the present disclosure.

[0066] Turning to Figure 4 , a schematic flowchart is shown that illustrates an exemplary method 400 for performing an ultrasound scan in a context-aware mode. Referring to Figure 1 , Figure 2 and Figure 3 the systems and components of, it should be understood that method 400 can be implemented with other systems and components without departing from the scope of the present disclosure. For example, method 400 can be stored as executable instructions in a non-transitory memory (such as memory 120) and can be executed by a processor (such as processor 116) of the ultrasound imaging system 100, or can be implemented by one or more of the systems disclosed above (such as the image processing system 202). Method 400 can be initiated by a medical professional (such as a doctor) or an operator (such as a sonographer).

[0067] Method 400 begins at 402. At 402, method 400 includes determining the current ultrasound imaging mode. The current ultrasound imaging mode can be determined based on user input received at the user interface of the ultrasound system (such as Figure 1 the user interface 115). The operator of the ultrasound system can select the imaging mode and / or protocol via the user interface, or otherwise enter an input indicating the desired ultrasound imaging mode and / or desired imaging protocol. Exemplary ultrasound imaging modes can include B-mode imaging, Doppler imaging, M-mode imaging, etc.

[0068] In addition, the ultrasound imaging mode indicated by the operator may include a context-aware mode, during which the scanned image is processed to generate and display a context-aware graphic to the user, as further detailed below. In some examples, the selection of the imaging mode may be optional, and the scanned image may be displayed together with the context-aware graphic even without such user selection. Other imaging modes (such as a split-display mode, where split graphics identify and depict one or more anatomical features visible in the scanned image) are also within the scope of the present disclosure.

[0069] In addition, exemplary ultrasound imaging scenarios may include cardiac imaging scenarios (e.g., echocardiography), abdominal imaging, fetal imaging, renal imaging, and / or other anatomy-specific scenarios. Additionally, some exemplary ultrasound imaging scenarios may be based on the type of procedure performed along with or during imaging, such as a resuscitation scenario, a biopsy scenario, etc. The imaging mode and / or scenario may specify which type of ultrasound probe to use, how to control the ultrasound probe during the imaging session (e.g., signal frequency, gain, beam focus, etc.), how to process the acquired image information, and / or what type of images the operator is to acquire during the imaging session (which may include how the operator positions and controls the ultrasound probe during the imaging session). In some examples, additionally, the user may indicate the desired type of examination. For example, a list of possible examination types may be displayed, which includes internal organs, muscle tissue, vasculature, tendons, etc., and the operator may click on the desired examination type, thereby instructing the processor to load settings suitable for that examination type.

[0070] Continuing to 404, method 400 includes confirming whether the context-aware mode has been selected. If so, method 400 proceeds to 410 to operate the ultrasound scan in the context-aware mode. If the context-aware mode is not selected, method 400 may include prompting the user to select the context-aware mode at 406 and also confirming the selection of the context-aware mode at 408. If the context-aware mode is selected after the prompt at 408, method 400 proceeds to 410. Otherwise, method 400 may end.

[0071] At 410, method 400 includes acquiring a target scan image. In one example, the target scan image can be acquired based on user selection and / or input. For example, the user can enter one or more of a desired body plane (e.g., transverse, longitudinal, or coronal), a desired internal body region (e.g., upper or lower abdomen, right or left subcostal, etc.), one or more desired anatomical features to be imaged, and a medical condition to be evaluated (e.g., gallstones, etc.). Additionally, as discussed at 402, user input related to the imaging protocol can be used to select the target scan image. Based on the user input, a plurality of predetermined target scan images can be displayed to the user for selection. Thus, the target scan image can be acquired based on the user selection from the plurality of predetermined scan images stored in the non-transitory memory. In another example, the target scan image can be acquired based on a selected region of interest within a predetermined desired scan image.

[0072] After acquiring the target scan plane, method 400 can include generating, at 412, a context-aware graphic of the target scan plane (also referred to herein as the target context-aware graphic). Details regarding the generation of the context-aware graphic will be Figure 5 described. Briefly, the target context-aware graphic can be generated based on a segmentation graphic of the target scan image (also referred to herein as the target segmentation graphic). The segmentation graphic can identify one or more features visible in the scan image of the target scan plane. After generating the context-aware graphic of the target scan image based on the segmentation graphic, the processor can store the context-aware graphic of the target scan image, the segmentation graphic of the target scan image, and the target scan image in the non-transitory memory.

[0073] In addition, after generating the context-aware graphic of the target scan image, method 400 can include displaying the context-aware graphic of the target scan plane to the user. In one example, the context-aware graphic can be displayed as an overlay on the target scan image on a user interface (such as the display device 118 of the ultrasound system 110). In another example, the context-aware graphic can be displayed adjacent to the target scan image.

[0074] Continuing to 414, method 400 includes determining whether the user has initiated image acquisition of the subject. The initiation of image acquisition can be determined based on one or more signals received from the ultrasound probe (such as Figure 1 probe 106). If the image acquisition has not yet started, method 400 includes prompting the user to start imaging at 416. If image acquisition is initiated, method 400 includes acquiring an initial scan image at 418 based on the ultrasound scan data received from the ultrasound probe.

[0075] Next, at 420, method 400 includes generating a context-aware graphic of the initial ultrasound scan image (hereinafter referred to as the initial scan image), also referred to herein as the initial context-aware graphic, based on the initial ultrasound scan image. Details of generating the context-aware graphic will be described hereinafter with respect to Figure 5 The details of generating the context-aware graphic will be described. Briefly, the context-aware graphic of the initial scan plane can provide an indication of the relative positions and dimensions of the internal features visible in the initial scan image. The context-aware graphic of the initial scan image can be generated based on the segmentation graphic of the initial scan image. The segmentation graphic can provide an indication of the identification of the internal features visible in the initial scan image of the initial scan plane. An artificial intelligence scheme can be used to generate the segmentation graphic based on a segmentation model, as discussed above with respect to Figure 2 and Figure 3 discussed.

[0076] After generating the initial context-aware graphic, method 400 proceeds to 422. At 422, method 400 includes generating a real-time context-aware graphic in real time for each acquired image during scanning from the initial scan plane to the target scan plane. Details of generating the context-aware graphic in real time will be elaborated with respect to Figure 6 Briefly, as the user moves the probe from the initial scan plane towards the target scan plane, for each scan image generated and displayed on the user interface, a corresponding context-aware graphic can be generated and displayed. When the user navigates the probe to turn to the target scan plane, the user can visualize the changes in the scan image through the context-aware graphic, and determine whether the shape of the context-aware graphic is advancing towards the context-aware graphic of the target scan plane, and adjust the probe position accordingly. In this way, the context-aware graphic can be used to guide the user to the target scan plane based on the scan image. Therefore, the need for sensors and additional electronics for tracking the probe movement and guiding the user is also reduced.

[0077] As a supplement or alternative to generating the context-aware graphic in real time, method 400 includes generating one or more desired transformation graphics at 424, the one or more desired transformation graphics depicting the desired changes of the context-aware graphic of the initial scan image to achieve the target context-aware graphic. This will be described hereinafter with respect to Figure 7Describe the generation of one or more desired transformed graphics. In short, the one or more desired transformed graphics provide graphical annotations of how the initial context-aware graphic is expected to change as it progresses from an initial scan plane to a target scan plane in a desired direction. Thus, a user can use the one or more transformed graphics to determine whether the user is moving the probe toward the target scan plane. Thus, the user can reach the target scan plane via a more efficient path. Thus, the time taken to perform a medical scan is also improved, and the efficiency and quality of the scan are improved because the context-aware graphic also enables the user to identify when the probe is at the target scan plane. Thus, the image quality is also improved.

[0078] Next, method 400 proceeds to 426. At 426, method 400 includes displaying an initial context-aware graphic, a target context-aware graphic, and one or more real-time context-aware graphics as the ultrasound scan progresses from an initial scan plane to a target scan plane. Additionally or alternatively, as indicated at 426, one or more desired transformed graphics can also be displayed, including the desired probe movement and the desired scan path relative to a larger human anatomy toward the target scan plane. As discussed above, the context-aware graphics can be displayed on a user interface (such as the display device 118 of the ultrasound system 110) as an overlay on the target scan image or adjacent to the target scan image. The type of display (overlay or side-by-side / adjacent) can be based on user selection and / or input.

[0079] In this way, when a user performs a medical scan and moves the probe from an initial scan plane to a target scan plane, the ultrasound system can operate in a context-aware mode based on the target context-aware graphic, the initial context-aware graphic, and the real-time context-aware graphic.

[0080] Thus, the context-aware graphics help the user to locate the scan plane within the human anatomy. Additionally, it can also be used to guide the user to a desired scan plane, which can be represented as a desired context-aware graphic.

[0081] Go to Figure 5, which shows a flowchart that illustrates an exemplary method 500 for generating a context-aware graphic of a scan image. As discussed herein, a context-aware graphic may include a representation of the relative dimensions and relative positions of one or more internal features, the one or more internal features including anatomical features visible in a scan image generated by a medical imaging scan, such as an ultrasound scan. The context-aware graphic provides context information for one or more of the internal features visible in the scan image relative to each other and relative to the larger human anatomy. The scan image may be obtained through image processing based on information received from an ultrasound probe at a scan plane. Method 500 may be implemented by one or more of the systems disclosed above, such as image processing system 202 and / or ultrasound system 100, but it should be understood that method 500 may be implemented with other systems and components without departing from the scope of the present disclosure.

[0082] Method 500 begins at 502. At 502, method 500 includes acquiring a scan image. The acquired scan image may include one or more of the following: a current scan image based on signals received from an ultrasound transducer and a target scan image based on a user's selection of a series of predetermined target scan images stored in non-transitory memory. For example, acquiring a scan image may include using an ultrasound transducer to acquire image information of a patient. The ultrasound transducer emits ultrasound signals and receives reflected signals while scanning the patient. The acquired data may be processed into a scan image by a processor. In another example, when operating the ultrasound system in a context-aware mode, an operator may select a target scan image from a series of predetermined target scan images stored in non-transitory memory.

[0083] Optionally, at 504, method 500 includes displaying the scan image without one or more of a segmentation graphic and a context-aware graphic. For example, the acquired ultrasound image may be displayed without an indication of the identified anatomical features, the relative positions and relative dimensions of the identified anatomical features, such as Figure 8A the ultrasound images shown at 805 and 810 and further described below.

[0084] After acquiring the scan image, method 500 includes generating a segmentation mask (also referred to as a segmentation output) of the scan image at 506. Generating a segmentation mask may include inputting the acquired scan image into a segmentation model. In some examples, the segmentation model is a CNN with an autoencoder-auto decoder type architecture, such as Figure 3The CNN 300 shown. It should be understood that other artificial intelligence models for learning internal features in ultrasound scan images may be implemented and are within the scope of the present disclosure. The segmentation model may generate, as a first output, a segmentation mask of the acquired scan image. Thus, the segmentation mask is the output of the segmentation model and may include the identities of one or more internal features present in the acquired scan image. The segmentation graphic may be generated from the segmentation mask and may include estimated contours and / or highlighting regions of one or more internal features in the acquired scan image. Exemplary segmentation graphics inserted on the scan image are shown at Figure 8B 820 and 830 of

[0085] Next, after generating the segmentation mask, at 508, method 500 includes separating one or more attributes of one or more internal features visible in the scan plane, including size, position, and density. In some examples, other additional attributes (such as axial and lateral resolution) that can be used to generate context-aware graphics may be separated. Separating one or more attributes from the segmentation mask may include entering the segmentation output of the segmentation model as a second input into the context-aware module. The context-aware module may implement an abstraction model (such as Figure 2 the abstraction model 216 at

[0086] which may be a directed or undirected, cyclic or acyclic graph to extract one or more attributes from the segmentation mask, including the size, position, density, resolution, and identification of each internal feature captured in the acquired scan image.

[0087] As an illustrative example, the acquired scan image may capture the liver, kidneys, and a portion of the large intestine / intestines. In the above example, the kidneys may be selected as the reference feature, and the relative sizes (normalized area and / or volume) of the liver and the portion of the large intestine / intestines may be determined.

[0088] It can be noted that the reference feature may be present in the acquired scan image, and thus the selection of the reference feature may change based on the acquired scan image, which depends on the portion of the anatomical structure being examined.

[0089] In addition, in some examples, when the target scan image is selected and the current scan image is being processed to generate the current context-aware graphic, the selected reference feature may be present in both the current scan image and the target scan image. In other words, the reference feature may be imaged at both the initial scan plane and the target scan plane. In yet another example, if the current scan image does not include any internal features that are also present in the target scan image, any internal feature in the current image may be used as the reference feature.

[0090] Next, at 512, method 500 includes determining the relative position of each internal feature in the acquired scan image relative to the reference feature. As discussed above, the reference feature may be an internal feature visible in the acquired scan image. In some embodiments, determining the relative position of the internal feature may include determining the x and y coordinates of a selected point of each internal feature, where the x coordinate is relative to an axial axis perpendicular to the propagation direction of the ultrasonic wave, and the y coordinate is relative to a transverse axis along the propagation direction of the ultrasonic wave. For example, the selected point may be the centroid or the innermost point of the internal feature. The relative position may be a fine-grained descriptor (connected graph of the centroid) or a coarse-grained descriptor (up / down, right / left).

[0091] In addition, the relative position may include the relative distance between the one or more internal structures. For example, the distance between two internal features may be indicated by the length of a line segment connecting two nodes corresponding to the two internal features.

[0092] In another embodiment, as indicated at 514, determining the relative positions of one or more anatomical features may include identifying a reference axis in a scanned image and determining the angular position of each anatomical feature relative to the reference axis. For example, in a scanned image that includes a first internal feature, a second internal feature, and a third internal feature, the reference axis may pass, for example, in a transverse direction through the first centroid of the first internal feature in the scanned image. The angle at which the second centroid of the second internal feature in the scanned image is positioned relative to the reference axis may be determined. Similarly, if the scanned image depicts a third internal feature, a second angle at which the third centroid of the third internal feature is positioned relative to the reference axis may be determined. The relative position information (including relative angular position) may be used to track the progress of an ultrasound evaluation procedure. For example, the relative position information may be used to determine whether an ultrasound scan is progressing from a current scanned image toward a target scanned image, as relative to Figure 11 Further discussion.

[0093] Continuing to 516, method 500 includes generating a context-aware representation (alternatively referred to herein as a context-aware graph) of the acquired scanned image, including relative size and relative position indications. Generating a context-aware representation of the acquired scanned image may include inputting the abstracted and normalized attributes (including relative size and relative position attributes (from 510 and 512, respectively)) as inputs into a context-aware graph generation model, such as Figure 2 the context-aware graph generation model 217 at

[0094] Generating the context-aware representation includes determining, at 518, the node size of each internal feature based on the relative size of the respective internal feature. That is, the relative size of the respective internal feature in the scanned image relative to other internal features. For example, for a scanned image that includes an internal feature and one or more other internal features, the internal feature may be represented by a node in the context-aware graph. The node may be any geometric shape (e.g., circular, oval, square, or any other geometric shape) and may provide a graphical representation of the internal feature visible in the scanned image, where the node size is based on the relative size of the internal feature relative to the one or more other internal features visible in the scanned image (e.g., a reference feature, as discussed at 510). Additionally, the node size may be adjusted based on a change in the relative size of the internal structure. Specifically, during the scan, if the relative size of an internal structure in the scanned image increases relative to the one or more other internal features, the node size in the context-aware representation increases, and if the relative size of the internal structure in the scanned image decreases, the node size in the context-aware representation decreases. Thus, there is a direct correlation between the relative size of the internal feature and the relative size of the internal feature represented by the node.

[0095] In addition, when the node size of the internal structure is reduced below a threshold node size, the corresponding internal feature may not be indicated in the context-aware representation.

[0096] In some examples, when a portion of an internal feature is visible in the scanned image, the node size may be based on the relative size of that portion of the internal feature.

[0097] As a non-limiting example, if the scanned image includes a portion of a first internal feature and a second internal feature, where the relative size of the portion of the first internal feature is greater than the second internal feature, the portion of the first internal feature may be represented by a first circular node and the second internal feature may be represented by a second circular node, where a first diameter of the first circular node is greater than a second diameter of the second circular node.

[0098] Generating the context-aware representation further includes determining, at 520, the centroid of each node. Determining the centroid of each node may include locating the corresponding node of each internal feature on the scanned image such that the position of the node corresponds to the position of the corresponding internal feature, and calculating the coordinates of the centroid on the scanned image based on the shape of the node.

[0099] After determining the node size and centroid, method 500 proceeds to 522. At 522, method 500 includes displaying each node and the connections between the nodes. In one implementation, the context-aware graphic may be displayed as an overlay on the acquired scanned image, where each node is located above its corresponding internal feature and the centroid of each node is connected in a certain manner via one or more line segments such that the nodes and line segments indicate the shape of the acquired scanned image specific to the acquired scan plane. In another implementation, the context-aware graphic may be displayed without the scanned image below.

[0100] As a non - limiting example, for the captured scan image that includes three internal features and three nodes as discussed above, the first circular node can be positioned as an overlay above the first internal feature on the captured scan image, the second circular node can be positioned as an overlay above the second internal feature, and the third circular node can be positioned as an overlay above the third internal feature. The first centroid of the first node can be connected to the second centroid of the second node via a first line segment and to the third centroid of the third node via a second line segment. Additionally, the second centroid and the third centroid can be connected via a third line segment. Thus, the first node, the second node, and the third node, along with the first line segment, the second line segment, and the third line segment, together represent the triangular shape of the context - aware graphic. The size of the nodes represents the relative size of the internal features in the captured scan image, and the line segments represent the relative positions of the internal features in the captured scan image (including the amount of spacing between the internal features and the angular positional relationship between the internal features, where the amount of spacing between the internal features is based on the length of the line segments). Thus, the context - aware graphic includes one or more nodes that form one or more vertices of the context - aware graphic and one or more line segments that form the edges of the context - aware graphic, with each line segment connecting at least two nodes; and where the number of nodes is based on the number of the one or more anatomical features identified in the scan image. In this way, the context - aware graphic including one or more nodes and line segments can be presented as an overlay graphic on the captured scan image.

[0101] Additionally, in some embodiments, a threshold node size can be determined, and only those nodes with a node size greater than the threshold node size can be displayed.

[0102] In this way, a context - aware graphic can be generated for the scan image, and the context - aware graphic can provide the operator with relative size and position information of one or more internal features captured in the scan image. Figure 8C An overlay of an exemplary scan image and the context - aware graphic is shown at [location].

[0103] Additionally, in one example, as discussed below with respect to Figure 6 the context - aware graphic can be generated in real - time and displayed on the user interface of the ultrasound imaging system together with the scan image.

[0104] Moving on to Figure 6, an exemplary method 600 for real-time monitoring of a current scanned image based on real-time generation of context-aware graphics is shown. In short, real-time monitoring of a current scanned image may include generating a current context-aware graphic for each scanned image during scanning, and may also include determining whether a target scanned image has been achieved based on a comparison of the current context-aware graphic of the current scanned image with the target context-aware graphic of the target scanned image. Method 600 may be implemented by one or more of the systems disclosed above, such as image processing system 202 and / or ultrasound system 100, but it should be understood that method 600 may be implemented with other systems and components without departing from the scope of the present disclosure.

[0105] Method 600 begins at 602. At 602, method 600 includes acquiring a target scanned image and generating a target context-aware graphic of the acquired target scanned image. Step 602 may be similar to steps 410 and 412 at Figure 4 .

[0106] As discussed above, the target scanned image may be selected by an operator from a set of predetermined target scanned images stored in the non-transitory memory of the processor. The target scanned image may be based on one or more of the type of ultrasound evaluation to be performed, the target scan plane, and the target anatomical feature to be examined. For example, when operating the ultrasound device in a context-aware mode, the user may select the target scanned image based on one or more of the factors indicated above.

[0107] After acquiring the target scanned image, a target context-aware graphic may be generated based on the target scanned image. The target context-aware graphic may indicate the relative sizes and relative positions of one or more internal features in the target scanned image, where each internal feature is depicted by a node (e.g., a circular node), the size of the node is based on the relative size of the internal feature, and the positional relationship between the internal features is depicted by one or more line segments connecting the centroids of each node.

[0108] As a non - limiting example, the target scan image may include the right lobe of the liver and the right kidney in the sagittal plane, where the right kidney is positioned at an obtuse angle relative to the liver. The target context - aware graphic may show a first node depicting the kidney and a second node depicting the liver, where a first dimension (e.g., area) of the first node is less than a second dimension (e.g., area) of the second node because the relative size of the kidney is less than that of the liver. Further, a first centroid of the first node is connected to a second centroid of the second node via a line segment. The first node is positioned at an obtuse angle relative to a horizontal reference axis that passes through the second centroid of the second node representing the liver. No additional anatomical or internal features are indicated in the exemplary target context - aware graphic. In one example, the target context - aware graphic may be displayed as an overlay above the target scan image. In other examples, the target context - aware graphic may be displayed adjacent to the target scan image. The display type (overlay or adjacent) of the target - aware graphic may be based on operator selection. Figure 8C An exemplary target context - aware graphic is shown as an overlay with the target scan image.

[0109] Next, at 604, method 600 includes the operator acquiring a current scan image based on real - time imaging, and at 606, method 600 includes generating a current context - aware graphic of the current scan image. Steps 604 and 606 may be similar to Figure 4 steps 418 and 420 at. Briefly, the current context - aware graphic may include the relative positions and dimensions of one or more internal features captured in the current scan image. Continuing with the above liver and kidney example, the operator may initiate an ultrasound scan in a scan plane different from the target scan plane. Thus, the current scan image may include three internal features such as the liver, the right kidney, and the intestine, which are quite different from the desired liver and right - kidney features (the target scan image). Accordingly, the current context - aware graphic includes a first node depicting the kidney, a second node depicting the liver, and a third node (also a circular node) depicting the intestine. Further, the first node is at an acute angle relative to a horizontal reference axis along the centroid of the second node, the third node is at an obtuse angle relative to the horizontal reference axis, and the three nodes are connected at their respective centroids, so the current context - aware graphic includes a triangular shape and has internal features at each vertex of the triangle.

[0110] Continuing to 608, method 600 includes displaying the current context - aware graphic. As discussed above, the current context - aware graphic may be displayed independently on the user interface at a location that does not overlap with the current scan image (e.g., above, below, or adjacent), or as an overlay on the current scan image. Figure 8C An exemplary current context - aware graphic is shown as an overlay with the current scan image.

[0111] Next, method 600 includes determining at 610 whether a target scanned image is achieved. This includes comparing the target context-aware graph (generated at 602) with the current context-aware graph (generated at 606). The comparison includes comparing the number of nodes, the relative size of each node, and the relative position of each node between the target context-aware graph and the current context-aware graph. The processor may determine that the target scanned image is achieved when the current context-aware graph matches the target context-aware graph. Specifically, the processor may determine that the target scanned image is achieved when all of the following are confirmed: 1. The target number of nodes in the target context-aware graph is equal to the current number of nodes in the current context-aware graph, 2. The size of each node in the target context-aware graph is equal to (or within an acceptable deviation) the size of the corresponding node in the current context-aware graph, and 3. The relative position of the nodes in the target context-aware graph (e.g., the position of the centroid relative to a reference axis) matches the relative position of the nodes in the current context-aware graph. Thus, if the overall shape of the target context-aware graph matches the overall shape of the current context-aware graph based on the number of nodes, the relative size, and the position of the nodes, the processor may determine that the target scanned image is achieved and the target scan plane is achieved.

[0112] The determination of whether the target scanned image is achieved is further illustrated below using the liver and kidney examples discussed above. For example, if the current context-aware graph shows three nodes and the target context-aware graph shows only two nodes, it may be determined that the target scanned image is not achieved. However, if the current graph shows two nodes, the processor may continue to determine whether the relative size and position of the nodes are within the corresponding threshold limits. For example, it may be determined whether the angle between the first centroid (representing the kidney) and the reference horizontal axis is equal or within a threshold angle in the target context-aware graph and the current context-aware graph, and whether the first node area of the first node and the second node area of the second node are equal or within a threshold area limit in the target context-aware graph and the current context-aware graph. If the answer is yes, the overall shape and size of the target context-aware graph match the current context-aware graph, and the processor may determine that the target scanned image is achieved.

[0113] Thus, at 610, if it is confirmed that the target scanned image is achieved, method 600 proceeds to 612. At 612, method 600 includes indicating to the user via the user interface that the target scanned image is achieved. The method may then return to step 426 to display the current context-aware graph and the target context-aware graph as an overlay with the current scanned image and the target scanned image, or separately from the current scanned image and the target scanned image.

[0114] Returning to 610, if it is determined that the current context-aware graphic does not match the target context-aware graphic, method 600 proceeds to 614. At 614, method 600 includes determining whether the current scan plane has changed based on one or more of probe movement and operator indication. If so, method 600 returns to 604 and continues to acquire the current scan image and update the current context-aware graphic based on the current scan image until the target scan image is achieved. At 614, if no change in the current scan plane is detected, method 600 includes displaying the current context-aware graphic of the current scan image until a change in the scan plane is detected. Additionally, method 600 may include indicating to the user at 616 via the user interface that the target scan plane has not been achieved.

[0115] Thus, the comparison of the current context-aware graphic with the target context-aware graphic can provide an indication to the user as to whether the target scan image has been achieved. Although the segmentation graphic may include the identification of one or more internal features, the user may still have to make multiple attempts to determine whether the scan is progressing toward the target scan image or whether the user is scanning at the target scan plane. By using the context-aware graphic, the user can more effectively evaluate changes in the scan image with increased precision and determine whether the target scan image has been achieved.

[0116] Next, Figure 7 An exemplary method 700 for generating one or more desired transformation graphics that indicate desired changes to a context-aware graphic of an initial scan image to achieve a desired scan image is shown. Method 700 may be implemented by one or more of the systems disclosed above such as image processing system 202 and / or ultrasound system 100, but it should be understood that method 700 may be implemented with other systems and components without departing from the scope of the present disclosure.

[0117] Method 700 begins at 702. At 702, method 700 includes identifying, estimating, and / or measuring one or more target scan image parameters based on the target context-aware graphic. Determining the target scan image parameters may include acquiring the target scan image and generating the target context-aware graphic, as discussed with respect to 410 and 412 of Figure 4 and 602 of Figure 5 and. Figure 6 The target scan image parameters may be based on one or more internal anatomical features identified in the target scan image and the target context-aware graphic and may include one or more of the number of nodes, the relative node size of each node, and the relative position of each node. In some examples, the identity of the one or more internal features may additionally be included as a target scan image parameter.

[0118] Proceeding to 704, method 700 includes identifying, estimating, and / or measuring one or more initial scan image parameters based on an initial context-aware graph. Determining the initial scan image parameters can include acquiring an initial scan image and generating an initial context-aware graph, as discussed with respect to 418 and 420 of Figure 4 and 604 and 606 of Figure 5 . The initial scan image parameters can be based on one or more internal anatomical features identified in the initial scan image and the initial context-aware graph, and can include one or more of the number of nodes, the relative node size of each node, and the relative position of each node. In some examples, the identity of the one or more internal features can additionally be included as a target scan image parameter. Figure 4 , Figure 5 418 and 420 of and Figure 6 as discussed with respect to 604 and 606 of. The initial scan image parameters can be based on one or more internal anatomical features identified in the initial scan image and the initial context-aware graph, and can include one or more of the number of nodes, the relative node size of each node, and the relative position of each node. In some examples, the identity of the one or more internal features can additionally be included as a target scan image parameter.

[0119] Additionally, in some examples, the initial scan image can be indicated by an operator. For example, after initiating an ultrasound scan and viewing an image of the initial scan, the user can enter the image via a user interface and indicate the image as the initial image to be used in the context-aware mode. In some other examples, after generating a first scan image from an ultrasound scan, the processor can automatically assign the first scan image as the initial scan image. The initial scan image can then be used as an input along with the target scan image to determine one or more transformed scan planes, as discussed further below.

[0120] Next, at 706, method 700 includes comparing the values between each corresponding target scan plane parameter and the initial scan plane parameter. For example, the number of nodes in the initial context-aware graph can be compared with the number of nodes in the target context-aware graph, the relative size of each node in the initial graph can be compared with the relative size of the corresponding node in the target graph, and the relative angular position of each node in the initial context-aware graph with respect to a reference node or reference axis can be compared with the relative angular position of the corresponding node in the target graph. Additionally, the identity of each internal feature in the initial scan image can be compared with the identity of each internal feature in the target scan image. As discussed above, the identity of each internal feature in the scan image can be based on a segmentation model, such as a CNN model having an autoencoder-auto-decoder architecture, as discussed at Figure 3 . One or more additional attributes of the segmentation map of the initial scan image can be abstracted and entered as an input into a context-aware module (such as module 209 at Figure 2 ) to determine a context-aware graph, as discussed with respect to Figure 5 . Additionally, the process described at Figure 7 can be performed by the context-aware module. Figure 3 One or more additional attributes of the segmentation map of the initial scan image can be abstracted and entered as an input into a context-aware module (such as module 209 at Figure 2 ) to determine a context-aware graph, as discussed with respect to Figure 5 . Additionally, the process described at Figure 7 can be performed by the context-aware module. Figure 2 to determine a context-aware graph, as discussed with respect to Figure 5 . Additionally, the process described at Figure 7 can be performed by the context-aware module. Figure 5 as discussed with respect to. Additionally, the process described at Figure 7 can be performed by the context-aware module. Figure 7 The process described at can be performed by the context-aware module.

[0121] After comparing the values between each initial scan plane parameter and the target scan plane parameter, method 700 includes determining, at 708, a desired direction of probe movement based on this comparison. The desired direction of probe movement may include a desired path from an initial position relative to a larger human anatomy to a target position relative to the larger human anatomy. The determination of the desired direction of probe movement may be based on the relationship between the learned scan plane orientation and the change in the appearance of anatomical features in the scan image (which depends on the scan plane orientation and anatomical location). For example, the context awareness module may be configured to learn the relationship between scan plane directivity and the change in the relative size and relative position of one or more anatomical features based on scan plane directivity.

[0122] As a non-limiting example, during scanning from an upper position relative to the human anatomy to a lower position in the sagittal plane, the relative size of the liver decreases relative to the kidneys, and the relative position of the kidneys changes relative to the liver. That is, in the scan image captured at the upper position of the human anatomy, the size of the liver is larger relative to the kidneys, and in the scan image captured at the relatively lower position of the human anatomy, the size of the liver is smaller relative to the kidneys. Although the above example shows the size change relative to the probe movement, it should be understood that the context awareness module may consider the change in relative position and the appearance / disappearance of one or more features when determining the directivity relative to the larger human anatomy. Thus, during ultrasound scanning, based on the change in relative node size (node size based on the relative size of anatomical features) and relative node position (node position based on the relative position of anatomical features) derived from, for example, consecutive scan images, the context awareness module may determine the direction of probe movement relative to the larger human anatomy. In addition, the context awareness module may be able to position the scan plane relative to the larger human anatomy based on the context awareness graph and the scan image.

[0123] In addition, when the initial scan image parameters (determined by the initial context awareness graph) and the target scan image parameters (determined by the target context awareness graph) are input, the context awareness module may determine in which direction to move the probe to achieve the target scan image from the initial scan image.

[0124] As another non - limiting example, if the initial scan image includes two anatomical features, where the size of the first anatomical feature is larger than the size of the second anatomical feature (such that the size of the first node is larger than the size of the second node) and the position of the first anatomical feature is at an acute angle relative to a reference axis, and the target scan image includes two anatomical features, where the size of the first anatomical feature is smaller than the size of the second anatomical feature (such that the size of the first node is larger than the size of the second node) and the position of the first anatomical feature is at an obtuse angle relative to the reference axis, then the context - aware module can determine that in order to achieve the target scan image, the probe movement can follow the upper - to - lower direction based on learning that as the ultrasound scan progresses in the upper - to - lower direction, the size of the first node decreases relative to the size of the second node and the angle between the first node and the reference axis changes from an acute angle to an obtuse angle.

[0125] In this way, based on the relationship between the directivity of the scan plane, the change in relative node size, and the change in relative position learned from the context - aware graph, the current scan plane relative to the larger human anatomical structure can be determined. Figure 10 A non - limiting example is shown, which shows the scan directivity relative to the larger human anatomical structure and its relationships.

[0126] In addition, in addition to the relative size and relative position of the representative nodes, other factors such as the visibility / presence of specific anatomical features in the initial scan plane and the target scan plane can also be included when determining the directivity of the probe movement.

[0127] Continuing to 710, method 700 includes identifying one or more transformed scan planes in the desired direction of the probe movement. The transformed scan planes can be positioned along the desired direction and desired path of the probe movement and can indicate the directivity of the desired probe movement.

[0128] After identifying the one or more transformed scan planes, method 700 includes generating and displaying, at 712, a context - aware graph for each of the one or more transformed scan planes to guide the operator from the initial scan plane to the desired scan plane. Generating and displaying the context - aware graph for the one or more transformed scan planes can include (e.g., from a set of predetermined scan images) acquiring a scan image for each of the one or more transformed scan planes and generating a transformed context - aware graph (also referred to herein as the desired transformed graph or desired transformed context - aware graph) for each of the acquired scan images, as Figure 5 discussed at. The transformed context - aware graph can provide an indication of how the appearance of the context - aware graph can change as the probe moves in the desired direction towards the target scan image. The transformed graph can provide an indication to the operator when the probe is moving in the desired direction towards the target scan image.

[0129] In this way, the context-aware graphic can be used to determine the probe position relative to the larger human anatomy and provide the operator with one or more indications related to the desired path of the probe movement relative to the larger human anatomy. For each desired transformed scan plane among the one or more desired transformed scan planes, a context-generated graphic can be generated and displayed to the user, thereby notifying the user of the desired change in the initial context-aware graphic / current context-aware graphic to achieve the target scan image. Figure 9 A non-limiting exemplary series of context-aware graphics including an initial context-aware graphic, a target context-aware graphic, and one or more desired transformation graphics is shown, where the one or more desired transformation graphics show the desired change in the graphic when the probe moves from the initial scan plane to the desired scan plane.

[0130] Figure 8A A first set of ultrasound images 800 is shown, which includes a first scan image 805 and a second scan image 810. The first scan image 805 can be the initial scan image acquired by the ultrasound probe and can include one or more internal features of the patient. The second scan image 810 can be the target scan image and can include one or more desired internal features to be examined relative to the patient during the ultrasound evaluation. In some examples, the target scan image can be selected from a set of predetermined target scan images based on one or more of the target scan plane, one or more desired internal features to be examined, and the desired condition to be evaluated. In other examples, the target scan image can be automatically selected based on one or more inputs entered by the operator including the target scan plane, one or more desired internal features to be examined, and the desired condition. The first scan image and the second scan image showing the initial scan image and the target scan image have no annotations or indications, which makes it difficult for the operator to identify the internal features visible in the scan images, determine the current scan plane relative to the larger human anatomy, and further determine the probe movement direction and probe position to achieve the target scan image, as shown at 810.

[0131] Figure 8B A second set of ultrasound images 815 is shown, which includes a first scan image 820 and a second scan image 830. The first scan image 820 and the second scan image 830 can be the initial scan image and the target scan image annotated with a segmentation graphic having one or more highlighted regions, where the one or more highlighted regions depict as according to relative to Figure 8A the initial scan image and the target scan image, the one or more highlighted regions depicting as according to relative to Figure 3Internal features determined by the output of the segmentation model. Specifically, the first scan image 820 and the second scan image 830 include highlighted regions 822, 824, and 826 that respectively identify the liver, kidneys, and intestines. Although the second set of images 815 identifies internal features in the scan images, thereby providing anatomical structure awareness, it should be understood that the second set of images 815 does not provide context awareness of the relative sizes and relative positions of the internal features in the scan images, and the operator may have difficulty identifying the current scan plane and the target scan plane relative to the larger human anatomy, and determining the probe movement direction and its position to reach the target scan image as shown at 830 from the initial scan image as shown at 820.

[0132] Figure 8C A third set of ultrasound images 835 is shown, which includes a first scan image 840 and a second scan image 850. The first scan image 840 and the second scan image 850 can be Figure 8B the initial scan image and the target scan image with a context-aware representation determined based on the output of the context-aware model described herein. The context-aware representation (also referred to as a context-aware graph) can include a first node 842 depicting a first internal feature (which is the kidney in this example), a second node 844 depicting a second internal feature (which is the liver in this example), and a third node 846 depicting a third internal feature (which is the intestine in this example). Although the nodes are represented as circular nodes in this example, other geometric shapes including squares, triangles, etc. are also within the scope of the present disclosure. Additionally, for greater clarity and ease of identification, different nodes can be highlighted with different color schemes. The size of each node (which can be the area of the circular node or based on the diameter of the circular node) can indicate the relative size of the internal feature in the scan image. For example, the first node 842 has a smaller diameter (and thus area) relative to the second node 844, and the third node 846 has a larger diameter relative to the first node 842 but a smaller diameter relative to the second node 844. Thus, in the initial scan plane, as depicted by the context-aware graph in 840, the second node 844 is larger than the third node 846, and the third node is larger than the first node 842. At the target scan plane, the relative sizes between the first node 842 and the second node 844 are nearly the same or within a threshold difference, but the third node 846 is not visible.

[0133] The context-aware representation further includes: a first line segment 841 connecting the first centroid of the first node 842 and the second centroid of the second node 844 in the initial scanned image, a second line segment 843 connecting the first centroid and the third centroid of the third node 846, and a third line segment 845 connecting the third centroid and the first centroid. The connections between these centroids show the positional relationships of these internal features. For example, in the initial scanned image, the first node 842 forms an acute angle with the horizontal reference axis passing through the second centroid. In the target scanned image, the first node is 842 which forms an obtuse angle with the horizontal reference axis. Additionally, the overall shape of the context-aware graph of the initial scanned image is triangular, while the overall shape of the context-aware graph of the target scanned image is a line segment due to the absence of the third node.

[0134] By observing the context-aware graph, the operator can move the probe such that the context-aware graph of the initial scanned image begins to change until the current context awareness (generated in real time based on real-time imaging) resembles the context-aware graph of the target scanned image. In some examples, as shown below with respect to Figure 9 and discussed with respect to Figure 7 , one or more transformation graphs can be generated and displayed to the operator to depict the desired changes to the initial scanned image to achieve the target scanned image. For example, the context-aware module can use the initial context-aware graph of the initial scanned image and the target context-aware graph of the target scanned image as inputs to determine one or more transformed context-aware graphs and display the transformation graphs to the operator. Additionally, the context-aware module can determine the direction of movement of the probe relative to the larger human anatomy based on the initial context-aware graph and the target context-aware graph. For example, based on the relative positions of the first node 842 and the second node 844 in the initial context-aware graph and the target context-aware graph (from the acute angle with respect to the reference axis in the initial graph to the obtuse angle in the target graph), the context-aware module can determine the desired direction of movement of the probe relative to the larger human anatomy (e.g., upper to lower, back to abdomen, etc.), and indicate the desired direction of movement of the probe to the operator via the user interface.

[0135] Furthermore, as described with respect to Figure 11 , the context-aware module can continuously monitor the current context-aware graph and compare it with the target context-aware graph, and indicate to the user when the current context-aware graph matches the target context-aware graph or whether the current context-aware graph deviates from the target context-aware graph by one or more threshold limits (with respect to the size, position, quantity, etc. of the nodes).

[0136] Now refer to Figure 11, a flowchart is shown, which shows an exemplary method 1100 for monitoring a current scan image in real time relative to a target scan image based on changes in a target context-aware graph and a current context-aware graph during an ultrasound scan in a context-aware mode. Method 1100 can be implemented by one or more of the systems disclosed above, such as image processing system 202 and / or ultrasound system 100. However, it should be understood that method 1100 can be implemented with other systems and components without departing from the scope of the present disclosure.

[0137] Method 1100 begins at 1102. At 1102, method 1100 includes acquiring a current scan image in real time based on information received from an ultrasound probe during an ultrasound scan. Next, at 1104, method 1100 includes generating a context-aware graph in real time based on the current scan image. For example, the context-aware graph can be generated as discussed relative to Figure 5 what has been discussed.

[0138] Next, after acquiring the current scan image and generating a real-time context-aware graph of the current scan image, method 1100 includes monitoring changes in the current context-aware graph in real time at 1106, which can include comparing current context-aware parameters (determined at time point t), such as node size, relative node position, and visibility of one or more nodes, with corresponding parameters in a previous context-aware graph (determined at time point t-1). Specifically, monitoring changes in the current context-aware graph in real time can include monitoring changes in corresponding node sizes at 1108, monitoring changes in relative node positions at 1110, and monitoring the visibility of one or more nodes at 1112 between the current context-aware graph and the previous context-aware graph. Monitoring changes in corresponding node sizes can include, for each node in the current context-aware graph, calculating the change between the current node size and the corresponding node size in the previous context-aware graph. Monitoring changes in relative node positions can include, for each node in the current context-aware graph, calculating the change in the angle between the node in the current graph and a reference axis and the angle between the corresponding node and the corresponding reference axis in the previous graph. Monitoring the visibility of one or more nodes includes determining whether all nodes in the previous graph are present in the current graph and determining whether there are any additional nodes in the current context-aware graph compared to the previous context-aware graph or whether one or more nodes have disappeared. Additionally, in some examples, the overall shape of the current context-aware graph can be compared with the previous context-aware graph.

[0139] Continuing to 1114, method 1100 includes determining whether each change determined above with respect to step 1106 (i.e., the difference between the current context-aware parameter and the previous context-aware parameter) is progressing towards the target parameter value. For example, if the size of the first target node in the target graph is greater than the size of the corresponding first previous node in the previous graph, if the size of the first current node (corresponding to the first node in the previous graph and the target graph) is greater than the size of the previous node, then the change may be progressing towards the target graph. In other words, if the size of the first node in the current graph is greater than the size of the corresponding first node in the previous graph, if the size of the first node in the target graph is greater than the size of the current graph, it can be determined that the change is progressing towards the target graph. As another example, if the current angle between the first node in the current graph and the reference axis is greater than the corresponding angle in the previous graph, if the target angle of the first node with respect to the reference axis in the target graph is greater than the current angle, it can be determined that the change is progressing towards the target graph. As yet another example, if the target shape of the target graph is a line segment and the previous shape of the previous graph is a triangle, if the current shape tends towards a line segment, it can be determined that the change is progressing towards the target graph.

[0140] If it is determined that the real-time change is progressing towards the target context-aware graph, the method proceeds to 1116. At 1116, method 1110 includes indicating to the user via the user interface that the current scan direction is progressing towards the target plane. Otherwise, if the change in the context-aware parameter does not tend towards the target graph, the method proceeds to 1118 to indicate to the user that the current scan is not progressing towards the target scan plane. Based on these indications, the operator can adjust the scan path of the probe to change the current scan plane and / or scan direction with respect to the larger human anatomy until the current scan is progressing towards the target scan plane to achieve the target scan image.

[0141] In this way, method 1100 can continue to monitor the changes in the current scan image based on the currently generated context-aware graph in real time, and indicate to the user whether the scan direction is progressing towards the target scan plane to achieve the target scan image. In some examples, based on the changes determined as discussed above, the context-aware module can determine the directionality of the current scan with respect to the larger human anatomy (e.g., top to bottom, front to back, etc.), compare it with the desired scan directionality with respect to the larger human anatomy (based on the target context-aware graph), and indicate to the user the desired direction for the probe to move from the current scan plane to achieve the target scan plane.

[0142] Although described above with respect to an ultrasound imaging system Figures 4 to 7 and Figure 11The method at [location], it should be understood that context-aware graphics can be generated for any medical scan image produced by a medical imaging modality, and the medical imaging modality can include, but is not limited to, imaging systems such as MRI, CT, SPECT, and x-ray imaging systems.

[0143] Go to Figure 9 , which shows a set of context-aware graphics 900 that can be displayed on a user interface (such as Figure 1 interface 115 at [location] or Figure 2 display device 214 at [location]), including an initial context-aware graphic 910, a target context-aware graphic 940, a first transformation graphic 920, and a second transformation graphic 930. The initial context-aware graphic 910 can be based on an initial scan image obtained based on signals from an ultrasound probe during an ultrasound scan. In addition, the target context-aware graphic can be based on a selected target scan image. It should be understood that depicting the target scan image in a context-aware representation enables a user to select the target scan image. The first transformation graphic 920 and the second transformation graphic 930 can be generated by a processor, as discussed above with respect to Figure 7 . The first transformation graphic and the second transformation graphic show the desired changes to the initial context-aware graphic in order to achieve the target scan image. The initial context-aware graphic 910 and the target context-aware graphic 940 are similar to the initial context-aware graphic and the target context-aware graphic superimposed on the initial scan image and the target scan image shown at 840 and 850 in FIG. 8, respectively, and are thus similarly numbered. For the sake of brevity, the description of similarly numbered features is omitted here. Although this example only depicts context-aware graphics, it should be understood that the context-aware graphics can be displayed to an operator as one or more of an overlay on the corresponding scan image (as shown in FIG. 8), a context-aware graphic without a scan image (as shown here at Figure 9 ), and a context-aware graphic positioned adjacent to the scan image (not shown).

[0144] As depicted at 910, the initial context-aware figure 910 shows the triangular shape of the initial context-aware figure including three nodes, and the first node 842 is at an acute angle θ1 with respect to the reference axis 912 passing through the centroid of the second node 844. However, the target context-aware figure 940 shows a line segment instead of the overall triangular shape, and the first node 842 is at an obtuse angle θ4 with respect to the reference axis 912. Based on the initial figure parameters and the target figure parameters (including node size, node position, and number of nodes), the first transformed context-aware figure 920 and the second transformed context-aware figure 930 can be displayed to the user, thereby depicting how the subsequent context-aware figures after the initial figure can appear as the scan progresses in the direction towards the target scan image. For example, as the scan progresses in the direction towards the target scan image, the angle θ1 can increase to θ2 at 920 and further increase to θ3 at 930 until the target angle θ4 is achieved. The change in the relative position of the first node with respect to the reference axis indicates the desired change in the relative position of the internal feature represented by the first node. In addition, the processor can draw the desired change of the initial context-aware figure with respect to the larger human anatomy and indicate the desired direction of probe movement. In this example, the processor can determine that in order to achieve the target scan image from the initial scan plane via the first transformed scan plane and the second transformed scan plane, the probe can move from the initial position towards the anterior direction with respect to the larger human anatomy. With respect to Figure 10 shows a non-limiting example of depicting the changes in the context-aware parameter node size and relative node position with respect to the larger human anatomy.

[0145] Go to Figure 10 , figures 1000 and 1050 show non-limiting examples that illustrate the changes in the relative size and position of one or more nodes in the context of the larger human anatomy. Specifically, figures 1000 and 1050 respectively show the changes in the relative size and relative position of the first internal feature with respect to the second internal feature as the ultrasound probe moves in the anterior / upper to posterior / lower direction of the human body.

[0146] The change in the relative size of a first internal feature relative to a second internal feature as the scan progresses in an anterior-to-posterior direction relative to the human anatomy can be depicted by 1010. Specifically, the change in the relative size of the internal features is depicted based on the change in the node sizes. Thus, the Y-axis indicates the size of a first node representing the first internal feature relative to the size of a second node representing the second internal feature, and the X-axis indicates the probe movement direction relative to the larger human anatomy. The current relative size is indicated at 1020, the initial relative size is indicated at 1030, and the target relative size is indicated at 1040. Based on graph 1010, the current size has increased relative to the initial size, thus indicating that the probe is moving in a posterior-to-anterior direction. However, the desired direction of probe movement based on the initial size and the target size can be an anterior-to-posterior direction such that the relative size decreases. Thus, based on the initial size, current size, and target size of the first node and the second node, the processor can plot the current position / current scan plane relative to the larger human anatomy and provide the user with one or more indications, including the desired direction of probe movement to achieve the target scan plane, the expected (desired) change in the node size in the desired direction of probe movement, and the expected (desired) change in the overall shape of the initial context-aware graph (i.e., the initial graph based on the initial scan image at the initial scan plane) to achieve the target context-aware graph based on the target scan image at the target scan plane.

[0147] An exemplary change in the relative position of a first internal feature relative to a second internal feature as the scan progresses in an anterior-to-posterior direction relative to the human anatomy is depicted by 1090. Specifically, the change in the relative position of the internal features is depicted based on the change in the angular position of the first internal feature relative to a reference axis passing through the second internal feature. Thus, the Y-axis indicates the angular position of a first node representing the first internal feature relative to the angular position of a second node representing the second internal feature, where the angular position increases from an acute angle to an obtuse angle in the y-axis direction, and the X-axis indicates the probe movement direction relative to the larger human anatomy. Similar to graph 1000, the current position, initial position, and target position are indicated by 1060, 1070, and 1080, respectively.

[0148] As indicated by 1090, to achieve the target angular position depicted at 1080, the probe can move in an anterior-to-posterior direction such that the relative angle decreases from the initial position to the target position. However, when an increase in the relative angular position is indicated at the current position, it can be inferred that from the initial scan plane to the current scan plane, the probe has moved in a direction opposite to the desired direction, and thus the processor can provide an indication to the operator to move the probe in a posterior direction relative to the larger human anatomy to achieve the target relative position.

[0149] Thus, as discussed with respect to relative node sizes, based on the initial angular positions, current angular positions, and target angular positions of the first and second nodes, the processor can map the current position / current scan plane relative to a larger human anatomy and provide the user with one or more indications, including the desired direction of probe movement to achieve the target scan plane, the expected (desired) change in the relative node positions in the desired direction of probe movement, and the expected (desired) change in the overall shape of the initial context-aware graphic (i.e., the initial graphic based on the initial scan image at the initial scan plane) to achieve the target context-aware graphic based on the target scan image at the target scan plane. In summary, any scan image of a scan plane can be used to generate a context-aware graphic based on the segmentation data of the scan image. The context-aware graphic can include one or more nodes that represent one or more internal anatomical features in the scan image and form one or more vertices of the context-aware graphic, and also include one or more line segments that form the edges of the context-aware graphic, each line segment connecting at least two nodes, where the number of nodes is based on the number of the one or more internal anatomical features identified in the scan image. Additionally, context awareness relative to a larger human anatomy can be determined based on the current context-aware graphic or the initial context-aware graphic and the target context-aware graphic.

[0150] The technical effects of generating and displaying context-aware annotations on the identified internal anatomical features of a medical scan image can include providing real-time visual feedback to the operator performing the medical scan to indicate the current scan plane and guiding the operator towards the target scan plane. Thus, the reliance on sensor-based systems (e.g., for haptic feedback) coupled to a medical device such as an ultrasound probe is reduced. This in turn reduces manufacturing and maintenance costs while improving accuracy and efficiency. The technical effects of generating and displaying context-aware annotations on the identified internal anatomical features also include the real-time perception of the current scan plane relative to the target scan plane in the context of a larger human anatomy. Thus, the user can reach the target scan plane faster and further identify target planes that may otherwise be difficult to discern due to poor resolution, lack of experience, etc. Thus, medical scans can be performed with increased accuracy and improved efficiency.

[0151] Embodiments of a method for a medical imaging processor include: acquiring a medical scan image; identifying one or more internal features in the medical scan image; generating a context-aware graphic based on the relative sizes and relative positions of the one or more internal features; and displaying the context-aware graphic on a display portion of a user interface communicatively coupled to the medical imaging processor; wherein the context-aware graphic includes relative position annotations and relative size annotations for each of the one or more internal features identified in the medical scan image.

[0152] A first example of the method includes wherein identifying the one or more internal features is based on a segmentation model; and wherein generating the context - aware graphic includes separating the relative size and relative position attributes of each of the one or more internal features from the output of the segmentation model. In a second example of the method (which optionally includes the first example), the method further includes wherein generating the context - aware graphic includes depicting each of the one or more internal features by nodes, wherein the size of each node is based on the relative size of the one or more internal features. In a third example of the method (which optionally includes one or both of the first example and the second example), the method further includes wherein generating the context - aware graphic includes determining the centroid of each node and depicting the relative position of each node by one or more line segments connecting the centroids of each node. In a fourth example of the method (which optionally includes one or more or each of the first example to the third example), the method further includes wherein the overall shape of the context - aware graphic is based on the relative position of the one or more internal features. In a fifth example of the method (which optionally includes one or more or each of the first example to the fourth example), the method further includes wherein the acquired medical scan image is based on ultrasound signal data received from an ultrasound probe during an ultrasound scan. In a sixth example of the method (which optionally includes one or more or each of the first example to the fifth example), the method further includes: selecting a target scan image from a set of predetermined scan images stored in a non - transient memory; generating a target context - aware graphic of the target scan image based on one or more target internal features identified in the target scan image and determining a desired direction of movement of the ultrasound probe relative to the human anatomy based on the context - aware graphic and the target context - aware graphic; wherein the target awareness graphic includes second relative position annotations and second relative size annotations of each of the one or more target internal features identified in the target scan image. In a seventh example of the method (which optionally includes one or more or each of the first example to the sixth example), the method further includes displaying the context - aware graphic, the target context - aware graphic, and the desired direction of movement of the ultrasound probe on a display portion of a user interface.

[0153] One embodiment relates to a method for a medical imaging processor, the method comprising: acquiring a current scan image during an ultrasound scan, the current scan image being based on real-time scan data from an ultrasound probe; identifying one or more anatomical features in the current scan image based on a segmentation model; separating one or more attributes of each internal feature among the one or more internal features based on an output of the segmentation model, the one or more attributes including dimension information and position information of each anatomical feature among the one or more anatomical features; generating a current context-aware graphic based on the separated one or more attributes; and displaying the current context-aware graphic on a display portion of a user interface coupled to the processor; wherein the current context-aware graphic includes a first annotation and a second annotation for each internal feature among the one or more internal features; and wherein the first annotation is based on the relative dimensions of each internal feature among the one or more internal features, and the second annotation is based on the relative position of each internal feature among the one or more internal features. A first example of the method includes wherein the first annotation includes nodes representing each internal feature among the one or more internal features, each node having a geometric shape and the size of each node being based on the relationship between the size of the corresponding internal feature and the sizes of one or more other internal features in the current scan image. In a second example of the method (which optionally includes the first example), the second annotation includes line segments connecting at least two nodes at their respective centroids, the length of each line segment being based on the distance between two corresponding internal features represented by the two nodes; and the angle between the line segment and a reference axis being based on the relative position of the two corresponding internal features. In a third example of the method (which optionally includes one or both of the first example and the second example), the method further includes selecting a target scan image during the ultrasound scan, the target scan image being selected from a set of predetermined target scan images; and wherein the target scan image is based on one or more of a target scan plane, one or more desired anatomical features for ultrasound evaluation, and a medical condition to be evaluated during the scan. In a fourth example of the method (which optionally includes one or more or each of the first example to the third example), the method further includes: determining one or more transformed scan planes based on a current context transformation graphic and a target context transformation graphic; identifying one or more transformed scan images based on the one or more transformed scan planes; generating one or more transformed context-aware graphics based on the one or more transformed scan images; and displaying the one or more context-aware graphics on the display portion; wherein determining the one or more transformed scan planes is based on a desired direction of probe movement based on the current transformation graphic and the target transformation graphic.In a fifth example of the method (which optionally includes one or more or each of the first example to the fourth example), determining one or more transformed scan planes includes determining a current scan plane relative to a larger human anatomy based on a current context-aware graphic and a target context-aware graphic. In a sixth example of the method (which optionally includes one or more or each of the first example to the fifth example), the method further includes displaying a desired direction of probe movement on a display portion.

[0154] Embodiments of an imaging system are provided. The imaging system includes: an ultrasound probe; a user interface including a display portion; and a processor configured with instructions in a non-transitory memory, the instructions when executed cause the processor to acquire an ultrasound scan image generated based on scan data from the ultrasound probe; identify one or more anatomical features present in the ultrasound scan image based on a first model; determine one or more context-aware parameters for each of the one or more anatomical features that are outputs from the first model; generate a context-aware graphic of the ultrasound scan image based on the one or more context-aware parameters; and display the context-aware graphic on the display portion; and wherein the one or more context-aware parameters include relative dimensions and relative positions of each of the one or more anatomical features. In a first example of the imaging system, the context-aware graphic includes one or more nodes that form one or more vertices of the context-aware graphic and one or more line segments that form edges of the context-aware graphic, each line segment connecting at least two nodes; and wherein the number of nodes is based on the number of the one or more anatomical features identified in the scan image. In a second example of the imaging system (which optionally includes the first example), each node represents each of the one or more anatomical features; wherein the size of each node is based on the relative size of the corresponding anatomical feature; and wherein the arrangement of the one or more nodes is based on the relative positions of the one or more anatomical features. In a third example of the imaging system (which optionally includes one or both of the first example and the second example), the number of nodes is further based on a node size greater than a threshold. In a fourth example of the imaging system (which optionally includes one or more or each of the first example to the third example), the processor further includes instructions to perform the following operations: acquire a target scan image selected from a set of predetermined target scan images stored in the non-transitory memory; generate a target context-aware graphic based on the target scan image; compare the current context-aware graphic with the target context-aware graphic; and provide an indication of achieving the target scan image via the display portion when the number of nodes in the context-aware graphic is the same as the target number of nodes in the target-aware graphic; the size of each node in the current context-aware graphic is within a threshold limit relative to the size of each corresponding node in the target context-aware graphic; and the overall shape of the context-aware graphic matches the overall target shape of the target context-aware graphic.

[0155] When introducing elements of the various embodiments of the present disclosure, the words "a", "an", and "the" are intended to mean that there is one or more of these elements. The terms "first", "second", etc. do not denote any order, quantity, or importance, but are used to distinguish one element from another. The terms "comprising", "including", and "having" are intended to be inclusive and mean that additional elements may be present in addition to the listed elements. As used herein, terms such as "connected to", "coupled to", etc., an object (e.g., a material, an element, a structure, a component, etc.) can be connected to or coupled to another object, regardless of whether the one object is directly connected or coupled to the other object, or whether there is one or more intervening objects between the one object and the other object. Further, it should be understood that references to "one embodiment" or "an embodiment" of the present disclosure are not intended to be construed as excluding the existence of additional embodiments that also incorporate the recited features.

[0156] In addition to any previously indicated modifications, those skilled in the art can devise many other variations and alternative arrangements without departing from the spirit and scope of this description, and the appended claims are intended to cover such modifications and arrangements. Accordingly, although the information has been specifically and detailedly described above in connection with the currently considered most practical and preferred aspects, it will be apparent to those of ordinary skill in the art that many modifications can be made without departing from the principles and concepts set forth herein, including but not limited to form, function, mode of operation, and use. Similarly, as used herein, in all respects, the examples and embodiments are only intended to be illustrative and should not be construed in any way as restrictive.

Claims

1. A method for a medical imaging processor, the method comprises: acquiring a medical scan image; identifying one or more internal features in the medical scan image; generating a context-aware graph based on the relative sizes and relative positions of the one or more internal features, wherein generating the context-aware graph includes depicting each of the one or more internal features by a node, wherein the size of each node is based on the relative sizes of the one or more internal features; generating the context-aware graph further includes determining the centroid of each node and depicting the relative position of each node by one or more line segments connecting the centroids of each node; and displaying the context-aware graph on a display portion of a user interface communicatively coupled to the medical imaging processor; wherein the context-aware graph includes relative position annotations and relative size annotations for each of the one or more internal features identified in the medical scan image.

2. The method according to claim 1, wherein identifying the one or more internal features is based on a segmentation model; and wherein generating the context-aware graph includes separating the relative size and relative position attributes of each of the one or more internal features from the output of the segmentation model.

3. The method according to claim 1, wherein the overall shape of the context-aware graph is based on the relative positions of the one or more internal features.

4. The method according to claim 1, wherein the acquired medical scan image is based on ultrasound signal data received from an ultrasound probe during an ultrasound scan.

5. The method according to claim 4, the method further comprises: selecting a target scan image from a set of predetermined scan images stored in a non-transitory memory; generating a target context-aware graph of the target scan image based on one or more target internal features identified in the target scan image; and determining a desired direction of movement of the ultrasound probe relative to the human anatomy based on the context-aware graph and the target context-aware graph; wherein the target context-aware graph includes second relative position annotations and second relative size annotations for each of the one or more target internal features identified in the target scan image.

6. The method according to claim 5, the method further includes displaying the context-aware graph, the target context-aware graph, and the desired direction of movement of the ultrasound probe on the display portion of the user interface.

7. A method for a medical imaging processor, the method comprises: during an ultrasound scan, acquiring a current scan image, the current scan image being based on real-time scan data from an ultrasound probe; identifying one or more anatomical features in the current scan image based on a segmentation model; separating one or more attributes of each of the one or more internal features based on the output of the segmentation model, the one or more attributes including size information and position information of each of the one or more anatomical features; Generate a current context-aware graphic based on one or more isolated attributes; and display the current context-aware graphic on a display portion of a user interface coupled to the processor; wherein the current context-aware graphic includes a first annotation and a second annotation for each internal feature of the one or more internal features; and wherein the first annotation is based on the relative size of each internal feature of the one or more internal features, and the second annotation is based on the relative position of each internal feature of the one or more internal features; wherein the first annotation includes nodes representing each internal feature of the one or more internal features, each node having a geometry and size of each node, the size being based on the size of the corresponding internal feature relative to the size of one or more other internal features in the current scan image; wherein the second annotation includes line segments connecting at least two nodes at their respective centroids, the length of each line segment being based on the distance between two corresponding internal features represented by the two nodes; and wherein the angle between the line segment and a reference axis is based on the relative position of the two corresponding internal features.

8. The method according to claim 7, the method further comprising selecting a target scan image during the ultrasound scan, the target scan image being selected from a set of predetermined target scan images; and wherein the target scan image is based on one or more of a target scan plane, one or more desired anatomical features for ultrasound evaluation, and a medical condition to be evaluated during the scan.

9. The method according to claim 8, the method further comprising: Determine one or more transformed scan planes based on a current context transformation graphic and a target context transformation graphic; Identify one or more transformed scan images based on the transformed scan planes; Generate one or more transformed context-aware graphics based on the one or more transformed scan images; and display the one or more transformed context-aware graphics on the display portion; wherein Determining one or more transformed scan planes is based on a desired direction of probe movement based on the current context transformation graphic and the target context transformation graphic.

10. The method according to claim 9, wherein determining one or more transformed scan planes includes determining a current scan plane relative to a larger human anatomy based on the current context-aware graphic and the target context-aware graphic.

11. The method according to claim 9, the method further comprising displaying the desired direction of probe movement on the display portion.

12. An imaging system, the imaging system comprising: An ultrasound probe; A user interface, the user interface including a display portion; and A processor, the processor being configured with instructions in a non-transitory memory, the instructions when executed causing the processor to: Acquire an ultrasound scan image generated based on scan data from the ultrasound probe; Identify one or more anatomical features present in the ultrasound scan image based on a first model; Determine one or more context-aware parameters for each of the one or more anatomical features that are the output from the first model; Generate a context-aware graphic of the ultrasound scan image based on the one or more context-aware parameters; And Display the context-aware graphic on the display portion; and wherein the one or more context-aware parameters include the relative size and relative position of each of the one or more anatomical features; wherein the context-aware graphic includes one or more nodes that form one or more vertices of the context-aware graphic and one or more line segments that form the edges of the context-aware graphic, each line segment connecting at least two nodes; and wherein the number of nodes is based on the number of the one or more anatomical features identified in the scan image; wherein each node represents each of the one or more anatomical features; wherein the size of each node is based on the relative size of the corresponding anatomical feature; and wherein the arrangement of the one or more nodes is based on the relative position of the one or more anatomical features.

13. The system according to claim 12, wherein the number of nodes is further based on a node size greater than a threshold.

14. The system according to claim 12, wherein the processor further comprises instructions to perform the following: Acquire a target scan image, the target scan image being selected from a set of predetermined target scan images stored in non-transitory memory; Generate a target context-aware graphic based on the target scan image; Compare the current context-aware graphic with the target context-aware graphic; And Provide an indication that the target scan image has been achieved via the display portion when the number of nodes in the context-aware graphic is the same as the target number of nodes in the target context-aware graphic; the size of each node in the current context-aware graphic is within a threshold limit relative to the size of each corresponding node in the target context-aware graphic; and the overall shape of the context-aware graphic matches the overall target shape of the target context-aware graphic.

Images