Systems and methods for uncertainty aware caliper placement

Abstract

Systems and methods for characterizing uncertainties on boundaries of a region of interest include inputting, via a processor (44), an ultrasound image having the region of interest into a trained neural network (58). The system and method also include outputting, via the processor (44), a first prediction (108) of a boundary of the region of interest and a second prediction (110) of a region segmentation of the region of interest from the trained neural network (58). The system and method also include determining, via the processor (44), an uncertainty on the boundary based on a mismatch between the first prediction (108) and the second prediction (110).

Description

Background Art

[0001] The subject matter disclosed herein relates to ultrasound image processing, and more particularly to systems and methods for uncertainty-aware caliper placement.

[0002] Ultrasound devices can be used to image targets such as organs and soft tissues in the human body as well as non-human targets. For example, in addition to ultrasound imaging of humans, animals, etc., ultrasound devices can also be used for applications such as ultrasound / acoustic sensing, non-destructive evaluation (NDE), ultrasound therapy (e.g., high-intensity focused ultrasound (HIFU)), etc.

[0003] Ultrasound devices can use real-time, non-invasive high-frequency sound waves to produce a series of two-dimensional (2D) images and / or three-dimensional (3D) images. Sound waves can be emitted by a transmitting transducer, and reflections of the emitted sound waves can be received by a receiving transducer. The received sound waves can then be processed to display an image of the target. A caliper is placed on the ultrasound image so that the target can be measured. Placing a caliper on an ultrasound image for biological measurement is a task that has been automated in many cases. The caliper is typically placed on the outer boundary of the region of interest. Region segmentation often encounters uncertainty at the edge of the region of interest, so the caliper position can be placed in a less desirable location. A less desirable location can be a shadow area in the ultrasound image or an area with low image gradient. Summary of the invention

[0004] The following shows an overview of certain embodiments disclosed herein. It should be understood that these aspects are provided only to provide the reader with a brief overview of these specific embodiments, and these aspects are not intended to limit the scope of the present disclosure. In fact, the present disclosure may cover various aspects that may not be shown below.

[0005] In one embodiment, a computer-implemented method for characterizing uncertainty in a boundary of a region of interest is provided. The computer-implemented method includes inputting, via a processor, an ultrasound image having a region of interest into a trained neural network. The computer-implemented method also includes outputting, via the processor, from the trained neural network a first prediction of a boundary of the region of interest and a second prediction of a region segmentation of the region of interest. The computer-implemented method also includes determining, via the processor, uncertainty in the boundary based on a mismatch between the first prediction and the second prediction.

[0006] In one embodiment, the computer-implemented method includes providing, via a processor, a user-perceptible indication of uncertainty when there is a mismatch between the first prediction and the second prediction.

[0007] In one embodiment, providing a user-perceivable indication of uncertainty includes indicating any areas of uncertainty on the boundary on the ultrasound image.

[0008] In one embodiment, the user-perceivable indication includes highlighting the uncertainty area on the ultrasound image.

[0009] In one embodiment, a computer-implemented method includes automatically placing, via a processor, a pair of calipers over a region of interest in an ultrasound image without uncertainty on a boundary.

[0010] In one embodiment, the ultrasound image includes a pair of calipers marked over the region of interest.

[0011] In one embodiment, the computer-implemented method includes providing an alert to a user via a processor when at least a portion of the pair of calipers is placed in an uncertainty region on the boundary.

[0012] In one embodiment, the computer-implemented method includes providing, via a processor, a user-perceptible indication on the ultrasound image of either end of a caliper placed in the uncertainty region on the boundary.

[0013] In one embodiment, the user-perceivable indication includes causing either end of the pair of calipers in the uncertainty region to appear a different color than the remainder of the pair of calipers.

[0014] In one embodiment, the computer-implemented method includes changing, via a processor, placement of the pair of calipers over the region of interest in the ultrasound image when at least a portion of the pair of calipers is placed in a region of uncertainty on a boundary.

[0015] In another embodiment, a system for characterizing uncertainty in a boundary of a region of interest is provided. The system includes a memory encoding a processor executable routine. The system also includes a processor configured to access the memory and execute the processor executable routine, wherein the routine, when executed by the processor, causes the processor to perform an action. The action includes inputting an ultrasound image having a region of interest into a trained neural network. The action also includes outputting a first prediction of a boundary of the region of interest and a second prediction of a region segmentation of the region of interest from the trained neural network. The action also includes determining the uncertainty in the boundary based on a mismatch between the first prediction and the second prediction.

[0016] In one embodiment, the action includes providing a user-perceivable indication of uncertainty when there is a mismatch between the first prediction and the second prediction.

[0017] In one embodiment, providing a user-perceivable indication of uncertainty includes indicating any areas of uncertainty on the boundary on the ultrasound image.

[0018] In one embodiment, the actions include causing a processor to automatically place a pair of calipers over the region of interest in the ultrasound image in the absence of uncertainty on the boundaries.

[0019] In one embodiment, the ultrasound image includes a pair of calipers marked over the region of interest.

[0020] In one embodiment, the actions include causing the processor to provide an alert to a user when at least a portion of the pair of calipers is positioned within an uncertainty region on the boundary.

[0021] In one embodiment, the action includes providing a user-perceptible indication on the ultrasound image of either end of the pair of calipers being placed in the uncertainty region on the boundary.

[0022] In one embodiment, the user-perceivable indication includes causing either end of the pair of calipers in the uncertainty region to appear a different color than the remainder of the pair of calipers.

[0023] In one embodiment, the actions include causing the processor to change placement of the pair of calipers over the region of interest in the ultrasound image when at least a portion of the pair of calipers is placed in a region of uncertainty on the boundary.

[0024] In another embodiment, a non-transitory computer readable medium includes processor executable code that, when executed by a processor, causes the processor to perform actions. The actions include inputting a medical image having a region of interest into a trained neural network. The actions also include outputting a first prediction of a boundary of the region of interest and a second prediction of a region segmentation of the region of interest from the trained neural network. The actions also include determining uncertainty on the boundary based on a mismatch between the first prediction and the second prediction.

[0025] In one embodiment, the action includes providing a user-perceivable indication of uncertainty when there is a mismatch between the first prediction and the second prediction.

[0026] In one embodiment, providing a user-perceivable indication of uncertainty includes indicating any areas of uncertainty on the boundary on the ultrasound image.

[0027] In one embodiment, the actions include causing a processor to automatically place a pair of calipers over the region of interest in the ultrasound image in the absence of uncertainty on the boundaries.

[0028] In one embodiment, the ultrasound image includes a pair of calipers marked over the region of interest.

[0029] In one embodiment, the actions include causing the processor to provide an alert to a user when at least a portion of the pair of calipers is positioned within an uncertainty region on the boundary.

[0030] In one embodiment, the action includes providing a user-perceptible indication on the ultrasound image of either end of the pair of calipers being placed in the uncertainty region on the boundary.

[0031] In one embodiment, the user-perceivable indication includes causing either end of the pair of calipers in the uncertainty region to appear a different color than the remainder of the pair of calipers.

[0032] In one embodiment, the actions include causing the processor to change placement of the pair of calipers over the region of interest in the ultrasound image when at least a portion of the pair of calipers is placed in a region of uncertainty on the boundary. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] These and other features, aspects, and advantages of the present subject matter will be better understood when the following detailed description is read with reference to the accompanying drawings, in which like characters refer to like parts throughout the several views, and in which:

[0034] Figure 1 is an embodiment of a block diagram of an ultrasound system according to aspects of the present disclosure;

[0035] Figure 2 is a schematic diagram of an example of a first type of inconsistency between region segmentation and boundary segmentation when predicting the boundary of a region of interest;

[0036] Figure 3 is a schematic diagram of an example of a second type of inconsistency between region segmentation and boundary segmentation when predicting the boundary of a region of interest;

[0037] Figure 4 is a schematic diagram showing how to characterize uncertainty on the boundary of a region of interest in an image according to aspects of the present disclosure;

[0038] Figure 5 are examples of an ultrasound image with a boundary segmentation mask and the same ultrasound image with a region segmentation mask (e.g., with boundary uncertainty) according to aspects of the present disclosure;

[0039] Figure 6 is another example of an ultrasound image with a boundary segmentation mask and the same ultrasound image with a region segmentation mask (e.g., with boundary uncertainty) according to aspects of the present disclosure;

[0040] Figure 7are examples of an ultrasound image with a boundary segmentation mask and the same ultrasound image with a region segmentation mask (e.g., with highly defined boundaries) according to aspects of the present disclosure;

[0041] Figure 8 is another example of an ultrasound image with a boundary segmentation mask and the same ultrasound image with a region segmentation mask (e.g., with highly determined boundaries) according to aspects of the present disclosure;

[0042] Fig. 9 is a flow chart of a method for characterizing uncertainty on a boundary of a region of interest (eg, where a pair of calipers are not used to mark an ultrasound image) according to aspects of the present disclosure;

[0043] Fig.10 is a flow chart of a method for characterizing uncertainty on a boundary of a region of interest (eg, where a pair of calipers are placed on an ultrasound image) according to aspects of the present disclosure;

[0044] Fig.11 is a schematic diagram illustrating training a neural network to characterize uncertainty on a boundary of a region of interest of an image according to aspects of the present disclosure;

[0045] Fig.12 is an example of segmenting a predicted boundary from a boundary overlaid on a predicted segmented region of interest in an image in accordance with aspects of the present disclosure;

[0046] Fig.13 According to the various aspects of the present disclosure, Fig.12 Examples of characterizations of uncertainty in predictions;

[0047] Fig.14 is another example of segmenting a predicted boundary from a boundary overlaid on a predicted segmented region of interest in an image according to aspects of the present disclosure;

[0048] Fig.15 According to the various aspects of the present disclosure, Fig.14 Examples of characterizations of uncertainty in predictions;

[0049] Fig.16 is an example of an ultrasound image of a properly placed caliper position (e.g., a pair of calipers for lesion diameter measurement of an ovarian lesion) in accordance with aspects of the present disclosure; and

[0050] Fig.17 is an example of an ultrasound image of properly placed caliper positions (eg, a pair of calipers for a solid portion of a lesion measurement of an ovarian lesion) in accordance with aspects of the present disclosure. DETAILED DESCRIPTION

[0051] One or more specific embodiments will be described below. In order to provide a concise description of these embodiments, not all features of an actual implementation are described in the specification. It should be understood that in the development of any such actual implementation, as in any engineering or design project, many implementation-specific decisions must be made to achieve the developer's specific goals, such as complying with system-related and business-related constraints that may vary from implementation to implementation. In addition, it should be understood that such development efforts may be complex and time-consuming, but are still routine tasks for design, fabrication, and manufacturing for ordinary technicians who benefit from this disclosure.

[0052] When introducing elements of various embodiments of the present subject matter, the articles "a," "an," "the," and "said" are intended to indicate that there are one or more elements. The terms "comprising," "including," and "having" are intended to be inclusive, and mean that there may be additional elements in addition to the listed elements. Furthermore, any numerical examples in the following discussion are intended to be non-limiting, and thus the additional numerical values, ranges, and percentages are within the scope of the disclosed embodiments.

[0053] Some general information is presented to provide a general background to various aspects of the disclosure and to facilitate understanding and explanation of certain technical concepts described herein.

[0054] As used herein, the term "image" refers broadly to both a visible image and data representing a visible image. However, many embodiments generate (or are configured to generate) at least one visible image. In addition, as used herein, the phrase "image" is used to refer to ultrasound modes, such as B-mode (2D mode), M-mode, three-dimensional (3D) mode, CF mode, PW Doppler, CW Doppler, MGD, and / or a sub-mode of B-mode and / or CF, such as shear wave elastography (SWEI), TVI, Angio, B-flow, BMI, BMI_Angio, and in some cases also includes MM, CM, TVD, wherein "image" and / or "plane" includes a single beam or multiple beams.

[0055] Furthermore, as used herein, the term processor or processing unit refers to any type of processing unit, whether single-core or multi-core: CPU, accelerated processing unit (APU), graphics board, DSP, FPGA, ASIC, or a combination thereof, that can perform the required computations required by various implementations.

[0056] The deep learning (DL) methods discussed herein may be based on artificial neural networks and may therefore encompass one or more of the following: deep neural networks, fully interconnected networks, convolutional neural networks (CNNs), unfolded neural networks, perceptrons, codecs, recurrent networks, transformer networks, wavelet filter banks, u-nets, generative adversarial networks (GANs), dense neural networks (e.g., residual dense networks (RDNs)), or other neural network architectures. Neural networks may include shortcuts, activations, batch normalization layers, and / or other features. These techniques are referred to herein as deep learning techniques, but the term may also be used with particular reference to the use of deep neural networks, which are neural networks with multiple layers.

[0057] As discussed herein, deep learning techniques (which may also be referred to as deep machine learning, hierarchical learning, or deep structured learning) are a branch of machine learning techniques that employ mathematical representations of data and artificial neural networks for learning and processing such representations. For example, deep learning methods may be characterized as using one or more algorithms to extract or model highly abstract concepts of a class of data of interest. This may be accomplished using one or more processing layers, wherein each layer typically corresponds to a different level of abstraction, and thus may employ or utilize different aspects of the initial data or the output of a previous layer (i.e., a hierarchical or cascaded structure of layers) as the target of the process or algorithm of a given layer. In the context of image processing or reconstruction, this may be characterized as different layers corresponding to different feature levels or resolutions in the data. In general, the processing of one representation space to the next level of representation space may be viewed as a "stage" of the process. Each stage of the process may be performed by a separate neural network or by different parts of a larger neural network.

[0058] The present disclosure provides systems and methods for characterizing uncertainty on the boundary of a region of interest. Specifically, the disclosed systems and methods include inputting an ultrasound image having a region of interest into a trained neural network. The disclosed systems and methods also include outputting a first prediction of the boundary of the region of interest and a second prediction of the regional segmentation of the region of interest from the trained neural network. The disclosed systems and methods also include determining the uncertainty on the boundary based on a mismatch between the first prediction and the second prediction. In certain embodiments, the disclosed systems and methods include providing a user-perceivable indication of uncertainty when there is a mismatch between the first prediction and the second prediction. This may include indicating any uncertainty areas on the boundary on the ultrasound image. In certain embodiments, the disclosed systems and methods include automatically placing a pair of calipers on the region of interest in the ultrasound image when there is no uncertainty on the boundary.

[0059] Please note that although the disclosed technology is described with respect to ultrasound images, the disclosed technology can also be used for medical images obtained from other imaging modalities (e.g., magnetic resonance imaging (MRI), nuclear medicine (NM) imaging, computed tomography (CT) imaging, conventional X-ray imaging, etc.).

[0060] In some embodiments, the input ultrasound image may include a pair of calipers placed or marked on the region of interest. In this case, in some embodiments, the disclosed system and method include providing an alert to the user when at least a portion of the pair of calipers is placed in an uncertainty region on the boundary. In some embodiments, the disclosed system and method include providing a user-perceivable indication of either end of the pair of calipers placed in the uncertainty region on the boundary on the ultrasound image. For example, either end of the pair of calipers in the uncertainty region has a different color from the rest of the pair of calipers. In another example, the uncertainty region itself may be highlighted on the ultrasound image. In some embodiments, the disclosed system and method include changing the placement of the pair of calipers on the region of interest in the ultrasound image when at least a portion of the pair of calipers is placed in the uncertainty region on the boundary. The disclosed embodiment enables characterization of uncertainty on the boundary of the region and indication thereof to the user, thereby enabling correct positioning (e.g., automatic or manual) of a pair of calipers to improve workflow and provide better patient outcomes. In addition, uncertainty is determined in a faster manner and more accurately.

[0061] With the foregoing in mind, and by providing useful context, Figure 1 A high-level view of components of an ultrasound system 10 that may be employed in accordance with the methods of the present invention is depicted. The ultrasound system 10 shown includes a transducer array 14 having transducer elements adapted to be in contact with a subject or patient 18 during an imaging procedure. The transducer array 14 may be configured as a bidirectional transducer capable of transmitting ultrasound waves into the subject or patient 18 and receiving such energy from the subject or patient 18. In such a specific implementation, in a transmit mode, the transducer array elements convert electrical energy into ultrasound waves and transmit them into the patient 18. In a receive mode, the transducer array elements convert ultrasound energy (backscattered waves) received from the patient 18 into electrical signals.

[0062] Each transducer element is associated with a corresponding transducer circuit, which may be provided as one or more application specific integrated circuits (ASICs) 20 that may be present in the probe or probe handle. That is, each transducer element in the array 14 is electrically connected to a corresponding pulse generator 22, a transmit / receive switch 24, a preamplifier 26, a scan gain 34, and / or an analog-to-digital (A / D) converter 28 that is part of or disposed on the ASIC 20. In other specific implementations, the arrangement may be simplified or otherwise altered. For example, the components shown in the circuit 20 may be disposed upstream or downstream of the depicted arrangement, however, the depicted basic functions will generally still be provided for each transducer element. In the depicted example, the circuit functions mentioned are conceptualized as being implemented on a single ASIC 20 (represented by the dashed lines), however, it is understood that some or all of these functions may be disposed on the same or different integrated circuits.

[0063] Likewise Figure 1 As depicted, a variety of other imaging components are provided to enable the formation of images using the ultrasound system 10. Specifically, the depicted example of the ultrasound system 10 also includes a beamformer 32, a control panel 36, a receiver 38, and a scan converter 40, which cooperate with the transducer circuit to produce an image or series of images 42 that can be stored and / or displayed to an operator or otherwise processed as discussed herein. The transducer array 14 can be connected via a wired connection or a wireless connection (e.g., via a wireless communication unit that is part of the transducer array communicating over a wi-fi network, utilizing a wireless communication device that communicates with the transducer array via a wireless network). The ultrasound data is transmitted to the beamformer by a processing component 44 (e.g., a microprocessor or processing circuit) and a memory 46 of the system 10 (such as may be present in the control panel 36) may be used to execute stored software code, instructions or routines for processing the acquired ultrasound signals to generate meaningful images and / or motion frames that may be displayed on a display 47 of the ultrasound system 10. The term "code" or "software code" as used herein refers to any instructions or set of instructions for controlling the ultrasound system 10. The code or software code may exist in a computer executable form such as machine code (which is a set of instructions and data that is directly executed by the processing component 44 of the control panel 36), a human understandable form such as source code (which may be compiled for execution by the processing component 44 of the control panel 36), or an intermediate form such as object code (generated by a compiler). In some embodiments, the ultrasound system 10 may include multiple controllers.

[0064] As an example, the memory 46 may store processor executable software code or instructions (e.g., firmware or software) that are tangibly stored on a non-transitory computer readable medium. Additionally or alternatively, the memory 46 may store data. As an example, the memory 46 may include volatile memory (such as random access memory (RAM)) and / or non-volatile memory (such as read-only memory (ROM), flash memory, hard drive, or any other suitable optical, magnetic or solid-state storage medium or a combination thereof. In addition, the processing unit 44 may include multiple microprocessors, one or more "general purpose" microprocessors, one or more special-purpose microprocessors, and / or one or more application-specific integrated circuits (ASICS) or a combination thereof. For example, the processing unit 44 may include one or more reduced instruction set (RISC) or complex instruction set (CISC) processors. The processing unit 44 may include multiple processors and / or the memory 46 may include multiple memory devices.

[0065] The processing component 44 may characterize the uncertainty on the boundary of the region of interest. Specifically, the processing component 44 may input an ultrasound image having the region of interest into a trained neural network, output a first prediction of the boundary of the region of interest and a second prediction of the region segmentation of the region of interest from the trained neural network, and determine the uncertainty on the boundary based on a mismatch between the first prediction and the second prediction. In some embodiments, when there is a mismatch between the first prediction and the second prediction, the processing component 44 provides a user-perceivable indication of the uncertainty. This may include indicating any uncertainty areas on the boundary on the ultrasound image. In some embodiments, in the absence of uncertainty on the boundary, the processing component 44 automatically places a pair of calipers on the region of interest in the ultrasound image.

[0066] In some embodiments, the input ultrasound image may include a pair of calipers placed or marked on the region of interest. In this case, in some embodiments, when at least a portion of the pair of calipers is placed in the uncertainty region on the border, the processing component 44 provides an alert to the user. In some embodiments, the processing component 44 provides a user-perceivable indication of either end of the pair of calipers placed in the uncertainty region on the border on the ultrasound image. For example, either end of the pair of calipers in the uncertainty region has a different color from the rest of the pair of calipers. In another example, the uncertainty region itself may be highlighted on the ultrasound image. In some embodiments, when at least a portion of the pair of calipers is placed in the uncertainty region on the border, the processing component 44 changes the placement of the pair of calipers on the region of interest in the ultrasound image.

[0067] The ultrasound information may be processed by other or different mode-related modules (e.g., B-mode, color Doppler, power Doppler, M-mode, spectral Doppler anatomical M-mode, strain, strain rate, etc.) to form a 2D or 3D data set of image frames, etc. For example, one or more modules may generate B-mode, color Doppler, power Doppler, M-mode, anatomical M-mode, strain, strain rate, spectral Doppler image frames, and combinations thereof, etc. The image frames are stored, and timing information indicating the time when the image frame was acquired in the memory may be recorded with each image frame. These modules may include, for example, a scan conversion module for performing a scan conversion operation, thereby converting the image frame from polar coordinates to Cartesian coordinates. A video processor module may be provided that reads the image frame from the memory and displays the image frame in real time while the patient is being operated on. The video processor module may store the image frame in an image memory, read and display the image from the image memory. The ultrasound system 10 shown may include a console system or a portable system, such as a handheld or laptop system.

[0068] The ultrasound system 10 is operable to continuously acquire ultrasound scan data at a frame rate suitable for the imaging situation under consideration. Typical frame rates may be in the range of 20 to 120, but may be lower or higher. The acquired ultrasound scan data may be displayed on the display 47 at a display rate that is the same as the frame rate, or slower or faster. An image buffer is included to store processed frames of acquired ultrasound scan data that are not scheduled for immediate display. Preferably, the image buffer has sufficient capacity to store frames of ultrasound scan data for at least several minutes. The frames of ultrasound scan data are stored in a manner that is easy to retrieve from them according to their acquisition order or time. The image buffer may be embodied as any known data storage medium.

[0069] Display 47 may be any device capable of conveying visual information to a user. For example, display 47 may include a liquid crystal display, a light emitting diode display, and / or any suitable one or more displays. Display 47 may be operable to present an ultrasound image and / or any suitable information.

[0070] The components of the ultrasound system 10 may be implemented in software, hardware, firmware, etc. The various components of the ultrasound system 10 are communicatively connected. The components of the ultrasound may be implemented separately and / or integrated in various forms.

[0071] For caliper placement, the boundary of the region of interest can be predicted in two (e.g., complementary and independent) ways. The first way is to first perform region segmentation of the region of interest and then perform boundary extraction. The first way is texture driven. The second way is to use direct boundary prediction (e.g., boundary segmentation prediction). The second way is gradient driven. If the first way is consistent with the second way, there is no uncertainty in the boundary prediction. However, when the first way is inconsistent with the second way, there will be uncertainty. These inconsistencies can be manifested in different ways. For example Figure 2 As depicted, the region segmentation (i.e., the first approach) performs under-segmentation, as shown by reference numeral 48, while the direct boundary segmentation prediction (i.e., the second approach) predicts the full boundary (i.e., boundary segmentation), as shown by reference numeral 50. Figure 3 As depicted, region segmentation (ie, the first approach) predicts or segments the entire region of interest, as shown by reference numeral 52 , while direct boundary segmentation prediction under-segments the boundary of the region of interest, as shown by reference numeral 54 .

[0072] Figure 4 is a schematic diagram showing how to characterize the uncertainty on the boundary of the region of interest in the image. Figure 4 As depicted, input 56 is provided to or input into a trained neural network 58 (e.g., a deep learning based neural network). Input 56 is an image (e.g., an ultrasound image) having a region of interest (e.g., a vaginal region). In some embodiments, the ultrasound image includes a pair of calipers placed or marked over the region of interest. In some embodiments, the ultrasound image does not have a pair of calipers placed or marked over the region of interest. The trained neural network 58 is configured to output a prediction of a boundary of the region of interest in the ultrasound image using a boundary segmentation model (i.e., a boundary segmentation prediction 60, y 边界 The trained neural network 58 uses the unsigned distance map to perform boundary segmentation prediction 60. The trained neural network 58 is also configured to output a prediction of the region segmentation of the region of interest in the ultrasound image using the region segmentation model (i.e., region segmentation prediction 62, y 区域). The trained neural network 58 uses conventional Dice loss to make region segmentation predictions 62. The trained neural network 58 is trained to infer inconsistencies (e.g., mismatches) between the boundary segmentation prediction 60 and the region segmentation prediction 62. This inconsistency enables the uncertainty 63 on the boundary (e.g., boundary uncertainty) to be determined based on the uncertainty 63 between the boundary segmentation prediction 60 and the region segmentation prediction 62 by comparing the boundary prediction 60 and the region segmentation prediction 62. Specifically, the edges estimated by the boundary segmentation prediction 60 and the edges of the region segmentation prediction 62 are analyzed. In some embodiments, the determination of the uncertainty 63 is performed by post-processing. In some embodiments, the determination of the uncertainty 63 is performed by another trained neural network or the same trained neural network 58. The uncertainty 63 can be determined by Figure 4 The process described in is determined once and for all.

[0073] The boundary segmentation prediction 60 and the region segmentation prediction 62 reinforce each other at locations along the edge where they have a high degree of certainty. In addition, locations along the edge where the boundary segmentation prediction 60 and the region segmentation prediction 62 lack reinforcement are considered uncertain. Figure 5 is an example of an ultrasound image with a boundary segmentation mask and the same ultrasound image with a region segmentation mask (eg, with boundary uncertainty). Figure 5 The ultrasound images in FIG. 64 are from a transvaginal ultrasound scan. Figure 4 An ultrasound image of a boundary segmentation mask 66 predicted by the trained neural network 58 in FIG. 5 overlaid on a region of interest 68. Image 70 is an ultrasound image having a boundary segmentation mask 66 predicted by the trained neural network 58 in FIG. Figure 4 FIG. 7 is a diagram of the same ultrasound image of the region segmentation mask 72 predicted by the trained neural network 58 in FIG. 6 overlaid on the region of interest 68. There is uncertainty in the boundary of the region of interest 68 between the boundary segmentation in image 64 and the region segmentation in image 70. Specifically, the boundary in the boundary segmentation in image 64 is under-segmented.

[0074] Figure 6 is another example of an ultrasound image with a boundary segmentation mask and the same ultrasound image with a region segmentation mask (eg, with boundary uncertainty). Figure 6 The ultrasound images in FIG. 74 are from a transvaginal ultrasound scan. Figure 4 FIG. 80 is an ultrasound image of a boundary segmentation mask 76 predicted by the trained neural network 58 in FIG. 8 , which is overlaid on a region of interest 78. Figure 480 , which is overlaid on the region of interest 78. There is uncertainty in the boundary of the region of interest 78 between the boundary segmentation in image 74 and the region segmentation in image 80. Specifically, the boundary in the boundary segmentation in image 74 is under-segmented.

[0075] Figure 7 is an example of an ultrasound image with a boundary segmentation mask and the same ultrasound image with a region segmentation mask (eg, with highly defined boundaries). Figure 7 The ultrasound images in FIG. 8 are from a transvaginal ultrasound scan. Image 84 is a Figure 4 FIG. 9 is an ultrasound image of a boundary segmentation mask 86 predicted by the trained neural network 58 in FIG. 8 , which is overlaid on a region of interest 88. Figure 4 90, with the region segmentation mask 92 predicted by the trained neural network 58 overlaid on the region of interest 88. As shown by the well-defined boundary features in both image 84 and image 90, there is a high degree of certainty in the boundary of the region of interest 88 between the boundary segmentation in image 84 and the region segmentation in image 90.

[0076] Figure 8 is another example of an ultrasound image with a boundary segmentation mask and the same ultrasound image with a region segmentation mask (eg, with highly defined boundaries). Figure 8 The ultrasound images in FIG. 9 are from a transvaginal ultrasound scan. Figure 4 FIG. 1 is an ultrasound image of a boundary segmentation mask 96 predicted by the trained neural network 58 in FIG. 1 , which is overlaid on a region of interest 98. Figure 4 100, with the region segmentation mask 102 predicted by the trained neural network 58 overlaid on the region of interest 98. As shown by the well-defined boundary features in both image 94 and image 100, there is a high degree of certainty in the boundary of the region of interest 98 between the boundary segmentation in image 94 and the region segmentation in image 100.

[0077] Fig. 9 is a flow chart of a method 104 for characterizing uncertainty in the boundaries of a region of interest (eg, where a pair of calipers are not used to mark an ultrasound image). One or more steps of the method 104 may be performed by a remote computing system or Figure 1 The processing circuit of the ultrasound system 10 in the embodiment of the present invention is executed. The processing circuit of the ultrasound system 10 in the embodiment of the present invention can be executed simultaneously and / or according to the Fig. 9The different orders shown perform one or more steps of the method 104. Note that in addition to ultrasound images, the method 104 can also be applied to medical images acquired using other types of imaging modalities (e.g., magnetic resonance imaging (MRI), nuclear medicine (NM) imaging, computed tomography (CT) imaging, conventional X-ray imaging, etc.).

[0078] The method 104 includes inputting an ultrasound image having a region of interest (e.g., not marked by a pair of calipers) into a trained neural network 58 (e.g., as Figure 4 The method 104 also includes outputting a first prediction 108 of a boundary of the region of interest from the trained neural network 58 (eg, Figure 4 The y in 边界 60) and a second prediction 110 of the region segmentation of the region of interest (e.g. Figure 4 The y in 区域 62) (box 112). The method 104 also includes determining uncertainty in the boundary (e.g., boundary uncertainty) based on a mismatch between the first prediction 108 of the boundary of the region of interest and the second prediction 110 of the regional segmentation of the region of interest by comparing the first prediction 108 and the second prediction 110 (box 114). Any inconsistency between the first prediction 108 of the boundary of the region of interest and the second prediction 110 of the regional segmentation of the region of interest enables the uncertainty in the boundary to be determined based on the mismatch between the first prediction 108 of the boundary of the region of interest and the second prediction 110 of the regional segmentation of the region of interest. Specifically, the edges estimated by the first prediction 108 of the boundary of the region of interest and the second prediction 110 of the regional segmentation of the region of interest are analyzed. In some embodiments, the determination of uncertainty is performed by post-processing. In some embodiments, the determination of uncertainty is performed by another trained neural network or the same trained neural network 58.

[0079] The method 104 includes determining whether there is a lack of certainty based on the determined uncertainty (block 116). In some embodiments, if there is no lack of certainty, the method 104 includes automatically placing a pair of calipers on the region of interest in the ultrasound image (block 118). In some embodiments, a user can overrule the automatic placement of the pair of calipers. In some embodiments, if there is no lack of certainty, the method 104 includes providing a notification (e.g., a text notification on a display or ultrasound image) that the boundaries of the region of interest are highly determined and / or the pair of calipers can be placed (block 120).

[0080] In some embodiments, if there is a lack of certainty, the method 104 includes providing a user-perceivable indication of the uncertainty (box 122). In some embodiments, the user-perceivable indication includes indicating any uncertainty areas on the border on the ultrasound image. In some embodiments, the user-perceivable indication may be an indicator or mark (which may or may not be accompanied by text indicating the uncertainty area) identifying the uncertainty area. In some embodiments, the user-perceivable indication may highlight the uncertainty area. In some embodiments, if there is a lack of uncertainty, the method 104 includes providing a suggestion to change the parameters of one of the segmentation models (e.g., the boundary segmentation model or the region segmentation model) and re-input the ultrasound image into the trained neural network 58 (box 124). For example, the filter parameters (e.g., the gradient filter) of the boundary segmentation model may be changed. In some embodiments, if there is a lack of uncertainty, the method 104 includes automatically changing the parameters of one of the segmentation models (e.g., the boundary segmentation model or the region segmentation model) and re-inputting the ultrasound image into the trained neural network 58 (box 126). If the parameters are automatically changed and the ultrasound image is re-input into the trained neural network 58, a notification may be provided or provided to the user.

[0081] Fig.10 is a flow chart of a method 128 for characterizing uncertainty in the boundaries of a region of interest (eg, by placing a pair of calipers on an ultrasound image). One or more steps of the method 128 may be performed by a remote computing system or Figure 1 The processing circuit of the ultrasound system 10 in the embodiment of the present invention is executed. The processing circuit of the ultrasound system 10 in the embodiment of the present invention can be executed simultaneously and / or according to the Fig.10 The different orders shown perform one or more steps of the method 128. Note that in addition to ultrasound images, the method 128 can also be applied to medical images acquired using other types of imaging modalities (e.g., magnetic resonance imaging (MRI), nuclear medicine (NM) imaging, computed tomography (CT) imaging, conventional X-ray imaging, etc.).

[0082] The method 128 includes inputting an ultrasound image having a region of interest (e.g., marked by a pair of calipers) into a trained neural network 58 (e.g., as Figure 4 The method 128 also includes outputting a first prediction of the boundary of the region of interest 108 from the trained neural network 58 (eg, Figure 4 The y in 边界 60) and a second prediction of the region segmentation of the region of interest 110 (eg Figure 4 The y in 区域62) (block 132). The method 128 also includes determining uncertainty on the boundary (e.g., boundary uncertainty) based on the mismatch between the first prediction 108 and the second prediction 110 by comparing the first prediction 108 and the second prediction 110 (block 134). Any inconsistency between the first prediction 108 and the second prediction 110 can cause the uncertainty of the boundary to be determined based on the mismatch between the first prediction 108 and the second prediction 110. Specifically, the margins estimated by the first prediction 108 and the second prediction 110 are analyzed. In some embodiments, the determination of uncertainty is performed by post-processing. In some embodiments, the determination of uncertainty is performed by another trained neural network or the same trained neural network 58.

[0083] The method 128 includes determining whether there is a lack of certainty based on the determined uncertainty (block 136). In some embodiments, if there is no lack of certainty, the method 104 includes providing a notification that the boundaries of the region of interest are highly determined and / or a pair of calipers can be placed (block 138). In some embodiments, if there is no lack of certainty, no notification may be provided and the user may continue.

[0084] In some embodiments, if there is a lack of certainty, the method 128 includes providing an alert to the user when at least a portion of the pair of calipers is placed in an uncertainty region on the boundary (box 140). In some embodiments, if there is a lack of certainty, the method 104 includes providing a user-perceivable indication on the ultrasound image of either end of the pair of calipers placed in the uncertainty region on the boundary (box 142). In some embodiments, the user-perceivable indication includes causing either end of the pair of calipers in the uncertainty region to be displayed as a different color than the rest of the pair of calipers. For example, if the color of the pair of calipers is yellow, the color of one end in the uncertainty region may be red. In some embodiments, if there is a lack of certainty, the method 104 includes providing a user-perceivable indication of uncertainty (box 144). In some embodiments, the user-perceivable indication may be an indicator or mark identifying the uncertainty region (which may or may not be accompanied by text indicating the uncertainty region). In some embodiments, the user-perceivable indication may highlight the uncertainty region. In some embodiments, if there is a lack of certainty, the method 128 includes changing the placement of the pair of calipers on the region of interest in the ultrasound image when at least a portion of the pair of calipers is placed in the uncertainty region between the boundaries (block 146). In some embodiments, the user can override the automatic placement of the pair of calipers.

[0085] In certain embodiments, if there is a lack of uncertainty, the method 128 includes providing a suggestion to change the parameters of one of the segmentation models (e.g., the boundary segmentation model or the region segmentation model) and re-inputting the ultrasound image into the trained neural network 58 (box 148). For example, the filter parameters of the boundary segmentation model (e.g., the gradient filter) can be changed. In certain embodiments, if there is a lack of uncertainty, the method 128 includes automatically changing the parameters of one of the segmentation models (e.g., the boundary segmentation model or the region segmentation model) and re-inputting the ultrasound image into the trained neural network 58 (box 150). If the parameters are automatically changed and the ultrasound image is re-input into the trained neural network 58, a notification may be provided or provided to the user.

[0086] Fig.11 is a schematic diagram showing the uncertainty of training a neural network to characterize the boundaries of a region of interest in an image. Figure 4 As depicted, input 152 is provided to or input into neural network 154. Input 152 includes input-output data pairs. Specifically, input 152 includes boundary line segmentation predictions (y 边界 ) and the actual boundary segmentation of the region of interest in the image (y gt边界 ). The input 152 also includes the region segmentation prediction (y 区域 ) and the true boundary of the region segmentation of the region of interest in the image (y gt(区域) ) composed of an image pair. The neural network 154 is trained using supervised learning. The neural network 154 is based on y 区域 and gt(区域) The neural network 154 is also trained based on y 边界 and gt边界 The following loss function 158 is used for training:

[0087] ∑(y 边界 * Distance map (y gt边界 )). (1)

[0088] When the boundary is accurately predicted, the loss function 158 approaches 0. When the predicted boundary is inside the true boundary, the loss function 158 is negative, and when the predicted boundary is outside the true boundary, the loss function 158 is positive. The neural network 154 is trained to determine the boundary uncertainty based on the comparison between the Dice loss 156 and the loss function 158 (Equation 1 above). The neural network 154 is trained to be Figure 4The trained neural network 58. As described above, the trained neural network 58 is trained to infer inconsistencies between boundary segmentation predictions and region segmentation predictions. The inconsistency enables uncertainty (e.g., boundary uncertainty) to be determined based on the mismatch between the boundary segmentation predictions and the region segmentation predictions by comparing the boundary segmentation predictions and the region segmentation predictions.

[0089] Figures 12 to 15 Examples of boundary segmentation predictions and region segmentation predictions and characterizations of their uncertainties (e.g., boundary uncertainty) are provided. Figure 4 The trained neural network 58) and uncertainty representation in Fig. 9 The method 104 described above is performed as described. Fig.12 is an example of a predicted boundary segmentation from a boundary overlaid on a predicted segmented region of interest in an image 160 (eg, an ultrasound image). Fig.12 As depicted, image 160 includes a region segmentation mask 162 predicted (via neural network 58) for region of interest 164 (and overlaid on region of interest 164) and a boundary segmentation mask 166 predicted (via neural network 58) for region of interest 164 (and overlaid on region segmentation mask 162). As depicted in image 160, the boundaries shown in boundary segmentation mask 166 are under-segmented, while the region segmentation (as shown in region segmentation mask 162) has been able to identify texture belonging to region of interest 164.

[0090] Fig.13 is to use Fig.12 168 is an example of a characterization of uncertainty in a prediction of . Image 168 is from the same ultrasound image as image 160. As depicted in image 168, a region 170 lacking enhancement between a boundary segmentation prediction and a region segmentation prediction is indicated by a label 172. In addition, label 172 is accompanied by text 174 to indicate the lack of enhancement. The lack of enhancement in region 170 is indicated by a graph 176 representing an intensity gradient curve in region 170. As shown in graph 176, the boundary is under-segmented due to lack of image gradient support. As depicted in image 168, a region 178 with good enhancement between a boundary segmentation prediction and a region segmentation prediction is indicated by a label 180. In addition, label 180 is accompanied by text 182 to indicate good enhancement. The good enhancement in region 178 is indicated by a graph 184 representing an intensity gradient curve in region 178.

[0091] In some embodiments, labels, accompanying text, and graphs may be provided for regions of segmentation predictions that characterize uncertainty (e.g., boundary uncertainty) for a given image. In some embodiments, labels and accompanying text may only be provided for regions of segmentation predictions that characterize uncertainty for a given image. In some embodiments, labels and accompanying text and graphs may only be provided for regions that lack reinforcement. In some embodiments, labels and accompanying text may only be provided for regions that lack reinforcement.

[0092] Fig.14 is another example of segmenting a predicted boundary from a boundary overlaid on a predicted segmented region of interest in an image 186 (e.g., an ultrasound image). Fig.14 As depicted, image 186 includes a region segmentation mask 188 predicted (via neural network 58) for region of interest 190 (and overlaid on region of interest 190) and a boundary segmentation mask 192 predicted (via neural network 58) for region of interest 190 (and overlaid on region segmentation mask 164). As depicted in image 186, the boundaries shown by boundary segmentation mask 192 are under-segmented, while the region segmentation (as shown by region segmentation mask 188) has been able to identify texture belonging to region of interest 190.

[0093] Fig.15 is to use Fig.14 193 is an example of a characterization of uncertainty in a prediction in FIG. 186. Image 193 is from the same ultrasound image as image 186. As depicted in image 193, a region 194 lacking enhancement between the boundary segmentation prediction and the region segmentation prediction is indicated by label 196. The lack of enhancement in region 194 is indicated by a graph 198 representing an intensity gradient curve in region 194. As shown in graph 198, the boundary is under-segmented due to lack of image gradient support. As depicted in image 193, a region 200 with good enhancement between the boundary segmentation prediction and the region segmentation prediction is indicated by label 202. The good enhancement in region 200 is indicated by a graph 204 representing an intensity gradient curve in region 200. In some embodiments, the image may be segmented similar to FIG. Fig.13 The text indicating lack of reinforcement and / or good reinforcement is indicated on the image 193 .

[0094] Fig.16 and Fig.17 is an example of an ultrasound image showing the correct caliper placement. Fig.16 and Fig.17 The ultrasound images in this image are from a transvaginal ultrasound scan of an ovarian tumor or lesion. Fig.16 is an ultrasound image 206 of an ovarian lesion, with a pair of calipers 208 marked or placed on the ultrasound image 206 to measure the maximum cross-section of the ovarian lesion. Fig.172 is an ultrasound image 210 of an ovarian lesion, wherein a pair of calipers 212 are marked or placed on the ultrasound image 210 to measure the maximum cross-section of the solid portion of the ovarian lesion. The pair of calipers 208, 212 in the images 206, 210 are automatically placed in the high enhancement area determined by the above-mentioned characterization technique. In some embodiments, the user may have adjusted one or more of the pair of calipers 208, 212 based on the initial caliper placement. As noted above, in some embodiments, the user can override the automatic placement of the pair of calipers 208, 212.

[0095] Technical effects of the disclosed embodiments include providing systems and methods that enable uncertainty on the boundaries of a region of interest in an ultrasound image to be characterized and indicated to a user so that a pair of calipers can be properly positioned (e.g., automatically or manually) to improve workflow and provide better patient outcomes. Furthermore, uncertainty is determined in a faster manner and with greater accuracy.

[0096] Reference is made to the technology presented and claimed herein and applied to physical and specific examples of a practical nature that clearly improves upon the present state of the art and, therefore, is not abstract, intangible, or purely theoretical. In addition, if any claim appended to the end of this specification contains one or more elements designated as "means for [performing] the function of ..." or "steps for [performing] the function of ...," it is intended that such elements be interpreted under 35 U.S.C. § 112(f). However, for any claim containing elements designated in any other manner, it is not intended that such elements be interpreted under 35 U.S.C. § 112(f).

[0097] This written description uses examples to disclose the subject matter, including the best mode, and also to enable those skilled in the art to practice the subject matter, including making and using any devices or systems and performing any included methods. The patentable scope of the subject matter is defined by the claims, and may include other examples that occur to those skilled in the art. Such other examples are intended to be within the scope of the claims if they have structural elements that do not differ from the literal language of the claims, or if they include equivalent structural elements with insignificant differences from the literal language of the claims.

Claims

1. A computer-implemented method for characterizing uncertainty at a boundary of a region of interest, the computer-implemented method comprising: Inputting the ultrasound image with the region of interest into a trained neural network (58) via a processor (44); outputting, from the trained neural network (58) via the processor (44), a first prediction (108) of the boundary of the region of interest and a second prediction (110) of a region segmentation of the region of interest; and An uncertainty in the boundary is determined, via the processor (44), based on a mismatch between the first prediction (108) and the second prediction (110).

2. The computer-implemented method of claim 1, further comprising providing, via the processor (44), a user-perceivable indication of the uncertainty when a mismatch exists between the first prediction (108) and the second prediction (110). 3 . The computer-implemented method of claim 2 , wherein providing the user-perceivable indication of the uncertainty comprises indicating any areas of uncertainty on the boundary on the ultrasound image. 4 . The computer-implemented method of claim 3 , wherein the user-perceivable indication comprises highlighting the uncertainty region on the ultrasound image.

5. The computer-implemented method of claim 1, further comprising automatically placing, via the processor (44), a pair of calipers on the region of interest in the ultrasound image in the absence of uncertainty on the boundary. The computer-implemented method of claim 1 , wherein the ultrasound image includes a pair of calipers marked on the region of interest.

7. The computer-implemented method of claim 6, further comprising providing an alert to a user via the processor (44) when at least a portion of the pair of calipers is placed in an uncertainty region on the boundary.

8. The computer-implemented method of claim 6, further comprising providing, via the processor (44), a user-perceivable indication on the ultrasound image of either end of the caliper being placed in a region of uncertainty on the boundary.

9. The computer-implemented method of claim 8, wherein the user-perceivable indication comprises causing either end of the pair of calipers in the uncertainty region to be displayed as a different color than the remainder of the pair of calipers.

10. The computer-implemented method of claim 6, further comprising changing, via the processor (44), placement of the pair of calipers over the region of interest in the ultrasound image when at least a portion of the pair of calipers is placed in an uncertainty region on the boundary.

11. A system for characterizing uncertainty at a boundary of a region of interest, the system comprising: a memory (46) encoding a processor-executable routine; and a processor (44) configured to access the memory (46) and execute the processor-executable routine, wherein the processor-executable routine, when executed by the processor (44), causes the processor (44) to: Inputting the ultrasound image with the region of interest into a trained neural network (58); outputting from the trained neural network (58) a first prediction (108) of the boundary of the region of interest and a second prediction (110) of a region segmentation of the region of interest; and An uncertainty in the boundary is determined based on a mismatch between the first prediction (108) and the second prediction (110).

12. The system of claim 11, wherein the processor executable routine, when executed by the processor (44), further causes the processor (44) to provide a user-perceivable indication of the uncertainty when a mismatch exists between the first prediction (108) and the second prediction (110).

13. The system of claim 12, wherein providing the user-perceivable indication of the uncertainty comprises indicating any areas of uncertainty on the boundary on the ultrasound image.

14. The system of claim 11, wherein the processor executable routine, when executed by the processor (44), further causes the processor (44) to automatically place a pair of calipers over the region of interest in the ultrasound image without uncertainty over the boundary.

15. The system of claim 11, wherein the ultrasound image includes a pair of calipers marked on the region of interest.

Images