System and method for analyzing uncertainty in multi-scale interpretation of medical images

Abstract

A method for analyzing uncertainty in multi-scale interpretation of a medical image includes inputting the medical image into a trained segmentation network. The method includes outputting, via a trained segmentation network, a segmentation output mask for each pixel of a medical image or a region of interest of the medical image. The method includes aggregating a segmentation output mask for all pixels of the medical image or the region of interest using a deterministic function and outputting a first classification prediction of the aggregated segmentation output mask. The method includes inputting a medical image into a trained classification network. The method includes outputting a second classification prediction of the medical image or the region of interest. The method includes determining an uncertainty between the first classification prediction and the second classification prediction via a comparison of the first classification prediction and the second classification prediction.

Description

Background Art

[0001] The subject matter disclosed herein relates to image processing, and more particularly to systems and methods for analyzing uncertainty in multi-scale interpretation of medical images.

[0002] Ultrasound devices can be used to image targets such as organs and soft tissues in the human body as well as non-human targets. For example, in addition to ultrasound imaging of humans, animals, etc., ultrasound devices can also be used for applications such as ultrasound / acoustic sensing, non-destructive evaluation (NDE), ultrasound therapy (e.g., high-intensity focused ultrasound (HIFU)), etc.

[0003] Ultrasound devices can use real-time, non-invasive high-frequency sound waves to produce a series of two-dimensional (2D) images and / or three-dimensional (3D) images. The sound waves can be emitted by a transmitting transducer, and reflections of the emitted sound waves can be received by a receiving transducer. The received sound waves can then be processed to display an image of the target. These images can be analyzed. However, sometimes the analysis of the images can be uncertain, and the reasons for this uncertainty cannot be explained or are not obvious to the user (e.g., a clinician). Summary of the invention

[0004] The following shows an overview of certain embodiments disclosed herein. It should be understood that these aspects are provided only to provide the reader with a brief overview of these specific embodiments, and these aspects are not intended to limit the scope of the present disclosure. In fact, the present disclosure may cover various aspects that may not be shown below.

[0005] In one embodiment, a computer-implemented method for analyzing uncertainty in a multiscale interpretation of a medical image is provided. The computer-implemented method includes inputting a medical image into a trained segmentation network via a processor. The computer-implemented method also includes outputting a segmentation output mask for each pixel of the medical image or a region of interest of the medical image via the trained segmentation network via the processor. The computer-implemented method also includes aggregating the segmentation output masks for all pixels of the medical image or the region of interest using a deterministic function via the processor and outputting a first classification prediction of the aggregated segmentation output masks. The computer-implemented method even includes inputting the medical image into a trained classification network via the processor. The computer-implemented method also includes outputting a second classification prediction of the medical image or the region of interest via the processor. The computer-implemented method also includes determining the uncertainty between the first classification prediction and the second classification prediction via the processor by comparing the first classification prediction with the second classification prediction.

[0006] In one embodiment, the computer-implemented method includes providing, via a processor, a user-perceivable indication of the uncertainty when uncertainty exists between the first classification prediction and the second classification prediction.

[0007] In one embodiment, providing the user-perceivable indication includes causing an explanation of the uncertainty to be displayed to the user in clinical terms.

[0008] In one embodiment, the computer-implemented method includes providing, via a processor, a recommendation to a user to potentially remedy the uncertainty.

[0009] In one embodiment, the computer-implemented method includes automatically altering, via a processor, one or more parameters related to the medical imaging and then repeating the analysis of the medical images.

[0010] In one embodiment, the computer-implemented method includes outputting, via a processor, a segmentation output mask for each pixel of the region of interest via a trained segmentation network. The computer-implemented method also includes aggregating, via a processor, the segmentation output masks for all pixels of the region of interest using a deterministic function and outputting a first classification prediction of the aggregated segmentation output masks. The computer-implemented method also includes outputting, via a processor, a second classification prediction of the region of interest.

[0011] In one embodiment, the medical image comprises an ultrasound image, and the region of interest comprises an ovarian lesion.

[0012] In one embodiment, the signatures used for both the first classification prediction and the second classification prediction include unilocular ovarian lesions, multilocular ovarian lesions, and solid ovarian lesions.

[0013] In one embodiment, the computer-implemented method includes outputting, via a processor, a segmentation output mask for each pixel of the medical image via a trained segmentation network. The computer-implemented method also includes aggregating, via the processor, the segmentation output masks for all pixels of the medical image using a deterministic function and outputting a first classification prediction of the aggregated segmentation output masks. The computer-implemented method also includes outputting, via the processor, a second classification prediction of the medical image.

[0014] In one embodiment, the medical images include ultrasound images.

[0015] In one embodiment, the labels to be used for both the first classification prediction and the second classification prediction include the presence of a lesion and the absence of a lesion.

[0016] In another embodiment, a system for analyzing uncertainty in a multi-scale interpretation of a medical image is provided. The system includes a memory that encodes a processor executable routine. The system also includes a processor that is configured to access the memory and execute the processor executable routine, wherein the routine causes the processor to perform an action when executed by the processor. The action includes inputting the medical image into a trained segmentation network. The action also includes outputting a segmentation output mask for each pixel of a region of interest of the medical image via the trained segmentation network. The action also includes a first classification prediction of aggregating the segmentation output masks of all pixels of the region of interest using a deterministic function and outputting the aggregated segmentation output masks. The action even includes inputting the medical image into a trained classification network, wherein the trained classification network is a network separated from the trained segmentation network, or the trained classification network and the trained segmentation network are both part of the same network. The action also includes outputting a second classification prediction of the region of interest. The action also includes determining the uncertainty between the first classification prediction and the second classification prediction via a comparison of the first classification prediction with the second classification prediction.

[0017] In one embodiment, the action includes causing the processor to provide a user-perceptible indication of the uncertainty when uncertainty exists between the first classification prediction and the classification prediction.

[0018] In one embodiment, providing the user-perceivable indication includes causing an explanation of the uncertainty to be displayed to the user in clinical terms.

[0019] In one embodiment, the actions further include causing the processor to provide a recommendation to the user to potentially remedy the uncertainty.

[0020] In one embodiment, the actions further include causing the processor to automatically modify one or more parameters associated with the medical image and then repeating the analysis of the medical image.

[0021] In one embodiment, the action includes outputting a segmentation output mask for each pixel of the region of interest via the trained segmentation network. The action also includes aggregating the segmentation output masks for all pixels of the region of interest using a deterministic function and outputting a first classification prediction of the aggregated segmentation output masks. The action also includes outputting a second classification prediction of the region of interest.

[0022] In one embodiment, the medical image comprises an ultrasound image, and the region of interest comprises an ovarian lesion.

[0023] In one embodiment, the signatures used for both the first classification prediction and the second classification prediction include unilocular ovarian lesions, multilocular ovarian lesions, and solid ovarian lesions.

[0024] In one embodiment, the action includes outputting, via a processor, a segmentation output mask for each pixel of the medical image via a trained segmentation network. The action also includes aggregating the segmentation output masks for all pixels of the medical image using a deterministic function and outputting a first classification prediction of the aggregated segmentation output masks. The action also includes outputting a second classification prediction of the medical image.

[0025] In one embodiment, the medical images include ultrasound images.

[0026] In one embodiment, the labels to be used for both the first classification prediction and the second classification prediction include the presence of a lesion and the absence of a lesion.

[0027] In another embodiment, a non-transitory computer-readable medium includes a processor executable code that causes the processor to perform an action when executed by the processor. The action includes inputting the medical image into a trained segmentation network. The action also includes outputting a segmentation output mask for each pixel of the medical image or a region of interest of the medical image via the trained segmentation network. The action also includes a first classification prediction of aggregating the segmentation output masks for all pixels of the medical image or region of interest using a deterministic function and outputting the aggregated segmentation output masks. The action even includes inputting the medical image into a trained classification network. The action also includes outputting a second classification prediction of the medical image or region of interest. The action also includes determining the uncertainty between the first classification prediction and the second classification prediction via a comparison of the first classification prediction with the second classification prediction.

[0028] In one embodiment, the actions further cause the processor to provide a user-perceivable indication of the uncertainty when uncertainty exists between the first classification prediction and the second classification prediction.

[0029] In one embodiment, providing the user-perceivable indication includes causing an explanation of the uncertainty to be displayed to the user in clinical terms.

[0030] In one embodiment, the actions further include causing the processor to provide a recommendation to the user to potentially remedy the uncertainty.

[0031] In one embodiment, the actions further include causing the processor to automatically modify one or more parameters associated with the medical image and then repeating the analysis of the medical image.

[0032] In one embodiment, the action includes outputting a segmentation output mask for each pixel of the region of interest via the trained segmentation network. The action also includes aggregating the segmentation output masks for all pixels of the region of interest using a deterministic function and outputting a first classification prediction of the aggregated segmentation output masks. The action also includes outputting a second classification prediction of the region of interest.

[0033] In one embodiment, the medical image comprises an ultrasound image, and the region of interest comprises an ovarian lesion.

[0034] In one embodiment, the signatures used for both the first classification prediction and the second classification prediction include unilocular ovarian lesions, multilocular ovarian lesions, and solid ovarian lesions.

[0035] In one embodiment, the action includes outputting, via a processor, a segmentation output mask for each pixel of the medical image via a trained segmentation network. The action also includes aggregating the segmentation output masks for all pixels of the medical image using a deterministic function and outputting a first classification prediction of the aggregated segmentation output masks. The action also includes outputting a second classification prediction of the medical image.

[0036] In one embodiment, the medical images include ultrasound images.

[0037] In one embodiment, the labels to be used for both the first classification prediction and the second classification prediction include the presence of a lesion and the absence of a lesion. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] These and other features, aspects, and advantages of the present subject matter will be better understood when the following detailed description is read with reference to the accompanying drawings, in which like characters refer to like parts throughout the several views, and in which:

[0039] Figure 1 is a block diagram of an ultrasound system according to aspects of the present disclosure;

[0040] Figure 2 is a schematic diagram of a model for performing multi-scale interpretation of medical images according to aspects of the present disclosure;

[0041] Figure 3 is a schematic diagram of different types of classifications of ovarian tumors or lesions according to various aspects of the present disclosure

[0042] Figure 4 is a flow chart of a method for analyzing uncertainty in multi-scale interpretation of medical images according to aspects of the present disclosure;

[0043] Figure 5 is a schematic diagram of a multi-scale analysis of an ovarian tumor or lesion (e.g., multi-chamber) in an ultrasound image without ambiguity according to aspects of the present disclosure;

[0044] Figure 6 is a schematic diagram of a multi-scale analysis of an ovarian tumor or lesion (eg, parenchyma) in an ultrasound image without ambiguity according to aspects of the present disclosure;

[0045] Figure 7 is a schematic diagram of a multi-scale analysis of an ovarian tumor or lesion (eg, single chamber) in an ultrasound image without ambiguity according to aspects of the present disclosure;

[0046] Figure 8 is a schematic diagram of a first example of multi-scale analysis of an ovarian tumor or lesion (eg, single chamber) in an ultrasound image with high uncertainty according to aspects of the present disclosure;

[0047] Fig. 9 is a schematic diagram of a second example of multi-scale analysis of an ovarian tumor or lesion (eg, single chamber) in an ultrasound image with high uncertainty in accordance with aspects of the present disclosure; and

[0048] Fig.10 is a schematic diagram of a third example of multi-scale analysis of ovarian tumors or lesions (eg, single chamber) in ultrasound images with high uncertainty in accordance with aspects of the present disclosure. DETAILED DESCRIPTION

[0049] One or more specific embodiments will be described below. In order to provide a concise description of these embodiments, not all features of an actual implementation are described in the specification. It should be understood that in the development of any such actual implementation, as in any engineering or design project, many implementation-specific decisions must be made to achieve the developer's specific goals, such as complying with system-related and business-related constraints that may vary from implementation to implementation. In addition, it should be understood that such development efforts may be complex and time-consuming, but are still routine tasks for design, fabrication, and manufacturing for ordinary technicians who benefit from this disclosure.

[0050] When introducing elements of various embodiments of the present subject matter, the articles "a," "an," "the," and "said" are intended to indicate that there are one or more elements. The terms "comprising," "including," and "having" are intended to be inclusive, and mean that there may be additional elements in addition to the listed elements. Furthermore, any numerical examples in the following discussion are intended to be non-limiting, and thus the additional numerical values, ranges, and percentages are within the scope of the disclosed embodiments.

[0051] Some general information is presented to provide a general background to various aspects of the disclosure and to facilitate understanding and explanation of certain technical concepts described herein.

[0052] As used herein, the term "image" refers broadly to both a visible image and data representing a visible image. However, many embodiments generate (or are configured to generate) at least one visible image. In addition, as used herein, the phrase "image" is used to refer to ultrasound modes, such as B-mode (2D mode), M-mode, three-dimensional (3D) mode, CF mode, PW Doppler, CW Doppler, MGD, and / or a sub-mode of B-mode and / or CF, such as shear wave elastography (SWEI), TVI, Angio, B-flow, BMI, BMI_Angio, and in some cases also includes MM, CM, TVD, wherein "image" and / or "plane" includes a single beam or multiple beams.

[0053] Furthermore, as used herein, the term processor or processing unit refers to any type of processing unit, whether single-core or multi-core: CPU, accelerated processing unit (APU), graphics board, DSP, FPGA, ASIC, or a combination thereof, that can perform the required computations required by various implementations.

[0054] The deep learning (DL) methods discussed herein may be based on artificial neural networks and thus may encompass one or more of the following: deep neural networks, fully interconnected networks, convolutional neural networks (CNNs), unfolded neural networks, perceptrons, codecs, recurrent networks, transformer networks, wavelet filter banks, u-nets, generative adversarial networks (GANs), dense neural networks (e.g., residual dense networks (RDNs)), or other neural network architectures. Neural networks may include shortcuts, activations, batch normalization layers, and / or other features. These techniques are referred to herein as deep learning techniques, but the term may also be used with particular reference to the use of deep neural networks, which are neural networks with multiple layers.

[0055] As discussed herein, deep learning techniques (which may also be referred to as deep machine learning, hierarchical learning, or deep structured learning) are a branch of machine learning techniques that employ mathematical representations of data and artificial neural networks for learning and processing such representations. For example, deep learning methods may be characterized as using one or more algorithms to extract or model highly abstract concepts of a class of data of interest. This may be accomplished using one or more processing layers, wherein each layer typically corresponds to a different level of abstraction, and thus may employ or utilize different aspects of the initial data or the output of a previous layer (i.e., a hierarchical or cascaded structure of layers) as the target of the process or algorithm of a given layer. In the context of image processing or reconstruction, this may be characterized as different layers corresponding to different feature levels or resolutions in the data. In general, the processing of one representation space to the next level of representation space may be viewed as a "stage" of the process. Each stage of the process may be performed by a separate neural network or by different parts of a larger neural network.

[0056] In the field of medical image analysis, uncertainty estimates of medical images (e.g., via artificial intelligence models) may not be accompanied by clinical insights that are available to users (e.g., radiologists or sonographers). In the disclosed systems and methods, clinical information derived from image classification is examined by image segmentation, both of which operate at different image scales to explain the uncertainty of predictions. In particular, image segmentation and classification operate on image data at different scales. Image classification typically provides a single class label to an image, while image segmentation provides a label for each pixel that makes up the image. However, in some cases, putting the segmented regions together can help derive the class label of the image. Classification is considered a top-down labeling of an image, while deriving class labels by aggregation of segmentation masks is considered a bottom-up approach to achieve the same purpose. Inconsistencies between the two can be determined as uncertainty in model performance. In medical image analysis, this can be used to remind users (e.g., clinicians) to consider this uncertainty when making any decisions.

[0057] The present disclosure provides systems and methods for analyzing uncertainty in multi-scale interpretation of medical images. In particular, the disclosed systems and methods include inputting a medical image into a trained segmentation network. The disclosed systems and methods also include outputting a segmentation output mask for each pixel of a medical image or a region of interest of the medical image via the trained segmentation network. The disclosed systems and methods also include a first classification prediction of aggregating the segmentation output masks for all pixels of the medical image or region of interest using a deterministic function and outputting the aggregated segmentation output masks. The disclosed systems and methods even include inputting the medical image into a trained classification network. In certain embodiments, the trained classification network is a network separate from the trained segmentation network. In certain embodiments, the trained classification network and the trained segmentation network are part of the same network. The disclosed systems and methods also include outputting a second classification prediction of the medical image or region of interest. The disclosed systems and methods also include determining the uncertainty between the first classification prediction and the second classification prediction via a comparison of the first classification prediction with the second classification prediction.

[0058] In some embodiments, when uncertainty exists between a first classification prediction and a second classification prediction, a user-perceptible indication of the uncertainty is provided. In some embodiments, providing the user-perceptible indication includes causing an explanation of the uncertainty to be displayed to the user in clinical terms. In some embodiments, a recommendation is provided to the user to potentially remedy the uncertainty. For example, the remedy may include changing one or more parameters associated with the medical image (e.g., changing a filter (e.g., a gradient filter) used for segmentation or changing a segmentation model utilized). In some embodiments, these one or more parameters may be automatically changed (e.g., without the user's knowledge), and the analysis of the medical image (segmentation, classification, uncertainty determination, etc.) is repeated.

[0059] In some embodiments, the medical image may be an ultrasound image. In some embodiments, the entire ultrasound image may be analyzed. In particular, the entire ultrasound image may be analyzed for the presence or absence of a lesion (e.g., a tumor) (and uncertainty determinations applied to the analysis). In some embodiments, a region of interest within an ultrasound image may be analyzed for lesion type (e.g., ovarian lesion type, such as a unilocular lesion, a multilocular lesion, a solid lesion, etc.) (and uncertainty determinations applied to the analysis).

[0060] In the disclosed embodiments, a clinician (e.g., a sonographer or radiologist) is alerted when there is uncertainty (e.g., inconsistency between classification labels and segmentation-derived labels) in model performance (e.g., for medical image analysis). In the presence of uncertainty, such uncertainty can be interpreted in clinical terms for the benefit of the end user so that the user can take control or take appropriate action (based on the interpretation of the uncertainty) to achieve better clinical outcomes. Although the disclosed techniques are discussed in the context of analyzing ultrasound images, the disclosed techniques can be used with any medical images (e.g., derived from conventional X-ray imaging, computed tomography (CT) imaging, nuclear medicine (NM) imaging, magnetic resonance imaging (MRI), or other types of imaging modalities).

[0061] With the foregoing in mind, and by providing useful context, Figure 1A high-level view of components of an ultrasound system 10 that may be employed in accordance with the methods of the present invention is depicted. The ultrasound system 10 shown includes a transducer array 14 having transducer elements adapted to be in contact with a subject or patient 18 during an imaging procedure. The transducer array 14 may be configured as a bidirectional transducer capable of transmitting ultrasound waves into the subject or patient 18 and receiving such energy from the subject or patient 18. In such a specific implementation, in a transmit mode, the transducer array elements convert electrical energy into ultrasound waves and transmit them into the patient 18. In a receive mode, the transducer array elements convert ultrasound energy (backscattered waves) received from the patient 18 into electrical signals.

[0062] Each transducer element is associated with a corresponding transducer circuit, which may be provided as one or more application specific integrated circuits (ASICs) 20 that may be present in the probe or probe handle. That is, each transducer element in the array 14 is electrically connected to a corresponding pulse generator 22, a transmit / receive switch 24, a preamplifier 26, a scan gain 34, and / or an analog-to-digital (A / D) converter 28 that is part of or disposed on the ASIC 20. In other specific implementations, the arrangement may be simplified or otherwise altered. For example, the components shown in the circuit 20 may be disposed upstream or downstream of the depicted arrangement, however, the depicted basic functions will generally still be provided for each transducer element. In the depicted example, the circuit functions mentioned are conceptualized as being implemented on a single ASIC 20 (represented by the dashed lines), however, it is understood that some or all of these functions may be disposed on the same or different integrated circuits.

[0063] Likewise Figure 1 As depicted, a variety of other imaging components are provided to enable the formation of images using the ultrasound system 10. Specifically, the depicted example of the ultrasound system 10 also includes a beamformer 32, a control panel 36, a receiver 38, and a scan converter 40, which cooperate with the transducer circuit to produce an image or series of images 42 that can be stored and / or displayed to an operator or otherwise processed as discussed herein. The transducer array 14 can be connected via a wired connection or a wireless connection (e.g., via a wireless communication unit that is part of the transducer array communicating over a wi-fi network, utilizing a wireless communication device that communicates with the transducer array via a wireless network). technology or in some other way) to transmit the ultrasound data to the beamformer.

[0064] The processing component 44 (e.g., a microprocessor or processing circuit) and memory 46 (such as may be present in the control panel 36) of the system 10 may be used to execute stored software code, instructions, or routines for processing acquired ultrasound signals to generate meaningful images and / or motion frames that may be displayed on a display 47 of the ultrasound system 10. As used herein, the term "code" or "software code" refers to any instructions or instruction sets for controlling the ultrasound system 10. The code or software code may exist in a computer executable form, such as machine code, which is a set of instructions and data that is directly executed by the processing component 44 of the control panel 36; a human understandable form, such as source code, which may be compiled for execution by the processing component 44 of the control panel 36; or an intermediate form, such as object code, which is generated by a compiler. In some embodiments, the ultrasound system 10 may include multiple controllers.

[0065] As an example, the memory 46 may store processor executable software code or instructions (e.g., firmware or software), which are tangibly stored on a non-transitory computer readable medium. In addition or alternatively, the memory 46 may store data. As an example, the memory 46 may include volatile memory (such as random access memory (RAM)) and / or non-volatile memory (such as read-only memory (ROM), flash memory, hard disk drive or any other suitable optical, magnetic or solid-state storage medium or a combination thereof). In addition, the processing unit 44 may include a plurality of microprocessors, one or more "general" microprocessors, one or more special-purpose microprocessors and / or one or more application-specific integrated circuits (ASICS) or some combination thereof. For example, the processing unit 44 may include one or more reduced instruction set (RISC) or complex instruction set (CISC) processors. The processing unit 44 may include a plurality of processors and / or the memory 46 may include a plurality of memory devices.

[0066] The processing component 44 can analyze the uncertainty in the multi-scale interpretation of the medical image. In particular, the processing component 44 can input the medical image into a trained segmentation network. The processing component 44 can input the medical image into the trained segmentation network and output a segmentation output mask for each pixel of the medical image or the region of interest of the medical image via the trained segmentation network. The processing component 44 can also utilize a deterministic function to aggregate the segmentation output masks for all pixels of the medical image or the region of interest and output a first classification prediction of the aggregated segmentation output masks. The processing component 44 can also input the medical image into a trained classification network. In some embodiments, the trained classification network is a network separate from the trained segmentation network. In some embodiments, the trained classification network and the trained segmentation network are part of the same network. The processing component 44 can also output a second classification prediction of the medical image or the region of interest. The processing component 44 can also determine the uncertainty between the first classification prediction and the second classification prediction via a comparison of the first classification prediction with the second classification prediction.

[0067] In some embodiments, the processing component 44 provides a user-perceivable indication of the uncertainty when there is uncertainty between the first classification prediction and the second classification prediction. In some embodiments, providing the user-perceivable indication includes: causing an explanation of the uncertainty to be displayed to the user in clinical terms. In some embodiments, the processing component 44 provides a recommendation to the user to potentially remedy the uncertainty. For example, the remedy may include changing one or more parameters related to the medical image (e.g., changing a filter (e.g., a gradient filter) used for segmentation or changing the segmentation model utilized). In some embodiments, these one or more parameters may be automatically changed via the processing component 44 (e.g., without the user's knowledge), and the analysis of the medical image (segmentation, classification, uncertainty determination, etc.) is repeated.

[0068] In some embodiments, the medical image may be an ultrasound image. In some embodiments, the processing component 44 analyzes the entire ultrasound image. In particular, the processing component 44 analyzes the entire ultrasound image for the presence or absence of a lesion (e.g., a tumor) (and uncertainty determinations applied to the analysis). In some embodiments, the processing component 44 analyzes a region of interest within the ultrasound image for lesion type (e.g., univentricular lesions, multiventricular lesions, parenchymal lesions, etc.) (uncertainty determinations applied to the analysis).

[0069] The ultrasound information may be processed by other or different mode-related modules (e.g., B-mode, color Doppler, power Doppler, M-mode, spectral Doppler anatomical M-mode, strain, strain rate, etc.) to form a 2D or 3D data set of image frames, etc. For example, one or more modules may generate B-mode, color Doppler, power Doppler, M-mode, anatomical M-mode, strain, strain rate, spectral Doppler image frames, and combinations thereof, etc. The image frames are stored, and timing information indicating the time when the image frame was acquired in the memory may be recorded with each image frame. These modules may include, for example, a scan conversion module for performing a scan conversion operation, thereby converting the image frame from polar coordinates to Cartesian coordinates. A video processor module may be provided that reads the image frame from the memory and displays the image frame in real time while the patient is being operated on. The video processor module may store the image frame in an image memory, read and display the image from the image memory. The ultrasound system 10 shown may include a console system or a portable system, such as a handheld or laptop system.

[0070] The ultrasound system 10 may be operable to continuously acquire ultrasound scan data at a frame rate suitable for the imaging situation under consideration. Typical frame rates may be in the range of 20 to 120, but may be lower or higher. The acquired ultrasound scan data may be displayed on the display 47 at a display rate that is the same as the frame rate, or slower or faster. An image buffer is included to store processed frames of acquired ultrasound scan data that are not scheduled for immediate display. Preferably, the image buffer has sufficient capacity to store frames of ultrasound scan data for at least several minutes. The frames of ultrasound scan data are stored in a manner that is easy to retrieve from them according to their acquisition order or time. The image buffer may be embodied as any known data storage medium.

[0071] Display 47 may be any device capable of conveying visual information to a user. For example, display 47 may include a liquid crystal display, a light emitting diode display, and / or any suitable one or more displays. Display 47 may be operable to present an ultrasound image and / or any suitable information.

[0072] The components of the ultrasound system 10 may be implemented in software, hardware, firmware, etc. The various components of the ultrasound system 10 are communicatively connected. The ultrasound components may be implemented separately and / or integrated in various forms.

[0073] Figure 2is a schematic diagram of a model 50 (e.g., a trained artificial intelligence model) for performing multi-scale interpretation of medical images. As discussed herein, the model 50 can be used for ultrasound images. In addition, the model 50 can be used for other types of medical images (e.g., from conventional X-ray imaging, computed tomography (CT) imaging, nuclear medicine (NM) imaging, magnetic resonance imaging (MRI) or other types of imaging modalities). The model 50 includes a trained segmentation network or model 52 (e.g., a deep learning-based segmentation network), which is configured to segment a medical image or a portion of a medical image. The model 50 also includes a trained classification network or model 54 (e.g., a deep learning-based classification network), which is configured to classify a medical image or a portion of a medical image. The segmentation network 52 and the classification network 54 can be trained via supervised learning or semi-supervised learning. In some embodiments, the trained classification network 54 is a network separate from the trained segmentation network 52. In some embodiments, the trained classification network 54 and the trained segmentation network 52 are part of the same network.

[0074] The model 50 includes an encoder 56 configured to: receive an input image (e.g., a medical image, such as an ultrasound image); generate a high-dimensional feature vector; and aggregate features at multiple levels. As shown, both the trained segmentation network 52 and the trained classification network 54 utilize the encoder 56. In certain embodiments, the trained segmentation network 52 and the trained classification network 54 may utilize separate encoders. The trained segmentation network 52 includes a decoder 58 (e.g., decoder 1) configured to obtain the high-dimensional feature vector from the encoder 56 and generate a segmentation mask for each pixel of the entire medical image or a region of interest in the medical image. The decoder 58 is also configured to decode the features aggregated by the encoder 56 at multiple levels. The trained classification network 54 includes a decoder 60 (e.g., decoder 2) configured to: obtain the high-dimensional feature vector from the encoder 56; reconstruct a medical image (e.g., an ultrasound image); and output a classification prediction 62 (y having a label for the reconstructed medical image (e.g., a reconstructed ultrasound image) c ).

[0075] Performing multi-scale interpretation using the model 50 includes inputting an image 64 (I), such as a medical image (e.g., an ultrasound image), into an encoder 56 and performing two separate tasks 66, 68 on the input image 64. In the first task 66, the trained segmentation network 52 performs segmentation on the image 64 to output a pixel-by-pixel prediction (e.g., a segmentation output mask) for each pixel in the image 64 or in a region of interest in the image 64. The model 50 includes a deterministic function 70 (f) that aggregates the segmentation output masks for all pixels of the image 64 or the region of interest in the image 64 and outputs a prediction 72 (y based on the aggregated segmentation output masks). s ) (e.g., a classification label). In some embodiments, the deterministic function 70 can convert (e.g., via derivation) the aggregated segmentation output mask into a classification label 72. For example, the presence or absence of a region of interest (e.g., a lesion or tumor) in the image 64 can be predicted. For example, if the number of pixels predicted to be a tumor is greater than a size threshold, then a tumor is predicted. In some embodiments, a segmentation mask can be generated and superimposed on the image 64 of the tumor. In some embodiments, the deterministic function 70 can be more complex. For example, in the case of ovarian tumor classification, multiple labels can be selected to classify a lesion or tumor, such as Figure 3 For example, for an ovarian tumor as indicated by reference numeral 76, the tumor or lesion 74 may be classified or labeled as unilocular (i.e., having only one locus). Also, for an ovarian tumor as indicated by reference numeral 78, the tumor or lesion 74 may be classified or labeled as multilocular (i.e., having multiple locules). Furthermore, for an ovarian tumor as indicated by reference numeral 80, the tumor or lesion 74 may be classified or labeled as solid (i.e., having no locules and having only a solid tumor portion). Return to Figure 2 When multiple chambers are segmented, the deterministic function 70 can deduce multiple chambers from the predicted segmentation mask. When a single chamber is segmented without a substantial portion, the deterministic function 70 can deduce a single chamber lesion. When only a substantial portion is segmented, the deterministic function 70 can deduce a substantial lesion.

[0076] In the second task 68, the trained classification network 54 directly classifies the image 64 or the region of interest in the image 64 to output a classification prediction 62 (e.g., a label). Therefore, the segmentation in the first task 66 occurs at the pixel level (i.e., in a bottom-up manner), while the classification in the second task 68 occurs at the image level (i.e., in a top-down manner). The first task 66 or segmentation serves as an explanation for the classification prediction in the second task 68. In particular, the first task 66 derives the top-down labels or classifications from the bottom-up content. The inconsistency between the classification labels and the labels derived from the segmentation gives rise to uncertainty. The uncertainty is interpreted as ambiguity between the classification labels. In medical image analysis, classification labels often have clinical terms or semantics associated with them. As described in more detail herein, this ambiguity (when present) is interpreted in clinical terms so that the end user (e.g., a clinician, such as a radiologist or sonographer) can enable them to take appropriate action.

[0077] Figure 4 is a flow chart of a method 82 for analyzing uncertainty in a multi-scale interpretation of a medical image. One or more steps of the method 82 may be performed by Figure 1 One or more steps of method 82 may be performed simultaneously and / or in different order. Figure 4 It should be noted that in addition to ultrasound images, method 82 can also be used for other medical images (e.g., from conventional X-ray imaging, computed tomography (CT) imaging, nuclear medicine (NM) imaging, magnetic resonance imaging (MRI) or other types of imaging modalities).

[0078] The method 82 includes inputting a medical image 84 (e.g., an ultrasound image) into the trained segmentation network 52 (box 86). The method 82 also includes outputting a segmentation output mask 88 for each pixel of the medical image 84 or a region of interest of the medical image 84 via the trained segmentation network 52 (box 90). The method 82 also includes aggregating the segmentation output masks 88 for all pixels of the medical image 84 or the region of interest of the medical image 84 using the deterministic function 70 and outputting a first classification prediction 92 (e.g., a segmentation-derived classification label) of the aggregated segmentation output masks (box 94).

[0079] The method 82 includes inputting the medical image 84 into the trained classification network 54 (box 96). The method 82 also includes outputting a second classification prediction 98 (eg, a classification label) for the medical image 84 or a region of the medical image 84 (box 100).

[0080] The method 82 includes determining an uncertainty between the first classification prediction 92 and the second classification prediction 98 via a comparison of the first classification prediction 92 and the second classification prediction 98 (block 102). The method 82 also includes determining whether there is a lack of certainty based on the determined uncertainty (block 104). If there is no lack of certainty, the method 82 ends (block 106).

[0081] If there is a lack of certainty, the method 82 includes providing a user-perceivable indication of the uncertainty when there is uncertainty between the first classification prediction 92 and the second classification prediction 98 (box 108). In some embodiments, providing a user-perceivable indication of the uncertainty (e.g., a reminder) includes causing an explanation of the uncertainty to be displayed to the user in clinical terms (e.g., on a console of the ultrasound system 10 or a separate display or monitor). In some embodiments, providing a user-perceivable indication of the uncertainty includes showing on the medical image 84 where the uncertainty occurs. For example, a segmentation mask may be superimposed on the medical image 84 and marked where the segmentation uncertainty occurs. In some embodiments, if there is a lack of certainty, the method 82 includes providing a recommendation to the user (e.g., on a console of the ultrasound system 10 or a separate display or monitor) to potentially remedy the uncertainty (box 110). For example, the remedy may include changing one or more parameters associated with the medical image (e.g., changing a filter (e.g., a gradient filter) used for segmentation or changing the segmentation model utilized). In some embodiments, if there is a lack of uncertainty, the method 82 includes automatically changing one or more parameters associated with the medical image or the multiscale analysis of the medical image (e.g., without letting the user know) and repeating the multiscale analysis (block 112). For example, the filter used for segmentation (e.g., a gradient filter) may be changed or the segmentation model utilized may be changed.

[0082] Repeating the multi-scale analysis (block 112) includes inputting the medical image 84 (e.g., ultrasound image) into the trained segmentation network 52 (block 86). Repeating the multi-scale analysis (block 112) also includes outputting a segmentation output mask 88 for each pixel of the medical image 84 or a region of interest of the medical image 84 via the trained segmentation network 52 (block 90). Repeating the multi-scale analysis (block 112) also includes aggregating the segmentation output masks 88 for all pixels of the medical image 84 or a region of interest of the medical image 84 using the deterministic function 70 and outputting a first classification prediction 92 (e.g., a segmentation-derived classification label) of the aggregated segmentation output masks (block 94).

[0083] Repeating the multiscale analysis (block 112) includes inputting the medical image 84 into the trained classification network 54 (block 96). Repeating the multiscale analysis (block 112) also includes outputting a second classification prediction 98 (e.g., a classification label) for the medical image 84 or a region of the medical image 84 (block 100).

[0084] Repeating the multi-scale analysis (block 112) includes determining an uncertainty between the first classification prediction 92 and the second classification prediction 98 via a comparison of the first classification prediction 92 and the second classification prediction 98 (block 102). Repeating the multi-scale analysis (block 112) also includes determining whether there is a lack of certainty based on the determined uncertainty (block 104). If there is no lack of certainty, the method 82 ends (block 106).

[0085] If there is a lack of certainty, repeating the multiscale analysis (box 112) includes providing a user-perceivable indication of the uncertainty when there is uncertainty between the first classification prediction 92 and the second classification prediction 98 (box 108). In some embodiments, providing a user-perceivable indication of the uncertainty (e.g., a reminder) includes causing an explanation of the uncertainty to be displayed to the user in clinical terms (e.g., on a console of the ultrasound system 10 or a separate display or monitor). In some embodiments, providing a user-perceivable indication of the uncertainty includes showing on the medical image 84 where the uncertainty occurs. For example, a segmentation mask may be superimposed on the medical image 84 and marked where the segmentation uncertainty occurs. In some embodiments, if there is a lack of certainty, repeating the multiscale analysis (box 112) includes providing a recommendation to the user (e.g., on a console of the ultrasound system 10 or a separate display or monitor) to potentially remedy the uncertainty (box 110). For example, the remedy may include changing one or more parameters associated with the medical image (e.g., changing a filter (e.g., a gradient filter) used for segmentation or changing the segmentation model utilized). In some embodiments, if there is a lack of uncertainty, the method 82 includes automatically changing one or more parameters associated with the medical image or the multiscale analysis of the medical image (e.g., without letting the user know) and repeating the multiscale analysis (block 112). For example, the filter used for segmentation (e.g., a gradient filter) may be changed or the segmentation model utilized may be changed.

[0086] In some embodiments, the medical image may be an ultrasound image. In some embodiments, the entire ultrasound image may be analyzed. In particular, the entire ultrasound image may be analyzed for the presence or absence of a lesion (e.g., a tumor) (and uncertainty determinations applied to the analysis). In some embodiments, regions of interest within the ultrasound image may be analyzed for lesion type (e.g., univentricular lesions, multiventricular lesions, solid lesions, etc.) (and uncertainty determinations applied to the analysis).

[0087] Figure 5 1 is a schematic diagram of a multi-scale analysis of an ovarian tumor or lesion (eg, multi-chamber) in an ultrasound image 114 without ambiguity. The ultrasound image 114 is acquired via a transvaginal ultrasound scan. Figure 4 The method 82 in the embodiment of the present invention is used to perform uncertainty analysis of a multi-scale interpretation of an ultrasound image 114. The lesion in the ultrasound image 114 has multiple loci segmented as indicated by various segmentation masks 116 superimposed on the ultrasound image 114. The segmentation predictions (segmentation-derived classification labels) are multi-locular. The classification predictions (classification labels) are also multi-locular. Both the segmentation predictions and the classification predictions are consistent with each other and correspond to the multi-locular ground truth (GT). The uncertainty analysis found no uncertainty in the multi-scale interpretation of the ultrasound image 114.

[0088] Figure 6 1 is a schematic diagram of a multi-scale analysis of an ovarian tumor or lesion (eg, parenchyma) in an ultrasound image 118 without ambiguity. The ultrasound image 118 is acquired via a transvaginal ultrasound scan. Figure 4 The method 82 in is used to perform uncertainty analysis of the multi-scale interpretation of the ultrasound image 118. The lesion in the ultrasound image 118 does not have segmented locules. The segmentation prediction (segmentation-derived classification label) is substantial. The classification prediction (classification label) is also substantial. Both the segmentation prediction and the classification prediction are consistent with each other and correspond to a substantial ground truth (GT). The uncertainty analysis finds no uncertainty in the multi-scale interpretation of the ultrasound image 118.

[0089] Figure 7 Schematic diagram of multi-scale analysis of an ovarian tumor or lesion (eg, single chamber) in an ultrasound image 120 without ambiguity. The ultrasound image 120 is acquired via transvaginal ultrasound scanning. Figure 4 The method 82 in is used to perform uncertainty analysis of the multi-scale interpretation of the ultrasound image 120. The lesion in the ultrasound image 120 has a single locus segmented as indicated by the segmentation mask 122 superimposed on the ultrasound image 120. The segmentation prediction (segmentation-derived classification label) is single-locus. The classification prediction (classification label) is also single-locus. Both the segmentation prediction and the classification prediction are consistent with each other and correspond to the ground truth (GT) of the single locus. The uncertainty analysis found no uncertainty in the multi-scale interpretation of the ultrasound image 114.

[0090] Figure 8 FIG. 1 is a schematic diagram of a first example of a multi-scale analysis of an ovarian tumor or lesion (eg, a single chamber) in an ultrasound image 124 with high uncertainty. The ultrasound image 124 is acquired via a transvaginal ultrasound scan. Figure 4The method 82 in the embodiment of the present invention is used to perform uncertainty analysis of a multi-scale interpretation of an ultrasound image 124. The lesion in the ultrasound image 124 has a single chamber segmented as indicated by a segmentation mask 126 superimposed on the ultrasound image 124. The segmentation prediction (segmentation-derived classification label) is single-chamber. The classification prediction (classification label) is substantial. The segmentation prediction and the classification prediction are inconsistent with each other. In addition, the classification prediction is inconsistent with the ground truth (GT) of the single chamber. Due to the inconsistency, the uncertainty analysis finds a high degree of uncertainty in the multi-scale interpretation of the ultrasound image 124. Since a high degree of uncertainty is determined, an explanation of the uncertainty is provided to the user, as indicated by reference numeral 128. The explanation 128 is performed in clinical terms so that the user can understand the cause of the uncertainty and take appropriate action.

[0091] Fig. 9 FIG. 1 is a schematic diagram of a second example of multi-scale analysis of an ovarian tumor or lesion (eg, a single chamber) in an ultrasound image 130 with high uncertainty. The ultrasound image 130 is acquired via a transvaginal ultrasound scan. Figure 4 The method 82 in is used to perform uncertainty analysis of the multi-scale interpretation of the ultrasound image 130. The lesion in the ultrasound image 130 has multiple chambers segmented as indicated by various segmentation masks 132 superimposed on the ultrasound image 130. The segmentation prediction (segmentation-derived classification label) is multi-chamber. The classification prediction (classification label) is single-chamber. The segmentation prediction and the classification prediction are inconsistent with each other. In addition, the classification prediction is inconsistent with the ground truth (GT) of the single chamber. Due to the inconsistency, the uncertainty analysis finds a high degree of uncertainty in the multi-scale interpretation of the ultrasound image 130. Since a high degree of uncertainty is determined, an explanation of the uncertainty is provided to the user, as indicated by the reference numeral 134. The explanation 134 is performed in clinical terms so that the user can understand the cause of the uncertainty and take appropriate action.

[0092] Fig.10 FIG. 1 is a schematic diagram of a third example of multi-scale analysis of an ovarian tumor or lesion (eg, single chamber) in an ultrasound image 136 with high uncertainty. The ultrasound image 136 is acquired via a transvaginal ultrasound scan. Figure 4The method 82 in is used to perform uncertainty analysis of a multi-scale interpretation of an ultrasound image 136. A lesion in the ultrasound image 136 has multiple chambers segmented as indicated by various segmentation masks 138 superimposed on the ultrasound image 136. The segmentation prediction (segmentation-derived classification label) is multi-chamber. The classification prediction (classification label) is single-chamber. The segmentation prediction and the classification prediction are inconsistent with each other. In addition, the classification prediction is inconsistent with the ground truth (GT) of the single chamber. Due to the inconsistency, the uncertainty analysis finds a high degree of uncertainty in the multi-scale interpretation of the ultrasound image 136. Since a high degree of uncertainty is determined, an explanation of the uncertainty is provided to the user, as indicated by reference numeral 140. The explanation 140 is performed in clinical terms so that the user can understand the cause of the uncertainty and take appropriate action.

[0093] Technical effects of the disclosed embodiments include providing systems and methods for analyzing uncertainty in multi-scale interpretations of medical images. Technical effects of the disclosed embodiments include alerting a clinician (e.g., a sonographer or radiologist) when there is uncertainty in model performance (e.g., for medical image analysis) (e.g., inconsistency between classification labels and segmentation-derived labels). In the presence of uncertainty, such uncertainty can be interpreted in clinical terms for the benefit of the end user, so that the user can take control or take appropriate action (based on the interpretation of the uncertainty) to achieve better clinical outcomes. Technical effects of the disclosed embodiments include providing a one-pass uncertainty calculation.

[0094] Reference is made to the technology presented and claimed herein and applied to physical and specific examples of a practical nature that clearly improves upon the present state of the art and, therefore, is not abstract, intangible, or purely theoretical. In addition, if any claim appended to the end of this specification contains one or more elements designated as "means for [performing] the function of ..." or "steps for [performing] the function of ...," it is intended that such elements be interpreted under 35 U.S.C. § 112(f). However, for any claim containing elements designated in any other manner, it is not intended that such elements be interpreted under 35 U.S.C. § 112(f).

[0095] This written description uses examples to disclose the subject matter, including the best mode, and also to enable those skilled in the art to practice the subject matter, including making and using any devices or systems and performing any included methods. The patentable scope of the subject matter is defined by the claims, and may include other examples that occur to those skilled in the art. Such other examples are intended to be within the scope of the claims if they have structural elements that do not differ from the literal language of the claims, or if they include equivalent structural elements with insignificant differences from the literal language of the claims.

Claims

1. A computer-implemented method for analyzing uncertainty in a multi-scale interpretation of a medical image, the computer-implemented method comprising: inputting the medical image into a trained segmentation network; outputting a segmentation output mask for each pixel of the medical image or a region of interest of the medical image via the trained segmentation network; aggregating segmentation output masks for all pixels of the medical image or the region of interest using a deterministic function and outputting a first classification prediction of the aggregated segmentation output masks; inputting the medical image into a trained classification network; outputting a second classification prediction of the medical image or the region of interest; as well as An uncertainty between the first classification prediction and the second classification prediction is determined via a comparison of the first classification prediction and the second classification prediction.

2. The computer-implemented method of claim 1 , further comprising providing a user-perceivable indication of the uncertainty when uncertainty exists between the first classification prediction and the second classification prediction.

3. The computer-implemented method of claim 2 , wherein providing the user-perceivable indication comprises: An explanation of the uncertainty is caused to be displayed to a user in clinical terms. 4 . The computer-implemented method of claim 2 , further comprising providing a recommendation to a user to potentially remedy the uncertainty. 5 . The computer-implemented method of claim 1 , further comprising automatically altering one or more parameters associated with the medical image and then repeating the analysis of the medical image.

6. The computer-implemented method of claim 1 , comprising: outputting the segmentation output mask for each pixel of the region of interest via the trained segmentation network; aggregating segmentation output masks for all pixels of the region of interest using the deterministic function and outputting the first classification prediction of the aggregated segmentation output masks; as well as The second classification prediction of the region of interest is output.

7. The computer-implemented method of claim 6, wherein the medical image comprises an ultrasound image and the region of interest comprises an ovarian lesion.

8. The computer-implemented method of claim 7, wherein labels to be used for both the first classification prediction and the second classification prediction include unilocular ovarian lesions, multilocular ovarian lesions, and solid ovarian lesions.

9. The computer-implemented method of claim 1 , comprising: outputting the segmentation output mask for each pixel of the medical image via the trained segmentation network; aggregating segmentation output masks for all pixels of the medical image using the deterministic function and outputting the first classification prediction of the aggregated segmentation output masks; as well as The second classification prediction of the medical image is output.

10. The computer-implemented method of claim 9, wherein the medical image comprises an ultrasound image.

11. The computer-implemented method of claim 10, wherein labels to be used for both the first classification prediction and the second classification prediction include the presence of a lesion and the absence of the lesion.

12. A system for analyzing uncertainty in a multi-scale interpretation of a medical image, the system comprising: a memory encoding a processor-executable routine; and A processor configured to access the memory and execute the processor-executable routine, wherein the processor-executable routine, when executed by the processor, causes the processor to perform any of the methods according to claims 1 to 11.

13. A non-transitory computer readable medium comprising processor executable code which, when executed by a processor, causes the processor to perform any of the methods of claims 1 to 11.

Images