Automated intraoperative tumor identification using machine learning

Abstract

To determine a tumor position during a brain shift.SOLUTION: An electronic processor is programed so as to control a US scanner and a US probe for acquiring a series of preoperative images of a tumor in a region of interest of a patient and peripheral blood vessels; provide a graphical user interface in which the acquired preoperative images are labelled with contours of the tumor in the ROI and the peripheral blood vessels; adjust a neural network trained for the patient using the preoperative images labelled so as to generate a patient adjustment training neural network; execute live imaging by controlling the US scanner and the US probe so as to acquire live images of the tumor in the ROI of the patient and the peripheral blood vessels; input the live images to the patient adjustment training NN for outputting live contours of the tumor and the peripheral blood vessels; and control a display device so as to display the live images in which the live contours are overlapped.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The following relates generally to image-guided interventional procedures, image-guided neurosurgery, image-guided brain surgery, ultrasound imaging techniques, ultrasound probe tracking techniques, ultrasound image display, and related techniques. [Background technology]

[0002] Preoperative magnetic resonance imaging (MRI) in neurosurgery is the usual standard used to identify pathology and plan surgery prior to surgical resection of the tumor. One or more tumors and the surrounding vasculature are segmented on the preoperative MR images and used for preoperative planning. Neuronavigation is used during neurosurgery to allow the surgeon to navigate the pathology through the craniotomy. High-precision tracking techniques, such as optical tracking, are used to track the surgical tools for neuronavigation. The tracking information is registered to the patient and the preoperative images so that the surgeon can navigate within the skull and perform the surgery as planned based on the preoperative imaging. However, several factors deform the brain during surgery, reducing the usefulness of preoperative imaging for neuronavigation. This phenomenon is well known in neurosurgery, where brain shift usually begins with the craniotomy and becomes noticeable after the subsequent removal of the dura, a thick fibrous membrane that holds the brain tissue in place. Other factors that can cause brain shift include swelling, gravity, tumor resection, and cerebrospinal fluid drainage. Generally speaking, brain shifts always worsen throughout treatment.

[0003] Intraoperative or "live" imaging can reveal brain shifts and guide the surgeon to modify and update the surgical plan. Among the available modalities, MRI offers excellent image quality and resolution. However, it has a negative impact on the surgical flow (ferrous metals cannot be found in the room, and other metals cannot be placed near the imaging area, and therefore extensive patient preparation is necessary for each intraoperative MRI image acquisition), is time-consuming (MR sequences take 45 minutes), and is unavailable except in high-end operating rooms. On the other hand, intraoperative ultrasound (US) is less problematic than MRI (no metal restrictions and no extensive patient preparation), can be performed in minutes with real-time image visualization, and is widely available. However, US image quality and resolution are poor compared to MRI, US registration to MRI is not a simple task, and not all tumors are visible in US imaging.

[0004] To address these difficulties in using intraoperative ultrasound, an initial (preoperative) ultrasound image is acquired after the craniotomy but before removing the dura. With the dura still in place, brain shift is minimal and the preoperative US image is well registered with the preoperative MRI. Such image registration provides the surgeon with a visual understanding of how the tumor will look under ultrasound when viewed together with the overlaid segmentation of the tumor. After removing the dura, rigid registration of the ultrasound to the preoperative MR image is usually not accurate enough for intraoperative navigation. In this regard, the segmentation can be presented overlaid on the intraoperative ultrasound image to aid in the interpretation of the intraoperative (i.e., live) image, although registration of the segmentation to the live image may not be very accurate due to suboptimal registration.

[0005] After brain shift, preoperative MRI tumor segmentation and overlay on US images becomes invalid because the US images show the deformed tissue, but the segmentation is for non-deformed tissue. In this regard, the surgeon's experience in US and MRI image interpretation, as well as understanding how a particular tumor is located within the anatomy and how it deforms, is a differentiating factor and helps the surgeon determine what is and what is not residual tumor based on intraoperative US images of deformed brain tissue.

[0006] Various non-rigid registration techniques have been used by researchers to address the issue of usability of intraoperative US for brain surgery in the presence of brain shift, but with limited success. In a first exemplary technique, preoperative MR images are deformed based on intraoperative ultrasound images, but this approach can lead to significant inaccuracies because in most cases two-dimensional (2D) ultrasound images corresponding to a plane are used and do not capture deformations in out-of-plane directions. In a second exemplary technique, optical surface scans of deformed brain tissue are performed to calculate brain volumetric deformations by finite element modeling. This approach is limited to visualizing lesions close to the brain surface and does not provide accurate information about deeper lesions. Summary of the Invention [Problem to be solved by the invention]

[0007] The following discloses new and improved systems and methods for overcoming these problems. [Means for solving the problem]

[0008] In one disclosed embodiment, a US system includes a US scanner and a US probe operatively connected to the US scanner. At least one electronic processor is programmed to: control the US scanner and the US probe to acquire a series of preoperative images of a tumor and surrounding blood vessels in a region of interest (ROI) of a patient; provide a graphical user interface (GUI) in which the acquired preoperative images are labeled with contours of the tumor and surrounding blood vessels in the ROI; train a trained neural network for the patient using the labeled preoperative images to generate a patient-adjusted trained neural network; perform live imaging by controlling the US scanner and the US probe to acquire live images of the tumor and surrounding blood vessels in the patient's ROI; input the live images into the patient-adjusted trained NN to output live contours of the tumor and surrounding blood vessels; and control a display device to display the live images superimposed with the live contours.

[0009] In another disclosed aspect, a non-transitory storage medium stores instructions readable and executable by a processor in operative communication with an US system and a display device to perform a live brain imaging method, the method including controlling the US system to acquire a two-dimensional (2D) live brain image, applying a neural network to the 2D live brain image to generate a live contour of a tumor and surrounding blood vessels imaged in the 2D live brain image, and controlling the display device to present the 2D live brain image overlaid with the live contour.

[0010] In another disclosed embodiment, a US system includes a US scanner and a US probe operatively connected to the US scanner, wherein at least one electronic processor is programmed to control the US scanner and the US probe to acquire a 2D live image of a ROI, apply a neural network to the 2D live brain image to generate a live contour of the tumor and surrounding blood vessels imaged in the 2D live image, and control a display device to present the 2D live image superimposed with the live contour.

[0011] One advantage is in determining tumor location during brain shift.

[0012] Another advantage is the use of machine learning to train a neural network to learn the correlation between pre-operative tumor contours and intra-operative tumor contours during brain shifts.

[0013] Another advantage is in predicting the new location of the tumor due to brain shifts during surgery.

[0014] Another advantage is that it provides an overlay of MR and US images with tumor contours altered by brain shift.

[0015] Another advantage is providing one or more of the above advantages using only ultrasound imaging without the need for MR imaging after acquisition of the initial preoperative MR image and without the need to update the tumor or vessel contouring during the neurosurgery procedure.

[0016] Another advantage is that additional information in the form of intraoperative MR images, if available, can be used to improve contour estimation in intraoperative US images, providing one or more of the aforementioned advantages.

[0017] Another advantage is that additional information in the form of manual contouring of intraoperative MR images, if available, can be used to improve contour estimation in subsequent intraoperative US images, providing one or more of the aforementioned advantages.

[0018] A given embodiment may provide none, one, two, more, or all of the aforementioned advantages and / or may provide other advantages, as will be apparent to one of ordinary skill in the art upon reading and understanding this disclosure.

[0019] The disclosure may take form in various components and arrangements of components, and in various steps and arrangements of steps. The drawings are only for purposes of illustrating preferred embodiments and are not to be construed as limiting the disclosure. [Brief description of the drawings]

[0020] [Figure 1] 1 illustrates generally an exemplary ultrasound (US) image guidance device for use in neurosurgery, according to one embodiment. [Diagram 2] 2 illustrates an exemplary flow chart operation of the apparatus of FIG. 1. [Diagram 3] 2 shows a schematic example of a neural network of the device of FIG. 1; [Figure 4] 2 shows an image with contours produced by the device of FIG. 1; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] In image-guided brain surgery, ultrasound (US) is commonly used to provide intraoperative imaging during the surgical procedure. In a typical workflow, preoperative MRI images are acquired, tumors and important vessels are segmented, and surgery is planned. On the day of surgery, the patient is prepared and a portion of the skull is removed to provide access to the outermost dura that encases the brain (i.e., craniotomy). At this point, preoperative US images are acquired and displayed. The US images are tracked using a camera or other positioning sensor, and tumor / vessel contours manually drawn on the preoperative MRI images are superimposed on the indicia of the preoperative US images, possibly after rigid spatial registration of the preoperative MRI and US images. Alternatively, the US imaging probe can be a 2D matrix array probe that creates a three-dimensional (3D) image of a volume. In this case, the MRI-based segmentation can be superimposed as contours on 2D slices of the 3D volumetric image, or as a volume (possibly with different color coding and transparency) on the 3D US volume itself. The 2D matrix array probe can be tracked to obtain smooth images by spatial averaging. The visual appearance of a tumor may be quite different in US modality and MRI, and the surgeon examines the preoperative US images to identify the target tumor for resection. A portion of the dura is then removed to provide access to the brain, and surgical tumor removal begins. Live US images acquired intermittently throughout the procedure (i.e., intraoperative) are visualized as a reference, with tumor / vessel contours from the MRI images superimposed on the live US images.

[0022] However, problems associated with brain shift arise, where the brain can deform during surgery, for example, when the dura is resected and when tumor tissue is excised to reduce the size of the tumor during surgery. Brain shift can result in the superimposed contours from the MRI no longer accurately reflecting the location of the corresponding anatomical structures shown in the live US imaging. Traditionally, this is addressed by the surgeon visually (and based on other feedback, e.g., haptic feedback) detecting the shift in the live US image and taking the shift into account in mind. If an intraoperative MRI scanner is available in the operating room, updated MRI images may be acquired and segmented from time to time to generate updated MRI contours. However, MRI generally cannot provide real-time live imaging. It is noted that surgical removal of a brain tumor can take several hours and sometimes requires precision on the order of a few microns. Brain shift can occur gradually during surgery; for example, as more tumor tissue is removed, this changes the pressure of the surrounding brain tissue, causing progressively larger brain shifts, such that occasional intraoperative MRI is not sufficient for guidance throughout the procedure.

[0023] In the approach disclosed herein, a neural network is described that is trained to generate updated contours of tumors and major vessels directly from the intraoperative US images. This allows tracking of brain shifts as they may change gradually throughout the surgical procedure. In some embodiments, the contours from the MRI are kept superimposed and these newly generated updated contours from the neural network are also superimposed, for example using different color coding, to provide maximum information to the surgeon.

[0024] The training of the neural network must be highly accurate and yet responsive to patient-specific variability. To this end, the disclosed approach employs patient-specific fine-tuning, including initial training of the neural network using historical patient data, and update training of the neural network using initial preoperative US images labeled with contours from the preoperative MRI or corresponding contours adjusted or newly drawn by the neurosurgeon in the preoperative US images.

[0025] During surgery, the input data to the neural network is a sequence of 2D ultrasound images (i.e., intraoperative or "live" US images used to provide visual guidance to the surgeon during surgery), or a single 3D ultrasound image or sequence of 3D ultrasound images that can be sliced ​​and processed by the neural network, or alternatively, the network can be trained to work with a 3D US volumetric image as input. The output of the neural network is one or more tumor and / or vascular contours.

[0026] For the initial training, suitable training data can be collected from previous similar brain surgeries using preoperative US images labeled with corresponding MRI-derived tumor and vessel contours that serve as ground truth contours. Since the preoperative US images are acquired after craniotomy but before incising the dura and before removal of any tumor tissue, brain shift should be negligible in the training data. Conventional backpropagation training may be used to train the neural network to output contours that best match ground truth contours manually drawn in the preoperative MRI images.

[0027] The fine-tuning uses as training data the preoperative images of the current patient labeled with the MRI contours, again serving as ground truth. (More precisely, this training data consists of a set of preoperative 2D US image slices that can be combined to provide a 3D preoperative US image, labeled with the MRI contours and / or contours adjusted or newly drawn in the preoperative US images by the neurosurgeon, or 3D US volumetric images obtained using a 2D matrix array probe for 3D imaging.) The same back-propagation based training used in the initial training of the neural network may be used in the update training to provide the fine-tuning, and the resulting fine-tuned neural network is specific to the current patient.

[0028] If an MRI scanner is available in the operating room, it is contemplated to perform additional fine-tuning steps at various times during the surgical procedure. For example, after several hours and after removal of a significant portion of the brain tumor, the fine-tuned neural network may be out of date (i.e., lacking the predictive accuracy required for surgery) and a new MRI image may be acquired and manually contoured along with the corresponding current US image. This then serves as updated training data for further fine-tuning (i.e., updated training) of the neural network.

[0029] In some embodiments, the neural network may be a recurrent neural network (RNN) where the contour output for a current US image depends on the previous contours output for the previous US image in the sequence. This reflects the expectation that (1) the tumor will decrease in size over the course of surgery due to tumor tissue removal, and (2) this shrinkage will be gradual. However, to train the RNN, it is necessary to have labeled sequence training data over the course of surgery, which can be provided if an MRI scanner is present in the operating room to provide updated MRI (i.e., ground truth) contour labels from time to time.

[0030] Yet another variation contemplates the use of reinforcement learning (RI), where occasionally US images of a live sequence are manually contoured and the fidelity of this manual US contour to the contour output by the trained and fine-tuned neural network is used as further feedback for the update training (i.e., RI) of the neural network. In some examples, the RI can be performed using MRI-driven contours rather than manual contours. Any errors made by the neural network during prediction can be taken into account by the RI, and thus the neural network can be more conservative for that particular type of error.

[0031] In general, the neural network is configured to receive as input a sequence of 2D US images (which may or may not collectively constitute a 3D image) and output one or more contours in the spatial reference of the US image plane. Any neural network architecture may be used in the embodiments disclosed herein.

[0032] In an exemplary embodiment, the disclosed approach is used in brain surgery procedures using preoperative MRI imaging and intraoperative US imaging. Brain imaging for neurosurgical guidance requires high precision compared to many other interventional procedures where organ shifts or deformations can be fully accommodated by mental correction or elastic registration techniques. More generally, however, the disclosed approach may find application in other types of interventional procedures, such as prostate, breast, or liver biopsy procedures, where organ deformations may occur and be beneficially tracked over the course of the surgical procedure.

[0033] With reference to FIG. 1, an exemplary interventional imaging device suitable for carrying out the preceding description is shown. The ultrasound (US) imaging device 10 may be, for example, an EPIQ® ultrasound imaging system available from Koninklijke Philips N.V., Amsterdam, The Netherlands, a UroNav® system for US / MRI fusion guided prostate biopsy available from Koninklijke Philips N.V., Amsterdam, The Netherlands, a PercuNav® system for general fusion of US with conventional 3D imaging (CT, MR, cone beam CT, etc.) (available from Koninklijke Philips N.V., Amsterdam), or other commercially available or custom ultrasound imaging systems. The ultrasound imaging device 10 includes a US probe 12 operably connected to a US scanner 14 to perform ultrasound imaging. The exemplary ultrasound probe 12 is connected to the ultrasound imaging system 10 via cable wiring 15 by wireless connection. The US probe 12 includes a sensor array 16 for acquiring two-dimensional (2D) image frames in an ultrasound plane. The surgeon or other operator can adjust the position and orientation (i.e., "pose") of the image frames by freehand or supported movements of the ultrasound probe 12. Such freehand movements may involve a translational sweep of the US probe 12 (and thus of the ultrasound plane) and / or may include rotating the US probe 12 about an axis. The US scanner 14 and the US probe 12 are configured to acquire a "live" succession of two-dimensional or three-dimensional (2D / 3D) image frames of a portion of the patient, each 2D image frame corresponding to a current pose of the ultrasound plane, or in case of 3D imaging, multiple planes can be considered.

[0034] The US scanner 14 also includes and / or is operatively connected to at least one electronic processor 20 (e.g., microprocessor, microcontroller, etc.), at least one user input device (e.g., mouse, keyboard, and / or trackball, etc.) 22, and a display device 24 for displaying ultrasound images and / or US scanner settings and / or image parameters, etc. The display device 24 is configured to display a graphical user interface (GUI) 26 that includes one or more fields for receiving user input from the user input device 22.

[0035] The at least one electronic processor 20 is operatively connected to a non-transitory storage medium 28 storing instructions readable and executable by the at least one electronic processor 20 to perform the disclosed operations, including, for example, operating the US scanner 14 to acquire pre-operative US images, performing live US imaging, and performing the imaging method or process 100 to acquire images of a patient for neurosurgical guidance. The non-transitory storage medium may include, for example, a hard disk drive or other magnetic storage medium, a solid state drive (SSD), a flash drive, an electronically erasable read-only memory (EEROM) or other electronic memory, an optical disk or other optical storage device, or various combinations thereof. In some embodiments, the non-transitory storage medium storing the instructions is located within the US scanner 14, for example in the form of an internal hard drive, SSD, flash memory, and / or the like.

[0036] In some embodiments, the at least one electronic processor 20 is programmed to apply the trained neural network (NN) 30 to determine the tumor contour. For example, the electronic processor can be a graphical processing unit (GPU) for performing in real time during the prediction of the tumor contour. The at least one electronic processor may be programmed to perform an initial training of the NN 30 using US images of past neurosurgical patients and manually drawn contours serving as ground truth. Alternatively, the initial training may be performed offline, for example by a vendor of the US scanner 14 or by a vendor of a neurosurgical image guidance package or add-on for the US scanner 14, and the trained NN 30 is provided by the vendor and stored in a non-transitory storage medium 28. In either case, patient-specific tuning of the trained NN 30 is preferably performed by the at least one electronic processor 20, since this tuning must be performed for each patient and cannot be provided as a pre-trained component. Since the training and patient-specific tuning of the NN 30 is computationally intensive, it is contemplated that the at least one electronic processor 20 has a remote server computer, cloud computing resources, or the like, operatively connected to the US scanner 14 by a wired or wireless connection (e.g., a wired or wireless connection to a hospital network). If the at least one electronic processor 20 has a remote server computer, the vendor may have access to fine-tuning data at different sites to improve the training and fine-tuning of the NN 30. The NN 30 may be updated frequently, for example, as software updates. In this configuration, the electronic processor of the US scanner 14 operates the scanner to acquire US images of the patient and provide a GUI for the user to outline selected US images of the patient, and then this patient-specific training data is uploaded to the remote server, which performs the patient-specific tuning by updating and training the NN 30 using the provided patient-specific training data.In some illustrative examples, the NN 30 can be a recurrent NN (RNN). In some embodiments, an MR device 32 can be located in a medical environment with the US probe 12 and the US scanner 14 to provide additional updated training data in the form of contours drawn in the updated MR images. For example, the NN 30 can be fine-tuned by retraining the NN several times during a procedure with MRI images. A user can perform the fine-tuning of the NN 30 remotely during the procedure.

[0037] With reference to FIG. 2, an exemplary embodiment of an imaging method 100 is shown diagrammatically as a flow chart. To begin the process, the US probe 12 is placed on or near a portion of the patient to be scanned (e.g., the brain, or more specifically, a region of the brain that contains one or more tumors to be resected). At 102, at least one electronic processor 20 is programmed to control the US scanner 14 and the US probe 12 to acquire a series of preoperative images (e.g., 2D, 3D, or 4D images) of a tumor and surrounding blood vessels within a region of interest (ROI) of the patient about to undergo neurosurgery. In some examples, the ROI can be the brain of the patient. Typically, operation 102 is performed after the craniotomy but before cutting the dura, so as to minimize the amount of brain shift in the preoperative US images. In an alternative embodiment, the preoperative US imaging operation 102 can be performed before the craniotomy.

[0038] At 104, at least one electronic processor 20 is programmed to provide a GUI 26 in which the acquired pre-operative images are labeled with the contours of the tumor and surrounding blood vessels within the ROI. In some examples, the surgeon can use the user input device 22 to label the tumor contour on the GUI 26. The GUI 26 can be displayed on the display device 24. The user may, for example, use a mouse or touch screen to draw a contour, mark points around the contour, and then connect these points, such as by a spline curve, or draw the entire contour or a portion of it as one or more continuous lines. In other examples, a pre-operative MR image is acquired with the MR device 32, and the tumor contour is manually labeled by a medical professional using the user input device 22. At the start of a procedure, the probe 12 is aligned with the pre-operative MR contour, and the contour is transferred to the acquired US image on the display device 24.

[0039] At 106, the at least one electronic processor 20 is programmed to train the NN 30 using US imaging data together with MR-based tumor contours of similar ROIs of different patients to generate a trained neural network. For example, the imaging data can be retrieved from the non-transitory computer-readable medium 28 and applied to the NN 30 to train the NN. It will be appreciated that operation 106 can be performed before, simultaneously with, or after operations 102 and / or 104. In an offline configuration, for example where a vendor supplies the trained NN 30, operation 106 is performed before operations 102, 104.

[0040] At 108, the at least one electronic processor 20 is programmed to tune the trained NN 30 to the patient using the labeled 2D pre-operative images generated in acts 102, 104 to generate a patient-tuned training neural network. In some embodiments, the at least one electronic processor 20 is programmed to update the patient-tuned training neural network 30 using the labeled 2D live US images (from 104). In addition, at a later stage of neurosurgery, if the neurosurgeon suspects that the contours superimposed on the live imaging have become misaligned due to gradual brain shifts, additional patient-specific training data can be generated by manual contouring of intraoperative images and / or MR images acquired by in-room MRI 32 (if available) and repeating act 108 on this additional patient-specific data to update the patient-specific tuning of the NN 30.

[0041] At 110, the at least one electronic processor 20 is programmed to perform live imaging by controlling the US scanner 14 and the US probe 12 to acquire live (i.e., intraoperative) images (e.g., 2D images) of the tumor and surrounding vessels within the patient's ROI (e.g., brain). These images are acquired during the surgical procedure. It should be noted that the live imaging does not necessarily have to be a continuous stream of images. Rather, in some embodiments, the neurosurgeon may (re)position the US probe 12 whenever the neurosurgeon desires to perform tumor visualization to acquire new live images.

[0042] At 112, the at least one electronic processor 20 is programmed to input the live image into the patient-tuned training NN to output live contours of the tumor and surrounding vessels. In other words, the at least one electronic processor 20 is programmed to apply the NN 30 to the live brain image to generate live contours of the tumor and surrounding vessels imaged in the live brain image. In this way, the contours of the tumor and surrounding vessels are automatically generated during the surgical procedure. This may be done directly if the US sensor array 16 is capable of insonifying a 3D volume, or in conjunction with a freehand or motor-driven sweep of the US probe 12. Preferably, the image is displayed on the display device 24 with the contours output by operation 112 superimposed (operation 114), providing live US imaging of the interventional procedure with indications (i.e., "live" contours) of the tumor and surrounding vessels.

[0043] In one embodiment, the at least one electronic processor 20 is programmed to train the NN 30 using the labeled pre-operative images (from 104). In another embodiment, the patient-adjusted training neural network 30 is updated at least in part with contours drawn in the intra-operative MR images. To do so, the at least one electronic processor 20 is programmed to control the MR imager 32 to acquire MR images of the ROI during the surgical procedure. A GUI 26 is provided, whereby the acquired MR images are labeled with the contours of the tumor and surrounding vessels (similar to 104). The US scanner 14 and the probe 12 are controlled by the at least one electronic processor 20 to acquire an updated series of US images. The GUI 26 is provided or updated with an updated series of the acquired US images that are labeled with the contours of the tumor and surrounding vessels. The updated series of US images are used to update the patient-adjusted training NN 30.

[0044] In another embodiment, acquisition of the live brain images is interrupted and at least one electronic processor 20 is programmed to provide a GUI 26 in which a selected live brain image is labeled with contours of the tumor and surrounding blood vessels, and NN 30 is updated to generate the live contours using the selected and labeled live brain images.

[0045] At 114, at least one electronic processor 20 is programmed to control display device 24 to display the live US image superimposed with the live contour. In some examples, the display on display device 24 includes the live contour superimposed with a reference contour from a spatially registered pre-operative magnetic resonance image. The MRI image can be spatially registered with the live US image, and in some examples, the pre-operative MRI image can be displayed in a separate window during surgery for viewing by the surgeon.

[0046] example

[0047] This disclosure includes a machine learning framework that uses deep learning with convolutional neural networks (CNNs) to train on segmented (i.e. contoured) pre-operative MRI, pre-operative US, segmented intra-operative MRI, and intra-operative US to understand how different tumors look in MRI vs. US and what are the similar or different features. During training, MRIs provide ground truth with manually performed segmentation, with tracked acquisitions overlaid on the US images, which serve as the training data.

[0048] Once the network is trained, it can be deployed as software on the workstation used by the surgeon during surgery, or alternatively, loaded onto an ultrasound scanner to provide additional information during intraoperative US examinations based on the preoperative MRI and US images.

[0049] The data required to train a deep neural network for automatic tumor identification during neurosurgery consists of aligned ultrasound (US) and magnetic resonance (MR) image pairs. The MR images serve as the ground truth since most brain lesions are visible in and segmented from the MR images. Here, all segmentations are assumed to be performed by experts in the field and are therefore the ground truth. The segmentation is represented as a binary volume matrix with values ​​of 1 inside the tumor volume and values ​​of 0 outside. The corresponding US image data is always acquired while being tracked so that a high-precision registration between the US and MR images can be performed. The tracked pre- and operative US images (intra-operative US can only be used if there is a paired matching intra-operative segmented MR image) are used to create a stack called training samples. For each training sample, the training is performed with the corresponding aligned segmentation volume. Finally, the segmentation information is propagated onto the ultrasound images in order to identify the tumor in the ultrasound images.

[0050] Figure 3 shows a schematic diagram of an exemplary convolutional neural network. The input is a volume generated from tracked and registered ultrasound images. The target is a binary segmentation map of the same geometry obtained manually from MR images.

[0051] The objective of the convolutional neural network is to determine the residual tumor segmentation from an ultrasound image, therefore a network architecture with US images as the input is proposed. To train this network, a target (ground truth output used during training) segmentation result obtained by processing the MR image superimposed on the input ultrasound image is used. Such an architecture uses several convolutional layers in the first layer combined with a pooling layer. Typically, the convolutional layers are followed by one or more fully connected layers and finally an output layer that returns a probability value for the pixel values ​​of the binary segmentation map.

[0052] The network is trained using standard neural network training techniques such as backpropagation for sufficiently large training data. There may be large inter-patient variability in tumor appearance, and the model output after initial population-based training may not be accurate enough for each case. To address this, the network is fine-tuned on the day of surgery using the pre-operative MR and US images acquired after craniotomy. This patient-specific fine-tuning step is an essential part of the workflow and helps the model to fit the specific case to be used during surgery. The neurosurgeon can perform ultrasound sweeps at any point during surgery without interrupting the workflow. The resulting tracked and registered US volume is fed to the neural network to make a prediction of the residual tumor within the swept volume. The predicted segmentation map showing the tumor area can be displayed either as a contour of the boundary of the binary segmentation map or with pseudocolor overlaid on the image and showing the probability of tumor area.

[0053] These two display formats are shown in FIG. 4, which shows contours predicting tumor location on the "left" side of the image and a heat map displaying tumor probability, with darker color shading as the likelihood of tumor location increases.

[0054] Instead, a tracked intraoperative US sweep can be performed and all output images are used to build a new updated registered volume where the remaining tumor location is marked using the neural network prediction. This recently acquired volume and its predicted segmentation are then used to display the remaining tumor for tracked resection or other interventional tools.

[0055] In an alternative embodiment, if intraoperative MR is present during surgery, the trained network can be further fine-tuned using a reinforcement learning framework. The trained network can still be used to make predictions from the intraoperative ultrasound volumes, which are then compared to the segmentations from the intraoperative MR. The model can be penalized for incorrectly estimating non-tumor tissue as tumor, or vice versa. This helps the model adapt to the current case and make more accurate future predictions of the current case.

[0056] The trained neural network is embedded in software for making intraoperative predictions, which can be run on the ultrasound scanner to estimate the residual tumor segmentation in real time as images are being acquired.

[0057] The present disclosure has been described with reference to the preferred embodiments. Modifications and alterations may occur to others upon reading and understanding the preceding detailed description. It is intended that the present disclosure be construed as including all such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.

Claims

1. At least one electronic processor In an ultrasound device having the at least one electronic processor: providing a graphical user interface in which the acquired pre-operative images are labeled by a medical professional with contours of the tumor and surrounding blood vessels within a region of interest; Tuning the trained neural network to a patient using the labeled preoperative images to generate a patient-tuned training neural network; inputting a live image from the ultrasound device into the patient-adjusted training neural network that has been trained to output a live contour of the tumor and surrounding blood vessels when the live image is input, and outputting the live contour; controlling a display device to display the live image overlaid with the live contour; providing said graphical user interface in which a selected one or more of said live images are labeled by a medical professional with contours of a tumor and surrounding blood vessels within said region of interest; and updating and training the patient-adjusted training neural network using the labeled live images and the live images. It is programmed to Ultrasonic device.

2. the at least one electronic processor: displaying the live image overlaid with contours from the spatially registered preoperative magnetic resonance image; The apparatus of claim 1 configured to:

3. the at least one electronic processor: training a neural network using imaging data of similar regions of interest of different patients to generate the trained neural network prior to the conditioning; 3. Apparatus according to claim 1 or 2.

4. the at least one electronic processor: controlling a magnetic resonance imaging device to acquire magnetic resonance images of the region of interest during a surgical procedure; providing said graphical user interface in which the acquired magnetic resonance images are labeled by a medical professional with contours of tumors and surrounding blood vessels; providing said graphical user interface, wherein an updated series of acquired ultrasound images are labeled by a medical professional with contours of said tumor and surrounding blood vessels; updating and tuning the patient-adjusted training neural network using the updated series of ultrasound images; The apparatus of claim 3 , further programmed to:

5. 1. A non-transitory storage medium storing instructions readable and executable by a processor for performing a live brain imaging method, the processor being in operative communication with an ultrasound system and a display device, the method comprising: controlling the ultrasound system to acquire two-dimensional live brain images; controlling the ultrasound system to acquire a series of two-dimensional pre-operative brain images prior to acquiring the two-dimensional live brain images; providing a graphical user interface in which the two-dimensional pre-operative brain image is labeled by a medical professional with contours of the tumor and surrounding blood vessels within a region of interest; generating live contours of a tumor and surrounding blood vessels by inputting the live 2D brain image from the ultrasound system into a neural network that has been trained to generate live contours of the tumor and surrounding blood vessels imaged in the live 2D brain image; using the labeled 2D pre-operative brain images to train the neural network used to generate the live contours; controlling the display device to present the two-dimensional live brain image overlaid with the live contour; providing said graphical user interface in which selected live brain images are labeled by a medical professional with contours of said tumor and surrounding blood vessels; using the labeled live brain images and the live brain images to update and tune the neural network used to generate the live contours; A non-transitory storage medium.

6. The live brain imaging method comprises: training the neural network used to generate the live contours using labeled ultrasound brain images of previous patients that have been labeled with contours of the tumor and surrounding blood vessels prior to controlling the ultrasound system to acquire the two-dimensional live brain images; The non-transitory storage medium of claim 5 , further comprising:

7. The step of controlling the display device further comprises: controlling the display device to present the two-dimensional live brain image superimposed with the live contour and with a reference contour obtained from a pre-operative magnetic resonance image that is spatially registered with the two-dimensional live brain image; 7. The non-transitory storage medium according to claim 5, comprising:

8. At least one electronic processor In an ultrasound system having the at least one electronic processor, providing a graphical user interface in which the acquired two-dimensional preoperative brain images are labeled by a medical professional with contours of the tumor and surrounding blood vessels within a region of interest; Tuning the trained neural network to a patient using the labeled pre-operative brain images to generate a patient-tuned training neural network; a neural network that has been trained to generate live contours of the tumor and surrounding blood vessels captured in the two-dimensional live brain image obtained by the ultrasound device by inputting the two-dimensional live brain image, and generating live contours; training the neural network used to generate the live contours using the labeled two-dimensional pre-operative brain images; controlling a display device to present the two-dimensional live brain image with the live contours superimposed thereon; providing a graphical user interface in which selected live brain images are labeled by a medical professional with contours of the tumor and surrounding blood vessels; using the labeled live brain images and the live brain images to update and tune the neural network used to generate the live contours. The ultrasound system is programmed to:

9. the at least one electronic processor: training the neural network used to generate the live contours using labeled ultrasound images of previous patients that have been labeled with contours of the tumor and surrounding blood vessels; The system of claim 8 , further programmed to:

10. the at least one electronic processor: controlling a magnetic resonance imaging device to acquire magnetic resonance images of the region of interest during a surgical procedure; providing said graphical user interface in which said acquired magnetic resonance images are labelled by a medical professional with contours of said tumour and surrounding blood vessels; providing said graphical user interface in which an updated series of acquired ultrasound images are labeled by a medical professional with contours of said tumor and surrounding blood vessels; updating and tuning the neural network using the updated series of ultrasound images. The system of claim 9 , further programmed to:

Images