Biopsy Workflow Using Multimodality Imaging
By simultaneously acquiring magnetic resonance and ultrasound image data, a patient-specific deformation transformation model is generated, solving the problem of real-time pathological tissue tracking during intervention and treatment phases. This achieves high-precision image alignment and real-time lesion visualization, reducing the invasive risks to healthy tissues.
Patent Information
- Application Number
- CN202010822971.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-09-13
- Filing Date
- 2020-08-13
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2040-08-13
AI Technical Summary
Existing technologies make it difficult to track the boundaries of pathological tissues in real time during intervention and treatment, which increases the risk of invasive entry into healthy tissue areas. Furthermore, the differences in image contrast and spatial resolution between ultrasound imaging and magnetic resonance imaging make image alignment difficult.
By acquiring MRI and ultrasound image data simultaneously or almost simultaneously, a patient-specific deformation transformation model is generated. The model is trained using deep learning for image alignment, and the lesion outline is visualized in real time during the biopsy procedure.
It enables the provision of real-time lesion location information during intervention and treatment, improves image alignment accuracy, reduces the risk of invasive entry, and simplifies the multimodal deformable alignment process of the prostate.
Smart Images

Figure CN112494028B_ABST
Abstract
Description
Background Art
[0001] The subject matter disclosed herein relates to image alignment, and more particularly, to systems and methods for image alignment utilizing deformable transformations.
[0002] Magnetic resonance imaging (MRI) is becoming increasingly important for guiding interventions and treatments because it provides soft tissue contrast, enabling the location and boundaries of pathological tissues (e.g., tumors) to be mapped during the planning / diagnosis phase. However, it is difficult to integrate real-time MRI into interventional and treatment workflows due to limited patient space in the magnet bore and low imaging frame rates. Therefore, real-time MR images are often not available during the interventional / treatment phase, which increases the risk of invasive access or treatment of healthy tissue areas that were not targeted during the planning phase. In addition, clinicians may not be able to fully approach or treat the pathological tissue due to the lack of real-time tracking of pathological tissue boundaries. Although segmenting or manually tracking lesions / tumors from MR images during the planning phase can provide an initial position of the benchmark, this initial position may not be the true position of the target lesion / tumor due to physiological motion.
[0003] Ultrasound imaging can provide real-time imaging. However, ultrasound imaging provides poor soft tissue contrast, which limits the ability to locate or delineate pathology or target lesions. Linking real-time ultrasound imaging with pre-acquired MR images (acquired at different times) is time-consuming, computationally intensive, and may have inaccuracies that adversely affect the precision and results of interventional procedures. In addition, the image contrast and spatial resolution in MRI and ultrasound imaging are different, and this increases the difficulty of aligning or registering MRI and ultrasound images. Therefore, there is a need for imaging methods that provide sufficient soft tissue contrast during the intervention / treatment phase while also providing real-time location information of target lesions. Summary of the invention
[0004] The following shows an overview of certain embodiments disclosed herein. It should be understood that these aspects are provided only to provide the reader with a brief overview of these specific embodiments, and these aspects are not intended to limit the scope of the present disclosure. In fact, the present disclosure may cover various aspects that may not be shown below.
[0005] In one embodiment, a multimodal imaging method for generating a patient-specific transformation model is provided. According to the method, pre-biopsy magnetic resonance (MR) image data and pre-biopsy ultrasound image data of the patient are collected simultaneously or successively. The anatomical region of interest and one or more lesions in the MR volume generated by the MR image data are segmented. The anatomical region of interest in the ultrasound volume generated from the ultrasound image data is also segmented. The contours of the segmented anatomical region of interest in the MR volume are aligned with the corresponding contours of the anatomical region of interest in the ultrasound volume to generate a pre-biopsy volume, in which the MR volume and the ultrasound volume are aligned and the contours of one or more lesions from the MR volume are aligned on the pre-biopsy volume. A deep learning transformation model is trained based at least in part on the pre-biopsy ultrasound image data to create a patient-specific deformable transformation model.
[0006] In another embodiment, a multi-modality imaging system is provided. According to this embodiment, the imaging system includes a memory encoding a processor executable routine and a processing component configured to access the memory and execute the processor executable routine. The routine, when executed by a processing component, causes the processing component to perform actions including at least the following: acquiring pre-biopsy magnetic resonance (MR) image data and pre-biopsy ultrasound image data of a patient; segmenting an anatomical region of interest and one or more lesions in an MR volume generated from the MR image data; segmenting the anatomical region of interest in an ultrasound volume generated from the ultrasound image data; aligning contours of the segmented anatomical region of interest in the MR volume with corresponding contours of the anatomical region of interest in the ultrasound volume to generate a pre-biopsy volume, the MR volume and the ultrasound volume being aligned in the pre-biopsy volume and contours of one or more lesions from the MR volume being aligned on the pre-biopsy volume; training a deep learning transformation model based at least in part on the pre-biopsy ultrasound image data to create a patient-specific deformable transformation model; acquiring one or more biopsy ultrasound images of the patient during a biopsy procedure; performing a deformable alignment using the deformable transformation model to align each biopsy ultrasound image with the pre-biopsy volume, wherein contours of one or more lesions associated with the pre-biopsy volume are aligned with each biopsy ultrasound image; and providing real-time visualization of contours of the one or more lesions on at least one of the biopsy ultrasound images.
[0007] In an additional embodiment, a method for registering real-time images for a biopsy system is provided. According to the method, one or more biopsy ultrasound images of a patient are acquired during a biopsy procedure. A deformable alignment is performed using a deformable transformation model to align each biopsy ultrasound image with a pre-biopsy volume. The deformable transformation is patient-specific, and the contours of one or more lesions are associated with each biopsy ultrasound image. Real-time visualization of the contours of the one or more lesions is provided on some or all of the biopsy ultrasound images. BRIEF DESCRIPTION OF THE DRAWINGS
[0008] These and other features, aspects and advantages of the present invention will be better understood when the following detailed description is read with reference to the accompanying drawings, in which like characters refer to like parts throughout the several views, and in which:
[0009] Figure 1 A magnetic resonance imaging (MRI) system according to aspects of the present disclosure is shown;
[0010] Figure 2 is an embodiment of a block diagram of an ultrasound system according to aspects of the present disclosure;
[0011] Figure 3 A schematic diagram showing an embodiment of a combined magnetic resonance and ultrasound imaging system according to aspects of the present disclosure; and
[0012] Figure 4 An ultrasound MRI fusion flow chart illustrating aspects of the planning phase according to the present disclosure.
[0013] Figure 5 An ultrasound MRI fusion flow chart showing aspects of a biopsy phase according to the present disclosure. DETAILED DESCRIPTION
[0014] One or more specific embodiments will be described below. In order to provide a concise description of these embodiments, not all features of an actual implementation are described in the specification. It should be understood that in the development of any such actual implementation, as in any engineering or design project, many implementation-specific decisions must be made to achieve the developer's specific goals, such as complying with system-related and business-related constraints that may vary from implementation to implementation. In addition, it should be understood that such development efforts may be complex and time-consuming, but are still routine tasks for design, fabrication, and manufacturing for ordinary technicians who benefit from this disclosure.
[0015] When introducing elements of various embodiments of the present disclosure, the words "a," "an," "the," and "said" are intended to mean that there are one or more of those elements. The terms "comprising," "including," and "having" are intended to be inclusive, and mean that there may be additional elements in addition to the listed elements. Furthermore, any numerical examples in the following discussion are intended to be non-limiting, and thus the additional numerical values, ranges, and percentages are within the scope of the disclosed embodiments.
[0016] Some general information is provided for a general background of various aspects of the disclosure and to facilitate understanding and explanation of certain technical concepts described herein.
[0017] As discussed herein, a method is disclosed, which includes simultaneously or nearly simultaneously (e.g., continuously in time) collecting MR and transabdominal ultrasound (TAUS) prostate images in the pre-biopsy stage. Relative to transrectal ultrasound (TRUS) imaging, the use of transabdominal prostate imaging provides certain benefits, such as improved patient comfort and the absence of prostate deformation caused by TRUS probe insertion. Therefore, between diagnostic or pre-biopsy MRI and ultrasound-guided biopsy procedures, deformation of the prostate is avoided, which simplifies the multimodal deformable alignment of the prostate. Pre-biopsy TAUS image data is integrated with information derived from pre-biopsy MR imaging, which provides useful prostate tissue contrast and is also a useful diagnostic tool for evaluating prostate disease. Therefore, unlike existing methods, the present disclosure includes simultaneous or nearly simultaneous pre-biopsy MR and TAUS imaging, which is then used for guidance in the prostate biopsy scenario.
[0018] With the foregoing in mind, the following provides material related to imaging techniques and terminology in order to impart some familiarity to such imaging systems and to provide useful real-world context for other aspects of the present disclosure. With respect to magnetic resonance imaging (MRI) systems, and turning to Figure 1 (where one such system is schematically shown for reference), the interaction of a primary magnetic field, a time-varying magnetic gradient field, and a radio frequency (RF) field with gyromagnetic materials within a subject of interest (e.g., a patient) is used to generate an image or volume representation of structural and / or functional relationships within the patient. Gyromagnetic materials (such as hydrogen nuclei in water molecules) have characteristic behaviors in response to applied electromagnetic fields (e.g., constant or time-varying electric fields, magnetic fields, or combinations thereof) that can be exploited in this manner. For example, the precession of the spins of these nuclei can be affected by manipulating the field to produce RF signals that can be detected, processed, and used to reconstruct a useful image.
[0019] Considering this, refer to Figure 1, a magnetic resonance imaging system 10 is schematically shown as including a scanner 12, a scanner control circuit 14, and a system control circuit 16. The imaging system 10 also includes: a remote access and storage system 18 and / or equipment, such as a picture archiving and communication system (PACS); or other equipment, such as a teleradiology device, to enable on-site or off-site access to data acquired by the imaging system 10. Although the imaging system 10 can include any suitable scanner or detector, in the illustrated embodiment, the imaging system 10 includes a whole body scanner 12 having a housing 20, and an opening (e.g., an annular opening) formed through the housing to accommodate a patient bore 22. The patient bore 22 can be made of any suitable material (such as a non-metallic material and / or a non-magnetic material) and generally includes components of the scanner 12 adjacent to the subject. A diagnostic table 24 can be moved into the patient bore 22 so that a patient 26 can be positioned therein for imaging selected anatomical structures within the patient 26. As described herein, the patient bore 22 can include one or more bore tubes to support various components of the scanner 12 and / or the patient 26. In some embodiments, the patient aperture 22 can support a diagnostic table 24 and / or articulation components (eg, motors, pulleys, and / or slides).
[0020] The scanner 12 may include a series of associated superconducting magnetic coils for generating a controlled electromagnetic field for exciting gyromagnetic material within the anatomical structure of the imaged subject. Specifically, a main magnet coil 28 is provided for generating a main magnetic field that is substantially aligned with the axis 44 of the patient bore 22. A series of gradient coils 30, 32 and 34 (collectively referred to as 35) allow the generation of controlled gradient magnetic fields during an examination sequence for position encoding of certain gyromagnetic nuclei within the patient 26. The RF coil 36 is configured to generate radio frequency pulses to stimulate certain gyromagnetic nuclei within the patient 26. According to one aspect of the present disclosure, the RF coil 36 may be implemented on the coil support tube 22, thereby defining at least a portion of the patient bore 38. In addition, an RF shield 42 may be implemented on the shield support tube 40, thereby also defining at least a portion of the patient bore 22, to reduce electromagnetic interference within the imaging system 10 and electromagnetic interference to devices separate from the imaging system 10. In addition to the coils that may be local to the scanner 12, the imaging system 10 may also include a set of receive coils 46 (e.g., a coil array) configured to be placed proximal to (e.g., against) the patient 26. For example, the receive coils 46 may include a cervical / thoracic / lumbar (CTL) coil, a head coil, a single-sided spine coil, etc. Generally, the receive coils 46 are placed proximal to or on top of the patient 26 to receive the weak RF signals (e.g., weak relative to the transmit pulses generated by the scanning coils) generated by certain gyromagnetic nuclei in the patient 26 as the nuclei return to their relaxed state. In some embodiments, the RF coil 36 may transmit and receive RF signals, thereby fulfilling the role of the receive coil 46.
[0021] The various coils of the imaging system 10 are controlled by external circuits to generate the required fields and pulses and read the emission from the gyromagnetic material in a controlled manner. In the illustrated embodiment, the main power supply 48 provides power to the primary magnetic coil 28 to generate the main magnetic field. The drive circuit 50 may include amplification and control circuits for supplying current to the coil as defined by the digitized pulse sequence output by the scanner control circuit 14.
[0022] An RF control circuit 52 is provided for regulating the operation of the RF coil 36. The RF control circuit 52 includes a switching device for alternating between an active mode of operation and a passive mode of operation, in which the RF coil 36 transmits and does not transmit signals, respectively. The RF control circuit 52 may also include an amplification circuit to generate RF pulses. Similarly, the receive coil 46 or the RF coil 36 (if a separate receive coil 46 is not implemented) is connected to a switch 54 that is capable of switching the receive coil 46 between a receive mode and a non-receive mode. Thus, in the receive mode, the receive coil 46 can resonate with the RF signal generated by the release of gyromagnetic nuclei in the patient 26, and in the non-receive mode, avoid resonating with the RF signal. In addition, the receive circuit 56 can receive data detected by the receive coil 46 and may include one or more multiplexing and / or amplification circuits.
[0023] It should be noted that although the scanner 12 and control / amplification circuits described above are shown as being connected by a single line, one or more cables or connectors may be used depending on the specific implementation. For example, separate lines may be used for control, data communication, power transmission, etc. In addition, appropriate hardware may be provided along each type of line for properly processing data and current / voltage. In practice, various filters, digitizers, and processors may be provided between the scanner 12 and the scanner control circuit 14 and / or the system control circuit 16.
[0024] As shown, the scanner control circuit 14 includes an interface circuit 58 that outputs signals for driving the gradient coil 35 and the RF coil 36 and for receiving data representing magnetic resonance signals generated in the examination sequence. The interface circuit 58 can be connected to a control and analysis circuit 60. Based on a defined protocol selected via the system control circuit 16, the control and analysis circuit 60 executes commands for the drive circuit 50 and the RF control circuit 52.
[0025] The control and analysis circuitry 60 may also be used to receive magnetic resonance signals and perform subsequent processing before transmitting the data to the system control circuitry 16. The scanner control circuitry 14 may also include one or more memory circuits 62 that store configuration parameters, pulse sequence descriptions, examination results, etc. during operation.
[0026] The second interface circuit 64 may connect the control and analysis circuit 60 to the system control circuit 66 for exchanging data between the scanner control circuit 14 and the system control circuit 16. The system control circuit 16 may include a third interface circuit 68 that receives data from the scanner control circuit 14 and transmits data and commands back to the scanner control circuit 14. As with the control and analysis circuit 60, the system control circuit 66 may include a computer processing unit (CPU) in a multi-purpose or special-purpose computer or workstation. The system control circuit 66 may include or be connected to a second memory circuit 70 to store programming code for operating the imaging system 10, and to store processed image data for later reconstruction, display, and transmission. The programming code may execute one or more algorithms that, when executed by a processor, are configured to perform reconstruction of acquired data or other operations involving acquired data.
[0027] Additional input-output (I / O) interfaces 72 may be provided for exchanging image data, configuration parameters, etc. with external system components (such as remote access and storage system 18). Finally, system control circuitry 66 may be communicatively coupled to various peripherals to facilitate operator interface and to produce hard copies of reconstructed images. In the illustrated embodiment, these peripherals include a printer 74, a display 76, and a user interface 78, which may include, for example, devices such as a keyboard, a mouse, a touch screen (e.g., integrated with display 76), etc.
[0028] In operation, a user (e.g., a radiologist) may configure and / or oversee the controls of the imaging system 10. Additionally, the user may assist in positioning a subject (e.g., patient 26) within the patient bore 22. In some embodiments, the patient bore 22 may surround the entire subject or only a portion thereof (e.g., the patient's head, chest, and / or extremities) while an imaging session is performed.
[0029] In addition to MRI imaging systems, certain examples discussed herein also utilize ultrasound data acquisition, such as to generate ultrasound images of the same anatomical structures of interest scanned using the MRI system 10. With this in mind, and to provide familiarity with various aspects of such ultrasound imaging systems, Figure 2A block diagram of an embodiment of an ultrasound imaging system 90 is shown that is capable of acquiring ultrasound data of a patient undergoing imaging, including during interventional procedures such as a biopsy. In the illustrated embodiment, the ultrasound system 90 is a digital acquisition and beamformer system, but in other embodiments, the ultrasound system 90 can be any suitable type of ultrasound system, not limited to the type shown. The ultrasound system 90 can include an ultrasound probe 94 and a workstation 96 (e.g., a monitor, a console, a user interface) that can control the operation of the ultrasound probe 94 and can process image data acquired by the ultrasound probe 94. The ultrasound probe 94 can be coupled to the workstation 96 by any suitable technology for transmitting image data and control signals between the ultrasound probe 94 and the workstation 96, such as wireless, optical, coaxial or other suitable connections.
[0030] The ultrasound probe 94 contacts the patient 26 during ultrasound examination. The ultrasound probe 94 may include a surface facing or contacting the patient, the surface including a transducer array 98, the transducer array having a plurality of transducer elements 100, the plurality of transducer elements being capable of operating in a switching manner between a transmit mode and a receive mode. Each individual transducer element 100 may be capable of converting electrical energy into mechanical energy for transmission, and converting mechanical energy into electrical energy for reception. It should be noted that the transducer array 98 may be configured as a bidirectional transducer that is capable of transmitting ultrasound waves into the body of the subject or patient 26 and receiving such energy from the subject or patient 26 during operation when the ultrasound probe 94 is placed in contact with the patient 26. More specifically, the transducer element 100 may convert electrical energy from the ultrasound probe 94 into ultrasound waves (e.g., ultrasound energy, sound waves) and transmit the ultrasound waves into the body of the patient 26. Ultrasonic waves may be reflected, such as from tissue of the patient 26, back to the transducer array 98, and the transducer elements 100 may convert the ultrasonic energy received from the patient 26 (the reflected signals or echoes) into electrical signals for processing by the ultrasound probe 94 and the workstation 96 to provide data that can be analyzed. The number of transducer elements 100 in the transducer array 98 and the frequency at which the transducer elements 100 operate may vary depending on the application. In some embodiments, the probe 94 may include Figure 2 Additional elements not shown, such as additional electronics, data acquisition, processing controls, etc.
[0031] As previously discussed, the ultrasound probe 94 is communicatively coupled to the workstation 96 of the ultrasound imaging system 90 to facilitate image collection and processing. As will be appreciated, the workstation 96 may include a number of components or features to control the operation of the ultrasound probe 94, facilitate the placement and / or guidance of the ultrasound probe 94, and facilitate the generation and / or interpretation of ultrasound data (including reconstructed ultrasound images). For example, as shown, the workstation 96 may include a controller 104, processing circuitry 106, one or more user input devices 108, and a display 110. In certain embodiments, the workstation 96 may include Figure 2 Additional elements not shown, such as additional data acquisition and processing controls, additional image display panels, multiple user interfaces, etc.
[0032] The controller 104 may include a memory 112 and a processor 114. In some embodiments, the memory 112 may include one or more tangible, non-transitory computer-readable media that stores instructions that can be executed by the processor 114 and / or data to be processed by the processor 114. For example, the memory 112 may include a random access memory (RAM), a read-only memory (ROM), a rewritable non-volatile memory (such as flash memory), a hard drive, an optical disk, etc. In addition, the processor 114 may include one or more general-purpose microprocessors, one or more application-specific processors (ASICs), one or more field programmable logic arrays (FPGAs), or any combination thereof. The controller 104 may control the transmission of ultrasound waves into the patient 26 via the transducer array 98.
[0033] The processing circuit 106 may include receiving and converting circuits. The processing circuit 106 may receive electrical signal data representing reflected ultrasonic energy returned from tissue interfaces within the patient 26 from the transducer array 98 of the ultrasound probe 94. The processing circuit 106 may process the data from the transducer array 98, such as correcting noise artifacts, etc. The processing circuit 106 may then convert the signal data into an ultrasound image for presentation via the display 110. The controller 104 may cause one or more ultrasound images (or a construction or model generated based on such images or raw image data) to be displayed, the image being generated by the processing circuit 106 based on the signal data received from the transducer array 98 of the ultrasound probe 94.
[0034] In operation, the controller 104 may receive a signal indicating a target anatomical structure of the patient 26 and / or a target scan plane of the target anatomical structure via one or more user input devices 108 of the workstation 96. The one or more user input devices 108 may include a keyboard, a touch screen, a mouse, buttons, switches, or other devices suitable for allowing an operator to input a target anatomical structure and / or a desired scan plane of the target anatomical structure. Based on the target anatomical structure and / or the target scan plane of the target anatomical structure, the controller 104 may output a signal to the transducer array 98 of the ultrasound probe 94 indicating instructions to convert electrical energy from the ultrasound probe 94 into ultrasonic waves and transmit the ultrasonic waves into the patient 26 and detect ultrasonic energy reflected back from tissue interfaces in the patient 26.
[0035] Taking into account the previous comments, Figure 3 A schematic diagram of an embodiment of a combined MR and ultrasound imaging system 130 is shown, which can be used for non-invasive motion management of radiation therapy or other treatments or surgical or interventional procedures, as described herein. The combined MR and ultrasound imaging system 130 can be similar to the system described in U.S. patent application No. 15 / 870,519, entitled "Image-guided Biopsy Techniques," filed on January 12, 2018, which application can be incorporated herein by reference in its entirety. The combined MR and ultrasound imaging system 130 includes a magnetic resonance (MR) imaging system 10 and an ultrasound imaging system 90. The ultrasound imaging system 90 can be communicatively coupled to an MR-compatible ultrasound probe 132. The MR-compatible ultrasound probe 132 can be an ultrasound probe configured to be used in conjunction with the MR imaging system 10. Thus, an MR-compatible ultrasound probe (as described in U.S. Patent Application No. 15 / 897,964, filed on February 15, 2018, entitled “Magnetic Resonance Compatible Ultrasound Probe”, which application may be incorporated herein by reference in its entirety) may contain low levels or no ferromagnetic materials (e.g., iron, nickel, cobalt), as described in reference to Figure 1 discussed in more detail.
[0036] To facilitate a simpler workflow, the ultrasound probe 132 may be capable of three-dimensional (3D) volume acquisition at high temporal resolution to allow ultrasound image volumes to be acquired at each time point. In addition, in addition to being MR compatible, the 3D ultrasound probe 132 may be electronically manipulable and hands-free. This allows the ultrasound image field of view to be electronically manipulated, thereby not requiring a robot or mechanical ultrasound probe holder to change the imaging field of view. In this way, simultaneous MR images and ultrasound images can be easily acquired. In addition, during interventional procedures (e.g., biopsy), the same ultrasound probe can be used and positioned without difficulty in substantially the same manner as during the pre-interventional MR+ultrasound procedure. This further simplifies the workflow because substantially the same imaging settings are used between the pre-interventional and interventional procedures, and the same ultrasound probe is used and the same manner is adopted. Data from the MR and ultrasound systems may be streamed to and stored in a memory system 134, which contains a trained network model 136, as discussed in more detail herein, and may be connected to other data storage or processing systems.
[0037] Although the foregoing describes relevant or used aspects of a combined imaging system that may be used prior to a procedure (e.g., pre-biopsy or other interventional procedure), other aspects of the system may be relevant or used during a procedure (e.g., a biopsy). By way of example, during a procedure, a biopsy system 152 may be present and may utilize information collected using the combined imaging system 130. In this example, the biopsy system 152 represents an interventional component, which in other contexts may be a radiotherapy system, a surgical interventional system, or another medical interventional component. In some embodiments, the biopsy system 152 may be a transperineal biopsy system. As discussed in more detail below, the biopsy system 152 may be guided by images obtained by the MR imaging system 10 in combination with images obtained by the ultrasound imaging system 90.
[0038] It should be noted that the systems and processes described herein require two stages in a biopsy procedure or interventional procedure, namely a pre-biopsy stage (e.g., a patient-specific planning stage) in which simultaneous MR and ultrasound imaging is performed; and a biopsy stage or procedure (e.g., an interventional / treatment stage) in which ultrasound imaging is performed. With this in mind, the combined MR and ultrasound imaging system 130 may also include a system controller block 140 that is communicatively coupled to other elements of the combined MR and ultrasound imaging system 130, including the MR imaging system 10, the ultrasound imaging system 90, and the treatment (e.g., biopsy) system 152. The controller 140 may include a memory 134 and a processor 138. In some embodiments, the memory 134 may include one or more tangible, non-transitory computer-readable media that stores instructions that can be executed by the processor 138 and / or data to be processed by the processor 138. For example, the memory 134 may include a random access memory (RAM), a read-only memory (ROM), a rewritable non-volatile memory (such as flash memory), a hard drive, an optical disk, etc. In addition, the processor 138 may include one or more general purpose microprocessors, one or more graphics processing units (GPUs), one or more application specific processors (ASICs), one or more field programmable logic arrays (FPGAs), or any combination thereof. In addition, the memory 134 may store instructions that can be executed by the processor 138 to perform the methods described herein. In addition, the memory 134 may store images obtained by the MR imaging system 10 and the ultrasound imaging system 90 and / or algorithms utilized by the processor 138 to help guide the biopsy system 152 based on image inputs from the MR imaging system 10 and the ultrasound imaging system 14, as discussed in more detail below. The memory 134 may also store a neural network 136 that, when trained, serves as a deformable registration network based on unsupervised deep learning, as described in more detail below. In some embodiments, the system may be coupled to a remote database that includes the neural network 136, rather than directly incorporating the neural network 136. In addition, the controller 140 may include a display 144 that may be used to display images obtained by the MR imaging system 10 and the ultrasound imaging system 90.
[0039] It should be noted that the types of MR images and ultrasound images collected may be different. For efficiency and also to match the acquisition frame rate, one embodiment has a two-dimensional (2D) MR image collected simultaneously with a three-dimensional (3D) ultrasound image over time, that is, the time series 2D MR image is matched with the time series 3D ultrasound image at each time point. In this case, the endogenous reference structures / markers in the time series 3D ultrasound image may be uncommon to the time series 2D MR images. In certain embodiments, the MR images and / or ultrasound images can be superimposed 2D images collected over time.
[0040] Alternatively, by classifying the time series 2D MR images into corresponding sets, each set representing the same respiratory state, a reformatted time series 3D (4D) MR image that is temporally matched with the time series 3D (4D) ultrasound image can be generated. This increases the likelihood that endogenous reference structures / markers in the ultrasound image are common in the MR image. However, even if this is not the case, the technology disclosed below provides sufficient matching or registration between the MR image and the ultrasound image. In another embodiment, at a sufficiently fast acquisition speed, the time series 3D (4D) MR images are acquired simultaneously with the time series 3D (4D) ultrasound images to produce corresponding time series 3D volume images that are temporally matched. Since the MR image set at the reference time point and the corresponding ultrasound image volume are acquired at the same time point, the organs of interest visualized in MR and ultrasound have the same shape, orientation, and characteristic contours, making the transformation from MR to ultrasound easier to perform.
[0041] Considering the foregoing, and using the prostate biopsy example to provide a real-world scenario and facilitate explanation, in conventional clinical practice, only the pre-biopsy MR volume (with a phased array surface coil) is usually collected. The pre-biopsy assessment of the prostate is usually not performed using a transrectal ultrasound (TRUS) probe. On the contrary, on the day of the biopsy procedure, the patient undergoes two TRUS probe insertions. In the first TRUS procedure, a 2D scan of the prostate is performed to reconstruct the 3D ultrasound volume of the prostate. Subsequently, the ultrasound volume is aligned with the pre-biopsy MR volume in a time-consuming process, which involves manual contour trimming of the prostate contour under two modalities. The second TRUS procedure is performed during the biopsy procedure and is used for image guidance. As can be appreciated, the deformation caused by patient positioning differences, physiological conditions (such as gas in the intestine filling or rectum, and bladder filling between pre-biopsy imaging sessions and biopsy imaging sessions) may lead to the accurate integration of MR information and the difficulty of poor lesion positioning for biopsy guidance.
[0042] As discussed in the present disclosure, a method is described, which includes simultaneously or nearly simultaneously (e.g., continuously in time) acquiring MR and transabdominal ultrasound (TAUS) prostate images in the pre-biopsy stage, such as can be acquired with an MR-compatible 3D ultrasound probe. The use of transabdominal prostate imaging, although not sufficient in itself to diagnose malignant tumors, can still be sufficient to assess prostate volume. In addition, TAUS imaging can solve certain problems associated with TRUS imaging, such as patient discomfort and prostate deformation caused by TRUS probe insertion. According to the present disclosure, pre-biopsy TAUS image data is integrated with information derived from pre-biopsy MRI, which provides useful prostate tissue contrast and is also a useful diagnostic tool for assessing prostate disease. Therefore, unlike existing methods, the present disclosure includes simultaneous or nearly simultaneous pre-biopsy MR and TAUS imaging, which is then used for guidance in prostate biopsy scenarios. Compared with TRUS, TAUS probes provide patient comfort and do not deform the prostate due to probe insertion. In addition, in some scenarios, perineal biopsy can be used instead of transrectal biopsy to reduce the risk of infection associated with increased bacteria in the rectum.
[0043] Considering the foregoing, and turning to Figure 4 and Figure 5 , showing a pair of workflows. In the planning phase (in Figure 4 During the process of performing an auto-segmentation (shown in FIG. 1 ), pre-biopsy 2D MR slices are collected (step 162). These 2D MR slices are reconstructed into one or more 3D MR volumes 164. The anatomical structures of interest, such as the prostate, on the reconstructed 3D MR volume 164 are then segmented (step 170) by an automatic segmentation method or manually by a radiologist. In this example, the segmentation step 170 results in a prostate contour 172 being identified and / or marked on the pre-biopsy reconstructed MR volume 164. As the prostate is segmented, the lesions present in the reconstructed 3D MR volume 164 are also segmented (step 166) by an automatic segmentation method or manually by a radiologist. Although shown as separate segmentation steps for simplicity and ease of explanation, it should be understood that the segmentation steps 166 and 170 can be the same segmentation operation (i.e., performed by the same automatic routine or process or performed in the same manual assessment step). At step 166, the segmentation of the lesion results in the lesion contour 168 being depicted on the pre-biopsy reconstructed MR volume 164.
[0044] Simultaneously with or after MR imaging (e.g., in a temporally continuous scan), a pre-biopsy 3D transabdominal ultrasound (TAUS) volume is acquired (step 174) (such as using an MR-compatible ultrasound probe). The 3D TAUS volume may require longitudinal volume frames and is part of the pre-biopsy ultrasound image data set. In this example, these 3D TAUS volumes are reconstructed into one or more representative 3D TAUS volumes 176. The anatomical structures of interest, such as the prostate in this example, are then segmented within the reconstructed 3D TAUS volume 176 by an automatic segmentation method or manually by a radiologist (step 178). The segmentation step 178 results in the prostate contour 180 being marked or otherwise identified within the pre-biopsy reconstructed TAUS volume 176.
[0045] Once the pre-biopsy MR and TAUS contours are acquired, a surface-based deformable alignment is performed (step 182) using one or more suitable alignment routines to generate a pre-biopsy volume 184 in which the ultrasound image data and the MR image data are aligned. This planning-stage alignment may be based on at least the MR prostate contour 172 and the TAUS contour 180. In the alignment step 182, the MR prostate contour 172 is aligned with the TAUS prostate contour 180, and in this example, these aligned prostate contours are integrated with the MR lesion contour 168. Implementing a surface-based alignment algorithm helps to take into account pre-biopsy deformation of the anatomical structure of interest (e.g., the prostate), and thus may help to align the pre-biopsy MR contour and the pre-biopsy TAUS contour. In addition, due to simultaneous or nearly simultaneous acquisition, such as in temporally consecutive scans of slices and volumes, the alignment accuracy of the pre-biopsy MR contour and the pre-biopsy TAUS contour may be improved. Due to the surface-based deformable alignment, a pre-biopsy TAUS volume 184 is created, in which the TAUS prostate contour 180 is aligned with the MR prostate contour 172 and in which the MR lesion contour 168 is integrated.
[0046] As part of the planning phase, a deformable transformation model 188 is also generated to account for the movement of the prostate itself or the deformation of the prostate caused by the movement of surrounding organs or by physiological functions. In this example, a deep-learned patient-specific transformation model (step 186) can be trained based at least in part on the pre-biopsy TAUS volume 174 to capture the range of motion of the pre-biopsy TAUS volume 174. The resulting deformable transformation model 188 captures the range of motion of the prostate within the pre-biopsy TAUS volume and can subsequently be used to help map subsequent ultrasound volumes to the pre-biopsy volume 184 to which the MR volume 164 and lesion contour 168 are mapped.
[0047] Go to Figure 5During the biopsy phase, a 3D biopsy TAUS volume 194 is acquired (step 192) using an MR-compatible 3D probe or another suitable probe, which may be left in place by the pre-biopsy acquisition. The 3D biopsy TAUS volume may require a longitudinal volume frame and is part of the biopsy ultrasound image data. A deformable alignment step 196 may be performed between the TAUS volume 194 acquired during the biopsy and the pre-biopsy volume. For example, the alignment performed at step 196 during the biopsy phase may utilize the 3D biopsy TAUS volume 194, the deformable transformation model 188 from the planning phase, and the pre-biopsy volume 184, wherein the TAUS prostate contour 180 is aligned with the MR prostate contour 172 and integrated with the MR lesion contour 168. The deformable alignment 196 produces a real-time biopsy TAUS volume 198, wherein the MR prostate contour 172 and / or the lesion contour 168 are overlaid on (or otherwise displayed with) the TAUS volume 194 acquired during the biopsy.
[0048] Based on the real-time transformed image, guidance can be provided to the surgeon or pathologist during or after the biopsy, such as by an image acquisition and / or analysis system as described herein, to facilitate the biopsy and / or evaluate post-biopsy procedures. For example, such guidance can include displaying an image or image sequence (block 198) based on the real-time transformed image, which the surgeon or pathologist can refer to to perform the biopsy or evaluate the biopsy. In other scenarios, virtual or augmented reality technology can, for example, display one or more lesions on an image displayed for the surgeon or pathologist.
[0049] The technical effects of the disclosed embodiments include providing systems and methods related to automatic multimodal imaging that utilize anatomical segmentation to facilitate accurate guidance in real-time tracking of moving (i.e., dynamic) anatomical objects. Specifically, in one embodiment, a deformable transformation model is created using a pre-biopsy ultrasound volume to ultimately help overlay a pre-biopsy MRI into the biopsy ultrasound domain. Such alignment based on a deformable transformation is faster, more accurate, and robust to intensity variations and ultrasound imaging artifacts often encountered in conventional medical procedures. In addition, in one embodiment, an MR-compatible ultrasound probe is used to simultaneously acquire MRI and ultrasound during the pre-biopsy stage. Such acquisition improves image alignment accuracy and allows better initialization of subsequent deformable image alignments. When fully deformed using real-time ultrasound for guidance, pre-biopsy MRI images can result in high tissue contrast visualization of the progress of a surgical resection procedure.
[0050] The techniques in this disclosure may be applicable to other problems related to automatic structure-guided rigid, affine, or deformable alignment.The structures to be aligned may be automatically segmented using a deformable transformation model.
[0051] This written description uses examples to disclose the invention, including the best mode, and also to enable those skilled in the art to practice the invention, including making and using any devices or systems and performing any included methods. The patentable scope of the invention is defined by the claims, and may include other examples that occur to those skilled in the art. Such other examples are intended to fall within the scope of the claims if they have structural elements that do not differ from the literal language of the claims, or if they include equivalent structural elements with minor differences from the literal language of the claims.
Claims
1. A multimodal imaging method for generating a patient-specific transformation model, the method comprising the following steps: collecting pre-biopsy magnetic resonance image data and pre-biopsy ultrasound image data of the patient; segmenting an anatomical region of interest and one or more lesions in a magnetic resonance volume generated from the magnetic resonance image data; segmenting the anatomical region of interest in an ultrasound volume generated from the ultrasound image data; aligning contours of the segmented anatomical region of interest in the magnetic resonance volume with corresponding contours of the anatomical region of interest in the ultrasound volume to generate a pre-biopsy volume in which the magnetic resonance volume and the ultrasound volume are aligned and on which contours of the one or more lesions from the magnetic resonance volume are aligned; training a deep learning transformation model based at least in part on the pre-biopsy ultrasound image data to create a deformable transformation model specific to the patient, wherein the pre-biopsy ultrasound image data comprises a longitudinal volume frame; and The motion range of the pre-biopsy longitudinal volume frame is captured by the deep learning transformation model. 2 . The method of claim 1 , wherein the pre-biopsy magnetic resonance image data, the pre-biopsy ultrasound image data, and the biopsy ultrasound image are of an abdominal region of the patient.
3. The method of claim 1 , wherein the pre-biopsy magnetic resonance image data comprises a two-dimensional (2D) magnetic resonance image reconstructed into a 3D representation, and the pre-biopsy ultrasound image data comprises a three-dimensional (3D) ultrasound image acquired over time.
4. The method of claim 1, wherein the deep learning transformation model is specific to the anatomical structure of interest. The method of claim 4 , wherein the anatomical structure of interest comprises a prostate. The method of claim 1 , wherein the pre-biopsy magnetic resonance image data and the pre-biopsy ultrasound image data are acquired simultaneously. The method of claim 6 , wherein the pre-biopsy ultrasound image data is acquired using a magnetic resonance compatible ultrasound probe.
8. The method according to claim 1, further comprising the steps of: acquiring one or more biopsy ultrasound images of the patient during a biopsy procedure; performing a deformable alignment using the deformable transformation model to align each biopsy ultrasound image with the pre-biopsy volume, wherein the contours of the one or more lesions were previously associated with each biopsy ultrasound image; as well as Real-time visualization of the contours of the one or more lesions is provided on some or all of the biopsy ultrasound images.
9. A multimodal imaging system, comprising: a memory encoding a processor-executable routine; and a processing component configured to access the memory and execute the processor-executable routine, wherein the routine, when executed by the processing component, causes the processing component to perform actions including: collecting pre-biopsy magnetic resonance image data and pre-biopsy ultrasound image data of the patient; segmenting an anatomical region of interest and one or more lesions in a magnetic resonance volume generated from the magnetic resonance image data; segmenting the anatomical region of interest in an ultrasound volume generated from the ultrasound image data; aligning contours of the segmented anatomical region of interest in the magnetic resonance volume with corresponding contours of the anatomical region of interest in the ultrasound volume to generate a pre-biopsy volume in which the magnetic resonance volume and the ultrasound volume are aligned and on which contours of the one or more lesions from the magnetic resonance volume are aligned; training a deep learning transformation model based at least in part on the pre-biopsy ultrasound image data to create a deformable transformation model specific to the patient; acquiring one or more biopsy ultrasound images of the patient during a biopsy procedure; performing a deformable alignment using the deformable transformation model to align each biopsy ultrasound image with the pre-biopsy volume, wherein the contours of the one or more lesions are associated with each biopsy ultrasound image; as well as providing real-time visualization of said contours of said one or more lesions on some or all of said biopsy ultrasound images, wherein the pre-biopsy ultrasound image data comprises a longitudinal volume frame; and The motion range of the pre-biopsy longitudinal volume frame is captured by the deep learning transformation model.
10. The multi-modality imaging system of claim 9, wherein the pre-biopsy magnetic resonance image data, the pre-biopsy ultrasound image data, and the biopsy ultrasound image are of an abdominal region of the patient.
11. The multimodal imaging system of claim 9, wherein the pre-biopsy magnetic resonance image data comprises a two-dimensional (2D) magnetic resonance image reconstructed into a 3D representation, and the pre-biopsy ultrasound image data comprises a three-dimensional (3D) ultrasound image.
12. The multi-modal imaging system of claim 9, wherein the deep learning transformation model is specific to the anatomical structure of interest.
13. The multimodal imaging system of claim 9, wherein the pre-biopsy magnetic resonance image data and the pre-biopsy ultrasound image data are acquired simultaneously.
14. A system for registering real-time images for a biopsy system, the system comprising: a memory storing instructions executable by a processor; and A processor, wherein the instructions, when executed by the processor, perform actions including: acquiring one or more biopsy ultrasound images of the patient during a biopsy procedure; performing a deformable alignment using a deformable transformation model to align each biopsy ultrasound image with a pre-biopsy volume, wherein the deformable transformation is specific to the patient and wherein contours of one or more lesions are associated with each biopsy ultrasound image; and providing real-time visualization of said contours of said one or more lesions on some or all of said biopsy ultrasound images, wherein the pre-biopsy ultrasound image data comprises a longitudinal volume frame; and The motion range of the pre-biopsy longitudinal volume frame is captured by a deep learning transformation model.
15. The system of claim 14, wherein the biopsy procedure comprises a transperineal biopsy procedure.
16. The system of claim 14, wherein the deformable transformation model is generated using a deep learning transformation model trained at least in part based on pre-biopsy ultrasound image data.
17. The system of claim 16, wherein the deep learning transformation model is specific to an anatomical structure of interest.
Citation Information
Patent Citations
Magnetic resonance compatible ultrasound probe
US20180271372A1
Image-guided biopsy techniques
US20190219647A1
Method and system for performing biopsies
CN102215770A
A full convolutional neural network epilepsy focus segmentation method based on multi-mode images
CN109949318A
Ultrasonic image and CT image fusion real-time navigation system and method
CN110025379A
