Method and system for 3-d segmentation of whole brain with tumor fused with augmented reality

Abstract

The present invention relates to a system for three-dimensional (3D) segmentation of the brain with a tumor, fused with augmented reality (AR). The system includes a data acquisition module that collects brain imaging data from multiple modalities and stores it in a common storage system. An image preprocessing module prepares the collected data, followed by an image registration module that aligns and fuses the preprocessed images. The system further incorporates an image fusion module that utilizes a trained convolutional neural network (CNN) to fuse the aligned images, dynamically adjusting weights based on the tumor's location. A segmentation module automatically segments various brain structures, including tumors, using a deep learning model. The segmented structures are converted into low-poly 3D models by a 3D model generation module, which are then rendered on AR glasses, with the user able to control visibility and transparency via an interface.

Description

[0001] Title: SYSTEM AND METHOD FOR 3-D SEGMENTATION OF WHOLE BRAIN WITH TUMOR FUSED WITH AUGMENTED REALITY

[0002] FIELD OF THE INVENTION

[0003] The present invention relates to techniques for segmenting tumors with brain image scans using deep convolutional neural networks for lesion analysis. The present invention more particularly relates to a method and a system for three dimensional (3- D) segmentation of whole brain with tumor fused with augmented reality.

[0004] BACKGROUND OF THE INVENTION

[0005] Segmentation of anatomical structures finds application in different fields of medicine. For example, in the neuro-surgical navigation, anatomical region of interests (ROIs), are manually segmented by a neuro-anatomy expert on a three dimensional (3D) image of the brain, act as seed points for computation of white matter tractography. Tractography is an important mapping tool for neuro-surgical planning and navigation, which is limited by stringent pre-op time-constraints and limited availability of neuroanatomy experts for manual segmentation of the seed ROIs.

[0006] There are convolutional neural network (CNN) models to provide brain segmentation. However, the major drawback of CNN lies in the fuzzy segmentation outcomes and the spatial information reduction caused by the strides of convolutions and pooling operations. To further improve the segmentation accuracy and efficiency, several advanced strategies have been applied to obtain better segmentation results with approaches like dilated convolution / pooling, skip connections, as well as additional analysis and new post-processing modules like Conditional Random Field (CRF) and Hidden Conditional Random Field (HCRF).

[0007] The Fully convolutional neural network (FCN) models are available but they do not perform well on brain scans (MRI / CT / PET) with tumor . At present, it is a semiautomatic process with segmenting tumor first, removing the tumor and then performing whole brain segmentation on the resulting data. At the final step, tumor is again added to get the final 3D model. CN115631196A discloses an “Image segmentation method, model training method, device, equipment and storage medium”. The medical image segmentation method comprises the following steps: acquiring a target image sequence, and performing feature extraction on a plurality of images to obtain a plurality of feature maps; performing M-layer feature dimension reduction on the feature graph to obtain M middle-layer feature graphs; performing N-layer feature dimensionality reduction on the optical flow feature map extracted from the region of interest in the feature map to obtain N intermediate-layer optical flow feature maps; fusing the intermediate layer feature map obtained by the M-th layer of feature dimension reduction and the intermediate layer optical flow feature map obtained by the N-th layer of feature dimension reduction to obtain a spatial cross feature map of the feature map; and segmenting the region of interest from the plurality of images in the target image sequence based on the spatial cross feature maps of the plurality of feature maps. The document thus segments one region of interest which is the tumour.

[0008] US11170508B2 discloses systems and methods for the segmentation of multi-modal image data. There is provided a computer implemented method of automatic segmentation of three dimensional (3D) anatomical region of interest(s) (ROI) that includes predefined anatomical structure(s) of a target individual, comprising: receiving 3D images of a target individual, each including the predefined anatomical structure(s), each 3D image is based on a different respective imaging modality. In one implementation, each respective 3D image is inputted into a respective processing component of a multi-modal neural network, wherein each processing component independently computes a respective intermediate, and the intermediate outputs are inputted into a common last convolutional layer(s) for computing the indication of segmented 3D ROI(s). In another implementation, each respective 3D image is inputted into a respective encoding-contracting component a multi-modal neural network, wherein each encoding-contracting component independently computes a respective intermediate output. The intermediate outputs are inputted into a single common decoding-expanding component for computing the indication of segmented 3D ROI(s). The invention under this patent document uses invasive biopsy images for segmentation.

[0009] However, there is no fully automated system or method for three dimensional (3-D) segmentation of whole brain with tumor fused with augmented reality for neurosurgeons to identifying “real” relationship between the lesion and adjacent brain tissue and measurements before surgery and improve preoperative planning by the surgeons.

[0010] SUMMARY OF THE INVENTION

[0011] To address the foregoing problems, in whole or in part, and / or other problems that may have been observed by persons skilled in the art, the present disclosure provides compositions and methods as described by way of example as set forth below.

[0012] The principal object of the present invention is toprovide a method for three dimensional (3-D) segmentation of whole brain with tumor fused with augmented reality.

[0013] Another object of the invention is to provide a tumor segmentation module configured with artificial intelligence to generate a 3-D image of the whole brain segmented along with tumor.

[0014] Another object of the invention is toprovide 3-D image of segmented images of whole brain with tumor to facilitate identifying “real” relationship between the lesion and adjacent brain tissue and measurements before surgery and improve preoperative planning by the surgeons.

[0015] Another object of the invention is to provide a system for three dimensional (3-D) segmentation of whole brain with tumor fused with augmented reality to facilitate identifying “real” relationship between the lesion and adjacent brain tissue and measurements before surgery and improve preoperative planning by the surgeons. In view of the foregoing, the present invention providesa system for three- dimensional (3D) segmentation of the brain with a tumor fused with augmented reality, comprising a data acquisition module configured to automatically collect brain imaging data from different modalities and store the collected data in a common storage system; an image pre-processing module configured to pre-process the collected imaging data; an image registration module configured to align and fuse the pre-processed imaging data from the different modalities; an image fusion module configured to fuse the aligned images using a weighted average method, with weights determined dynamically based on the location of the brain tumor, utilizing a trained convolutional neural network (CNN) with attention mechanisms; a segmentation module employing a deep learning model to automatically segment various anatomical structures of the brain, including tumors, blood vessels, and other critical regions, from the fused imaging data; a 3D model generation module configured to convert the segmented anatomical structures into low-poly 3D models; an augmented reality (AR) visualization module configured to render the generated 3D models on AR glasses, with the capability to control the visibility and transparency of different brain segments via a user interface.

[0016] In another aspect of the present invention, the different modalities includes computerized tomography (CT), non-contrast CT Angiography (CTA), and magnetic resonance imaging (MRI).

[0017] In another aspect of the present invention, pre-processing includes normalizing intensity values across the modalities and adjusting the resolution for optimized computation.

[0018] In another aspect of the present invention, registration by the image registration module involves correcting for gantry tilt and applying rigid or affine registration depending on the shape and size of the images.

[0019] In another aspect of the present invention, the data acquisition module uses a DICOM interface to connect with the CT, CTA, and MRI devices In another aspect of the invention, the image fusion module dynamically adjusts the fusion weights based on specific features present in different regions of the brain images, leveraging a CNN with an attention mechanism.

[0020] In another aspect of the present invention, the invention provides a method for three- dimensional (3D) segmentation of a brain with a tumor fused with augmented reality, comprising the steps of collecting brain imaging data from different modalities and storing the collected data in a common storage system; pre-processing the collected imaging data to prepare it for further analysis; aligning and fusing the pre-processed imaging data from the different modalities; fusing the aligned images using a weighted average method, wherein the weights are determined dynamically based on the location of the brain tumor, utilizing a trained convolutional neural network (CNN) with attention mechanisms; automatically segmenting various anatomical structures of the brain, including tumors, blood vessels, and other critical regions, from the fused imaging data using a deep learning model; converting the segmented anatomical structures into low-poly 3D models; rendering the generated 3D models on augmented reality (AR) glasses, wherein the visibility and transparency of different brain segments are controlled via a user interface.

[0021] Additional features of the invention will be or will become apparent to one with skill in the art upon examination of the following figures and detailed description. It is intended that all such additional features and advantages be included within this description, be within the scope of the invention, and be protected by the accompanying claims.

[0022] BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Having thus described the subject matter of the present invention in general terms, reference will now be made to the accompanying drawings, which are not necessarily drawn to scale, and wherein: Figure 1 illustrates a block diagram of thesystem for three-dimensional (3D) segmentation of brain with tumor fused with augmented reality, in accordance with an embodiment of the present invention;Figure 2shows a method flowchart for three- dimensional (3D) segmentation of a brain with a tumor fused with augmented reality, in accordance with an embodiment of the present invention;

[0024] Skilled artisans will appreciate that elements in the drawings are illustrated for simplicity and may not have necessarily been drawn to scale. Furthermore, in terms of the construction of the device, one or more components of the device may have been represented in the drawings by conventional symbols, and the drawings may show only those specific details that are pertinent to understanding the embodiments of the present invention so as not to obscure the drawings with details that will be readily apparent to those of ordinary skill in the art having the benefit of the description herein.

[0025] DETAILED DESCRIPTION OF THE INVENTION

[0026] The subject matter of the present invention now will be described more fully hereinafter with reference to the accompanying drawings, in which some, but not all embodiments of the subject matter of the present invention are shown. Like numbers refer to like elements throughout. The subject matter of the present invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will satisfy applicable legal requirements. Indeed, many modifications and other embodiments of the subject matter of the present invention set forth herein will come to mind to one skilled in the art to which the subject matter of the present invention pertains having the benefit of the teachings presented in the foregoing descriptions and the associated drawings. All illustrations of the drawings are for the purpose of describing selected versions of the present invention and are not intended to limit the scope of the present invention. Therefore, it is to be understood that the subject matter of the present invention is not to be limited to the specific embodiments disclosed and that modifications and other embodiments are intended to be included within the scope of the appended claims.

[0027] As a preliminary matter, it will readily be understood by one having ordinary skill in the relevant art that the present disclosure has broad utility and application. As should be understood, any embodiment may incorporate only one or a plurality of the abovedisclosed aspects of the disclosure and may further incorporate only one or a plurality of the above-disclosed features. Furthermore, any embodiment discussed and identified as being “preferred” is considered to be part of a best mode contemplated for carrying out the embodiments of the present disclosure. Other embodiments also may be discussed for additional illustrative purposes in providing a full and enabling disclosure. Moreover, many embodiments, such as adaptations, variations, modifications, and equivalent arrangements, will be implicitly disclosed by the embodiments described herein and fall within the scope of the present disclosure.

[0028] Accordingly, while embodiments are described herein in detail in relation to one or more embodiments, it is to be understood that this disclosure is illustrative and example of the present disclosure and are made merely for the purposes of providing a full and enabling disclosure. The detailed disclosure herein of one or more embodiments is not intended, nor is to be construed, to limit the scope of patent protection afforded in any claim of a patent issuing here from, which scope is to be defined by the claims and the equivalents thereof. It is not intended that the scope of patent protection be defined by reading into any claim a limitation found herein that does not explicitly appear in the claim itself.

[0029] Thus, for example, any sequence(s) and / or temporal order of steps of various processes or methods that are described herein are illustrative and not restrictive. Accordingly, it should be understood that, although steps of various processes or methods may be shown and described as being in a sequence or temporal order, the steps of any such processes or methods are not limited to being carried out in any particular sequence or order, absent an indication otherwise. Indeed, the steps in such processes or methods generally may be carried out in various different sequences and orders while still falling within the scope of the present invention. Accordingly, it is intended that the scope of patent protection is to be defined by the issued claim(s) rather than the description set forth herein.

[0030] Additionally, it is important to note that each term used herein refers to that which an ordinary artisan would understand such term to mean based on the contextual use of such term herein. To the extent that the meaning of a term used herein - as understood by the ordinary artisan based on the contextual use of such term - differs in any way from any particular dictionary definition of such term, it is intended that the meaning of the term as understood by the ordinary artisan should prevail.

[0031] Furthermore, it is important to note that, as used herein, “a” and “an” each generally denotes “at least one”, but does not exclude a plurality unless the contextual use dictates otherwise. When used herein to join a list of items, “or” denotes “at least one of the items”, but does not exclude a plurality of items of the list. Finally, when used herein to join a list of items, “and” denotes “all of the items of the list”.

[0032] The present invention provides a system and method for three-dimensional (3D) segmentation of the brain with a tumor, which is seamlessly integrated with augmented reality (AR) technology. The system is designed to enhance the accuracy and efficiency of brain imaging analysis, particularly in medical applications such as pre-surgical planning, diagnosis, and educational purposes. The invention aims to provide a robust platform that automates the collection, processing, and visualization of brain imaging data from various modalities, offering an intuitive AR interface for medical professionals.

[0033] At the core of the system is a data acquisition module that automatically collects brain imaging data from multiple modalities, such as computerized tomography (CT), non-contrast CT Angiography (CTA), and magnetic resonance imaging (MRI). This data is stored in a common storage system, ensuring that all relevant information is readily accessible for further processing. The imaging data undergoes preprocessing to standardize and optimize it for subsequent analysis. The system then employs an image registration module to align and fuse the preprocessed data, ensuring that the images from different modalities are precisely correlated.

[0034] The fused images are processed using a convolutional neural network (CNN) with attention mechanisms, dynamically determining the appropriate fusion weights based on the location of the brain tumor. This sophisticated image fusion technique enhances the clarity and relevance of the images used for segmentation. The segmentation module leverages a deep learning model to automatically delineate various anatomical structures of the brain, including tumors, blood vessels, and other critical regions. The segmented structures are then converted into low-poly 3D models, which are optimized for rendering in augmented reality.

[0035] The final step involves rendering these 3D models on AR glasses, providing an immersive and interactive visualization experience for medical professionals. The AR interface allows users to control the visibility and transparency of different brain segments, offering a detailed and customizable view of the brain's internal structures. This system and method significantly improve the accuracy of brain tumor segmentation and provide an advanced tool for medical diagnosis, surgical planning, and education.

[0036] In an embodiment, the disclosed method and the system allows uploading of heavy scans of health reports such as computerized tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET).

[0037] In an embodiment, the tumor segmentation module configured with artificial intelligence is run on the uploaded scans of the health reports and generates a 3-D image of the whole brain segmented along with tumor that assists the radiologists in creating a fool-proof analysis. The 3-D image generated by the system facilitates identifying “real” relationship between the lesion and adjacent brain tissue and measurements before surgery and improve preoperative planning by the surgeons. The surgeon can use an AR glass to view the segmentations directly over patient with reduced or lesser distractions.

[0038] In an embodiment, the data related to “real” relationship between the lesion and adjacent brain tissue and measurements is available on the system using role based login. The same data is available on augmented reality (AR) glasses to provide distraction free surgeries and thus saving of precious lives.

[0039] In an embodiment, the data on the system are safe, secure and accessible through role based log in with authorized credentials. The system is capable of distinguishing the slightest anomalies and produce accurate classifications.

[0040] In an embodiment, segmentation enhances image analysis and aids practicing radiologists and neurosurgeons. These models offer a second opinion regarding analysis to the diagnosis. The radiologists and neurosurgeons may even point out anomalies that may not be obvious to the naked eye or through other available models.

[0041] In an embodiment, the system of 3-D segmentation deep learning FCN based model for the brain with tumors. The resulting 3-D model of the segmented whole brain is available to be displayed on AR glasses for use during surgery with accurate measurements. There is also a feature to hide and show one or more segments on the AR glasses.

[0042] In an embodiment, the system can segment medical images with precision and fuse multiple segments, fed to 3-D rendering software for reproduction.The images and the analysis reports are stored in the system, they can be accessed from any device by any authorized physician at any time. Thus, it helps in teleradiology and collaborative surgery planning etc.

[0043] In accordance with an embodiment of the present invention, Figure 1 depicts a block diagram of the system for three-dimensional (3D) segmentation of the brain with a tumor, integrated with augmented reality (AR). The system comprises several interconnected modules, each responsible for a specific function in the process. The Data Acquisition Module 102 is the starting point, where brain imaging data is automatically collected from various modalities, such as CT, CTA, and MRI, and stored in a common storage system. This data is then passed to the Image Preprocessing Module 104, which prepares the imaging data by normalizing intensity values and adjusting the resolution for subsequent processing.

[0044] Next, the Image Registration Module 106 aligns and fuses the pre-processed data to ensure spatial consistency across the different modalities. The fused images are further processed by the Image Fusion Module 108, where a weighted average method is applied, with weights dynamically determined by a trained convolutional neural network (CNN) based on the brain tumor's location. Following this, the Segmentation Module 110 utilizes a deep learning model to automatically segment various anatomical structures of the brain, including tumors and other critical regions. These segmented structures are then converted into low-poly 3D models by the 3D Model Generation Module 112. Finally, the AR Visualization Module 114 renders these 3D models on AR glasses, providing an interactive interface where users can control the visibility and transparency of different brain segments, facilitating improved diagnosis and surgical planning. The arrows in the diagram represent the flow of data between these modules, indicating a seamless integration from data acquisition to AR visualization.

[0045] In accordance with an embodiment of the present invention, Figure 2 shows a method flowchart for three-dimensional (3D) segmentation of a brain with a tumor fused with augmented reality. The method begins with the crucial step of collecting and storing brain imaging data from various modalities, such as CT, CTA, and MRI scans, as indicated by step 202. This data is then subjected to pre-processing (step 204), which prepares it for further analysis by normalizing intensity values and adjusting the resolution to ensure compatibility across different imaging modalities. Following the pre-processing, the method proceeds to align and fuse the pre- processed imaging data (step 206). This alignment is critical to ensure that the images from different modalities are spatially consistent. The aligned images are then fused together (step 208) using a weighted average method, with weights determined dynamically by a convolutional neural network (CNN) based on the specific characteristics of the brain tumor. Once the fusion is complete, the method employs a deep learning model to automatically segment various anatomical structures of the brain, including the tumor, blood vessels, and other critical regions, as detailed in step 210. The segmented data is subsequently converted into 3D models (step 212), which are then rendered on AR glasses (step 214). This final step enables interactive visualization, allowing users to control the visibility and transparency of different brain segments, thereby enhancing the accuracy and effectiveness of medical diagnosis and surgical planning

[0046] The invention is designed to enhance the process of three-dimensional (3D) segmentation of brain structures with tumors, integrated with augmented reality (AR) for improved visualization. The disclosed steps begins with a data collection and preprocessing software that automatically gathers CT, CTA, and MRI imaging data from various modalities for a patient, storing it in a common storage bucket, which can be either on-premises or cloud-based. This software seamlessly integrates with hospital PACS and imaging machines, retrieving imaging data as soon as it becomes available. It connects to CT, CTA, and MRI devices via a DICOM interface, ensuring comprehensive data acquisition. The data acquisition module continuously monitors the PACS and imaging modalities, automatically retrieving new data and organizing it according to patient ID. The system anonymizes the data to remove patient identifiers, ensuring compliance with privacy regulations like HIPAA.

[0047] Following data collection, the system performs preprocessing and fusion of the CT, CTA, and MRI scans into a single coherent representation. Preprocessing involves normalizing intensity values across the three modality images to facilitate better alignment and fusion. The resolution of the images is adjusted to 256x256 to optimize computation complexity and time. Image registration is the next step, which aligns the images using the spatial information contained in the DICOM headers. CT and CTA images are registered first, followed by the MRI images. The system corrects for any gantry tilt during CT acquisition using a novel algorithm developed for this purpose. Depending on the shape and size of the images, either rigid or affine registration is used. The image fusion process involves using a weighted average method to fuse the aligned images, with weights determined dynamically based on the location of the brain tumor. A trained convolutional neural network (CNN) with attention mechanisms dynamically adjusts the weights based on the specific features present in different regions of the images.

[0048] The system also includes a novel brain segmentation architecture, NeuroSeg3D, designed for automatic segmentation of brain anatomy. This architecture is specifically created for 3D medical image segmentation, combining the strengths of transformer architecture with a focus on efficiency and performance. NeuroSeg3D utilizes a lightweight and efficient hierarchical transformer architecture that calculates attention across multiscale volumetric features, capturing both local and global context effectively. The model employs a Multi-layer Perceptron decoder that aggregates both local and global features efficiently, with a multi-scale attention mechanism that focuses on features at various resolutions simultaneously. The system ensures that as much spatial information as possible is retained through the use of 3D skip connections between the encoder and decoder.

[0049] Once the anatomical structures have been segmented using a deep learning method, the output includes both the original DICOM data and various segmented datasets representing structures such as the tumor, blood vessels, medial temporal lobe structures, and other critical brain regions. These segmented datasets are represented as 3D NumPy arrays, from which isosurfaces are extracted to transform the data into 3D models. The 3D models are generated in the same coordinate space to ensure spatial consistency, meaning they accurately represent the relative positions of different anatomical structures within the brain. The models are optimized using a novel decimation algorithm to reduce polycounts. Finally, these 3D models are exported as a single GLTF (GL Transmission Format) file, which is a widely-used format for efficient transmission and loading of 3D content. This GLTF file facilitates easy visualization and manipulation of the segmented brain structures in various software applications, aiding in further analysis, surgical planning, and patient education.

[0050] The visualization component of the system is designed to render and display the resulting 3D model of the brain on any Augmented Reality glass. A rendering engine is developed using web 3D technologies, ensuring that the model can be rendered on any AR glass without requiring code modifications. The user interface (UI) is designed to automatically retrieve the names of segments from the brain 3D model and display them as side buttons. These buttons allow users to turn the visibility of different segments on or off, providing a customizable and interactive visualization experience. This AR-based visualization significantly enhances the utility of the system, offering an intuitive tool for medical professionals in diagnostics, surgical planning, and educational contexts.

[0051] Some of the non-limiting advantages of the present invention are:

[0052] • Enhanced Accuracy in Brain Tumor Analysis: The system employs advanced image fusion and deep learning techniques, ensuring highly accurate segmentation and analysis of brain tumors and other critical anatomical structures.

[0053] • Seamless Integration with Existing Medical Systems: The software integrates effortlessly with hospital PACS and imaging machines, enabling automatic data acquisition from CT, CTA, and MRI modalities without disrupting existing workflows.

[0054] • Comprehensive and Efficient 3D Visualization: The system converts segmented 2D data into optimized 3D models, which can be easily rendered and visualized on AR glasses, providing a detailed and interactive view of the brain's internal structures.

[0055] • Automated, Dynamic Processing: The use of a trained CNN with attention mechanisms allows for dynamic adjustment of image fusion parameters, reducing the need for manual intervention and improving processing efficiency.

[0056] • Improved Patient Safety and Privacy: The system anonymizes patient data during preprocessing, ensuring compliance with privacy regulations like HIPAA, while also reducing the risk of errors in manual data handling.

[0057] Terms and phrases used in this document, and variations thereof, unless otherwise expressly stated, should be construed as open-ended as opposed to limiting. As examples of the foregoing: the term “including” should be read as mean “including, without limitation” or the like; the term “example” is used to provide exemplary instances of the item in discussion, not an exhaustive or limiting list thereof; and adjectives such as “conventional,” “traditional,” “standard,” “known” and terms of similar meaning should not be construed as limiting the item described to a given time period or to an item available as of a given time, but instead should be read to encompass conventional, traditional, normal, or standard technologies that may be available or known now or at any time in the future. Likewise, a group of items linked with the conjunction “and” should not be read as requiring that each and every one of those items be present in the grouping, but rather should be read as “and / or” unless expressly stated otherwise. Similarly, a group of items linked with the conjunction “or” should not be read as requiring mutual exclusivity among that group, but rather should also be read as “and / or” unless expressly stated otherwise. Furthermore, although item, elements or components of the disclosure may be described or claimed in the singular, the plural is contemplated to be within the scope thereof unless limitation to the singular is explicitly stated. The presence of broadening words and phrases such as “one or more,” “at least,” “but not limited to” or other like phrases in some instances shall not be read to mean that the narrower case is intended or required in instances where such broadening phrases may be absent.

[0058] For the purposes of this specification and appended claims, unless otherwise indicated, all numbers expressing amounts, sizes, dimensions, proportions, shapes, formulations, parameters, percentages, quantities, characteristics, and other numerical values used in the specification and claims, are to be understood as being modified in all instances by the term “about” even though the term “about” may not expressly appear with the value, amount, or range. Accordingly, unless indicated to the contrary, the numerical parameters set forth in the following specification and attached claims are not and need not be exact, but may be approximate and / or larger or smaller as desired, reflecting tolerances, conversion factors, rounding off, measurement error and the like, and other factors known to those of skill in the art depending on the desired properties sought to be obtained by the subject matter of the present invention. For example, the term “about,” when referring to a value can be meant to encompass variations of, in some embodiments ± 100%, in some embodiments ± 50%, in some embodiments ± 20%, in some embodiments ± 10%, in some embodiments ± 5%, in some embodiments ± 1%, in some embodiments ± 0.5%, and in some embodiments ± 0.1% from the specified amount, as such variations are appropriate to perform the disclosed methods or employ the disclosed compositions.

[0059] Further, the term “about” when used in connection with one or more numbers or numerical ranges, should be understood to refer to all such numbers, including all numbers in a range and modifies that range by extending the boundaries above and below the numerical values set forth. The recitation of numerical ranges by endpoints includes all numbers, e.g., whole integers, including fractions thereof, subsumed within that range (for example, the recitation of 1 to 5 includes 1, 2, 3, 4, and 5, as well as fractions thereof, e.g., 1.5, 2.25, 3.75, 4.1, and the like) and any range within that range.

[0060] All publications, patent applications, patents, and other references mentioned in the specification are indicative of the level of those skilled in the art to which the presently disclosed subject matter pertains. All publications, patent applications, patents, and other references are herein incorporated by reference to the same extent as if each individual publication, patent application, patent, and other reference was specifically and individually indicated to be incorporated by reference. It will be understood that, although a number of patent applications, patents, and other references are referred to herein, such reference does not constitute an admission that any of these documents forms part of the common general knowledge in the art. Although the foregoing subject matter has been described in some detail by way of illustration and example for purposes of clarity of understanding, it will be understood by those skilled in the art that certain changes and modifications can be practiced within the scope of the appended claims.

Claims

ClaimsWe Claim,1. A system (100) for three-dimensional (3D) segmentation of brain with tumor fused with augmented reality, comprising: a data acquisition module(102) configured to automatically collect brain imaging data from different modalities and store the collected data in a common storage system; animage pre-processing module(104) configured to pre-process the collected imaging data; animage registration module(106) configured to align and fuse the pre- processed imaging data from the different modalities; an image fusion module(108) configured to fuse the aligned images using a weighted average method, with weights determined dynamically based on the location of the brain tumor, utilizing a trained convolutional neural network (CNN) with attention mechanisms; a segmentation module (110) employing a deep learning model to automatically segment various anatomical structures of the brain, including tumors, blood vessels, and other critical regions, from the fused imaging data; a 3D model generation module (112) configured to convert the segmented anatomical structures into low-poly 3D models; anaugmented reality (AR) visualization module (114) configured to render the generated 3D models on AR glasses, with the capability to control the visibility and transparency of different brain segments via a user interface.

2. The system as claimed in claim 1, wherein the different modalities includes computerized tomography (CT), non-contrast CT Angiography (CT A), and magnetic resonance imaging (MRI).

3. The system as claimed in claim 1, wherein pre-processing includes normalizing intensity values across the modalities and adjusting the resolution for optimized computation.

4. The system as claimed in claim 1, wherein registration by the image registration module involves correcting for gantry tilt and applying rigid or affine registration depending on the shape and size of the images.

5. The system as claimed in claim 1, wherein the data acquisition module uses a DICOM interface to connect with the CT, CTA, and MRI devices.

6. The system as claimed in claim 1, wherein the pre-processing module anonymizes the imaging data to remove patient identifiers.

7. The system as claimed in claim 1, wherein the image registration module employs an algorithm to correct gantry tilt in CT images prior to registration.

8. The system as claimed in claim 1, wherein the image fusion module dynamically adjusts the fusion weights based on specific features present in different regions of the brain images, leveraging a CNN with an attention mechanism.

9. The system as claimed in claim 1, wherein the segmentation module utilizes the NeuroSeg3D architecture, which combines transformer-based models with multi-scale attention mechanisms to capture both local and global contexts for accurate segmentation.

10. The system as claimed in claim 1, wherein the segmentation module further includes a multi-layer perceptron (MLP) decoder to aggregate local and global features efficiently.

11. The system as claimed in claim 1 , wherein the 3D model generation module employs a decimation algorithm to optimize the polycount of the generated 3D models for efficient rendering on AR glasses.

12. The system as claimed in claim 1, wherein the user interface of the AR visualization module automatically generates and displays buttons corresponding to different brain segments, allowing the user to toggle the visibility of each segment.

13. The system as claimed in claim 1, wherein the segmentation module includes a post-processing unit configured to refine the segmented regions by applying a Conditional Random Field (CRF) to ensure smooth boundaries and reduce noise in the segmented images.

14. A method for three-dimensional (3D) segmentation of a brain with a tumor fused with augmented reality, comprising the steps of: collecting brain imaging data from different modalities and storing the collected data in a common storage system; pre-processing the collected imaging data to prepare it for further analysis; aligning and fusing the pre-processed imaging data from the different modalities; fusing the aligned images using a weighted average method, wherein the weights are determined dynamically based on the location of the brain tumor,utilizing a trained convolutional neural network (CNN) with attention mechanisms; automatically segmenting various anatomical structures of the brain, including tumors, blood vessels, and other critical regions, from the fused imaging data using a deep learning model; converting the segmented anatomical structures into low-poly 3D models; rendering the generated 3D models on augmented reality (AR) glasses, wherein the visibility and transparency of different brain segments are controlled via a user interface.

Images