Novel imaging method for realizing brain tumor functional area occupation based on mixed reality technology

Abstract

According to the novel imaging method for achieving brain tumor functional area occupation based on the mixed reality technology, anatomical relation imaging of a brain tumor and brain functional areas, nerves and blood vessels around the brain tumor can be achieved, precise positioning navigation and precise excision in an operation are achieved, and damage to the brain functional areas, the nerves and the blood vessels around the brain tumor is reduced to the maximum extent. Comprising the following steps: (1) acquiring a brain CT examination data source, and outputting the brain CT examination data source in a DICOM format; (2) acquiring a brain MRI examination data source, and outputting the brain MRI examination data source in a DICOM format; (3) acquiring a cerebrovascular CTA examination data source, and outputting the cerebrovascular CTA examination data source in a DICOM format; (4) registering the CT, MRI and CTA of the same patient to unify a coordinate system; (5) completing a brain reconstruction three-dimensional virtual model; and (6) generating the modeling result into a two-dimensional code or importing the modeling result into mixed reality glasses for result publishing.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a novel imaging method for realizing functional area occupancy of brain tumors based on mixed reality technology. Background Art

[0002] The mortality and disability rates of brain tumors have remained high, posing a serious threat to human health and quality of life. In particular, the resection of brain tumors involving functional brain areas carries a high surgical risk, and intraoperative damage to the cerebral cortex and nerve fibers in these functional brain areas is highly likely, resulting in permanent postoperative damage. Therefore, for current surgeries involving tumors in functional brain areas, maximizing tumor resection and preserving brain function are crucial for reducing disability and improving patients' quality of life. Functional brain tissue is composed of cortical and subcortical proteins related to motor, language, and sensory functions. Because tissue in functional areas cannot be repaired after damage, identifying the spatial relationship between lesions and functional tissue may be particularly important during preoperative preparation. Furthermore, there are differences between functional brain areas, and various factors, such as the displacement of functional areas by brain lesions and remodeling of functional areas, can affect function. Maximizing tumor resection while preserving important neurological functions has always been a challenge in neurosurgery.

[0003] Recently, new visual mixed reality (MR) technology, exemplified by the Metaverse, combines the advantages of virtual reality (VR) and augmented reality (AR), achieving a three-way fusion of virtual images, the real world, and the user, and a seamless information feedback pathway. Because medical MR technology reconstructs three-dimensional structures based on computed tomography (CT) and magnetic resonance imaging (MRI) data, it can achieve a 1:1 realistic reconstruction of the tumor and surrounding tissue structures, significantly enhancing the surgeon's three-dimensional (3D) spatial perception by transforming traditional two-dimensional film imaging. Using magnification, rotation, stretching, fading, and fusion functions, surgeons can create a 360° holographic display of the anatomical relationship between the tumor and functional brain regions, nerves, and blood vessels. This allows for precise intraoperative positioning, navigation, and resection, minimizing damage to surrounding functional brain regions, nerves, and blood vessels. This technology is expected to become a powerful tool for precise imaging of brain tumors within functional brain regions. Summary of the Invention

[0004] To overcome the defects of the prior art, the technical problem to be solved by the present invention is to provide a novel imaging method for realizing the occupation of functional areas of brain tumors based on mixed reality technology, which can realize the anatomical relationship imaging of brain tumors, surrounding brain functional areas, nerves and blood vessels, realize precise intraoperative positioning and navigation, and minimize the damage to surrounding brain functional areas, nerves and blood vessels.

[0005] The technical solution of the present invention is: a novel imaging method for realizing the occupation of functional areas of brain tumors based on mixed reality technology, which includes the following steps:

[0006] (1) Obtain brain CT data sources, and output the original image data in the Digital Imaging and Communications in Medicine (DICOM) format;

[0007] (2) Obtain brain MRI data sources, and output the original image data in the DICOM format;

[0008] (3) Obtain cerebral vascular CTA data sources, and output the original image data in the DICOM format;

[0009] (4) Register the CT, MRI and CTA of the same patient to unify the coordinate system;

[0010] (5) Reconstruct the 3D virtual model of the brain, including: reconstructing the skull, skin and marking points of CT / CTA; reconstructing the brain tissue and brain tumors of CT / MRI; reconstructing the cerebral arteries of CTA; reconstructing the cerebral arteries / veins of MRA / MRV; 3D modeling of brain functional areas;

[0011] (6) Publish the modeling results in a 3D scene, generate a QR code or import them into mixed reality glasses for result publication.

[0012] The present invention uses metaverse mixed reality technology to realize the anatomical relationship imaging of brain tumors, surrounding brain functional areas, nerves and blood vessels, realize precise intraoperative positioning and navigation, precise resection, and minimize the damage to surrounding brain functional areas, nerves and blood vessels, which has certain clinical significance for the treatment of brain tumor patients. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 Shows a flowchart of a novel imaging method for realizing the occupation of functional areas of brain tumors based on mixed reality technology according to the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0014] As Figure 1 shown, a novel imaging method for realizing the occupation of functional areas of brain tumors based on mixed reality technology, which includes the following steps:

[0015] (1) Obtain brain CT data sources, and output the original image data in the Digital Imaging and Communications in Medicine (DICOM) format;

[0016] (2) Obtain the brain MRI data source, and output the original image data in DICOM format;

[0017] (3) Obtain the cerebral CTA data source, and output the original image data in DICOM format;

[0018] (4) Register the CT, MRI, and CTA of the same patient to unify the coordinate system;

[0019] (5) Reconstruct the 3D virtual model of the brain, including: Reconstruct the 3D virtual model of the brain, including: Reconstruct the skull, skin, and marking points of CT / CTA; Reconstruct the brain tissue and brain tumors of CT / MRI; Reconstruct the cerebral arteries of CTA; Reconstruct the cerebral arteries / veins of MRA / MRV; 3D modeling of the brain functional area;

[0020] (6) Publish the modeling results in a 3D scene, generate a QR code or import them into mixed reality glasses for result publication.

[0021] The present invention uses the metaverse mixed reality technology to realize the anatomical relationship imaging of brain tumors, the brain functional areas, nerves, and blood vessels around the tumors, realizes intraoperative precise positioning and navigation, precise resection, and maximally reduces the damage to the brain functional areas, nerves, and blood vessels around the tumors, which has certain clinical significance for the treatment of brain tumor patients.

[0022] Preferably, in the step (1), before the brain CT scan, stick 3 to 6 markers near the scalp area to be operated on the patient, and then perform a thin-slice three-dimensional CT scan of the head. Use a 256-slice CT for spiral scanning, and the scanning parameter is a slice thickness of 1 mm.

[0023] Preferably, in the step (2), use a 3.0T magnetic resonance scanner and an 8-channel phased array head coil to perform magnetic resonance image scanning. The scanning parameter is a slice thickness of 1 mm, and the scanning sequences include plain scan and enhancement; Add other sequences according to the tumor situation, and the other sequences are: magnetic resonance angiography (MRA), magnetic resonance venography (MRV), functional magnetic resonance imaging (fMRI).

[0024] Preferably, in the step (3), before the brain CTA scan, stick 3 - 6 markers near the scalp area to be operated on the patient, and use a Philips 256-slice CT for spiral scanning. The scanning parameter is a slice thickness of 0.625 mm.

[0025] Preferably, in the step (4), the CT, MRI or CTA of the same patient are respectively registered to unify the coordinate system through the semi-automatic matching module or the automatic matching module of the 3D-slicer software.

[0026] Preferably, in the CTA reconstruction of cerebral arteriovenous in the step (5), load the patient's cranial CTA data, run the SegmentEditor module, create a new segment 1, draw the intracranial blood vessels by the threshold method; create a new segment 2, draw the skull by the threshold method; select grow in Margin, click the apply button once, display segment 1 and segment 2 at the same time, and observe the relationship between the blood vessels and the skull; use the subtraction in logical operation, subtract segment 2 from segment 1, remove segment 2, and segment 1 is displayed as blood vessels, and view the blood vessels in 3D; use islands to remove relevant bone mass and remove some discontinuous fragments; at the same time, use scissors to cut off some continuous bone mass, and alternate islands and scissors to remove the remaining bone mass; use the brush and eraser tools to remove the remaining bone mass to obtain blood vessels containing arteriovenous; continue the copy in logical operation to get segment 2, use scissors and islands alternately, subtract the venous system from segment 1 to obtain arteries; continue the subtraction in logical operation, subtract segment 1 from segment 2 to obtain the venous system, display the arteriovenous in 3D, export the arteriovenous model by segmentation, and save it in the STL format.

[0027] Preferably, in the CT / CTA reconstruction of the skull, skin, and marking points in the step (5), load the patient's cranial CT data, run the Segment Editor module, create a new segment 1, draw the skull by the threshold method; use islands to remove fragments, and display the skull model in 3D; the 3D-slicer software extracts the CT skin surface information (skin and marking point images) to realize the reconstruction model of the skin and marking points; export the skull, skin, and marking point models by segmentation, and save them in the STL format.

[0028] Preferably, in the MRI / CT reconstruction of the brain tissue in step (5), load the patient's cranial MRI / CT data, run the Segment Editor module, create a new segment 1, draw the brain tissue using the threshold method; select the foramen magnum level, use Erase to erase the single-layer label map, select shrink in Margin, click the Apply button twice, use keep selected island in Islands to select the brain tissue, select grow in Margin, click the Apply button twice, display the brain tissue in 3D, export the brain tissue model by segmentation, and save it in STL format.

[0029] Preferably, in the MRI reconstruction of the tumor in step (5), load the patient's brain MRI data, which includes diffusion tensor imaging (DTI) data, run the Segment Editor module, create a new segment 1, draw the tumor using the threshold method, alternately use Islands and Scissors to remove the debris outside the tumor, display the tumor in 3D, export the tumor model by segmentation, and save it in STL format.

[0030] Preferably, in the CTA, MRA / MRV reconstruction of the artery / vein in step (5), load the patient's cranial MRA / MRV data, run the Segment Editor module, create a new segment1, draw the artery / vein using the threshold method, alternately use Islands and Scissors to remove the debris outside the tumor, display the artery / vein in 3D, export the artery / vein model by segmentation, and save it in STL format.

[0031] Preferably, in the 3D modeling of the brain functional area in step (5), according to the different brain tissue marks in the MRI data, use the MC (Marching Cubes) algorithm to reconstruct each model: divide the input overall three-dimensional volume data into a series of small cubes; analyze the vertex state of each cube one by one, use the relationship between the mark values of the 8 vertices of the small cube and the preset threshold to generate a binary index, query the predefined lookup table to determine the corresponding triangular patches; use linear interpolation to insert isosurfaces on the corresponding edges, and the algorithm constructs a relatively smooth surface on the cube boundary to achieve the 3D modeling of different brain functional areas.

[0032] Preferably, in step (6), import the generated STL format file into the 3D scene editing mode, observe the modeling results, adjust the shading, transparency, contrast, and color of the model, verify the rotation, scaling, hidden surface removal, and stretching functions of the model. After passing the verification, generate a QR code, which can be read by scanning with an electronic device or directly imported into a mixed reality glasses for real-time imaging.

[0033] The beneficial technical effects of the present invention are as follows:

[0034] For the first time, Metaverse mixed reality technology is used to image the anatomical relationship between brain tumors and surrounding brain functional areas, nerves, and blood vessels. This allows for precise intraoperative positioning and navigation, as well as precise resection, minimizing damage to surrounding brain functional areas, nerves, and blood vessels, meeting the specific requirements of "precision surgery" currently advocated in the surgical field both domestically and internationally. This technology also addresses the current situation of significant brain tumor surgical resection damage and poor patient prognosis, particularly by protecting the normal brain tissue functional areas, nerves, and blood vessels surrounding the tumor, achieving "precision resection" powered by Metaverse mixed reality technology.

[0035] (1) Comparison of the advantages of mixed reality technology and related visualization technologies

[0036] Mixed reality technology combines the advantages of virtual reality (VR) and augmented reality (AR). (1) Mixed reality technology is different from VR technology. VR technology is a three-dimensional virtual world generated by computer simulation. The user is completely immersed in this virtual environment and has no interaction with the real world. Mixed reality technology does not cut off the user's connection with the real world. It allows the virtual environment and the real world to be maintained at the same time. The user can adjust the connection between the real world and the virtual world according to their needs to achieve seamless interaction. (2) The difference between mixed reality technology and AR technology is that AR technology applies virtual information to the real world, superimposing the real environment and virtual objects on the same interface or space at the same time. Usually, the medium of this interface or space is carried by a computer, mobile phone or iPad, and the outside of the medium is the real world. Mixed reality technology integrates the virtual world and the real world to generate a new visual environment. Mixed reality technology mainly imports the DICOM raw data collected by CT and MRI equipment into the 3D reconstruction software system, uses different colors to distinguish various tissues in the scanning area in turn, uses multiple different masks for visual segmentation, extraction and calculation to generate 3D virtual models, and saves and exports the 3D mesh model in STL format (StereoLithography). The reconstructed and rendered 3D virtual model is loaded into mixed reality glasses to achieve real-time imaging.

[0037] (2) Advantages of mixed reality technology in positioning and registration technology

[0038] Achieving the precise integration and real-time interaction between mixed reality technology and the real world largely depends on the accuracy of mixed reality registration technology. Traditional mixed reality registration methods often use intraoperative ultrasound combined with mixed reality glasses for anatomical site localization and registration. With the further development of mixed reality registration technology, major domestic and foreign R & D institutions have successively developed the following technologies: ① Visual space coordinate technology. Visual markers are set at preset positions in the operation area, and based on the spatial coordinates of the visual markers and the relative pose relationship between the visual markers and the operation area, the pose of the operation area in the visual coordinate system is obtained, and a virtual three-dimensional model including the operation path and the component placement position is transformed into the visual coordinate system to achieve mixed reality registration; ② Coordinate acquisition module technology. Visual markers are set at preset positions in the operation area, and through the coordinate conversion module, based on the spatial coordinates of the visual markers and the relative pose relationship between the visual markers and the operation area, the pose of the operation area in the visual coordinate system is obtained to achieve mixed reality registration; ③ Electronic storage and reading device. It mainly includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, mixed reality registration is achieved through visual space coordinate technology. This project intends to adopt technologies such as non-rigid registration of deep learning networks, semi-supervised small-sample joint learning network architecture, and uncertainty modeling and measurement to improve the real-time registration performance of mixed reality technology.

[0039] (3) Advantages of Mixed Reality Technology in the Establishment of Surgical Navigation System

[0040] The research on the mixed reality technology-assisted surgical navigation system is still in its initial stage. Although there have been research reports on the application of mixed reality technology in traditional surgery, minimally invasive surgery, and interventional surgery, there are no clear standards for the specific application conditions, best operation procedures, usage methods, and the selection of time points during the entire operation in different surgeries. ① Taking traditional surgery as an example, real-time positioning and registration of mixed reality with the patient's body surface enable the surgeon to find the location of the lesion immediately and effectively avoid damage to important blood vessels and nerves, improving surgical safety while achieving precise surgery and precise resection. ② Taking minimally invasive surgery as an example, fusing the mixed reality model with the endoscopic video stream enables the positioning and registration of the model within the endoscopic display screen, which is beneficial for precisely locating the lesion and effectively displaying the anatomical structure of the patient's surrounding tissues, playing a guiding role. ③ Taking interventional surgery as an example, 1:1 positioning and registration of the mixed reality model with the patient can reduce the X-ray tube exposure and the dose of contrast agent in interventional surgery, save surgical time, and play a protective role for both doctors and patients.

[0041] The above are only the preferred embodiments of the present invention, and do not impose any form of limitation on the present invention. Any simple modification, equivalent change, and modification made to the above embodiments based on the technical essence of the present invention still fall within the protection scope of the technical solution of the present invention.

Claims

1. A novel imaging method for occupying functional areas of brain tumors based on mixed reality technology, characterized in that: It includes the following steps: (1) Obtain the brain CT data source, and output the original image data in the DICOM format of Digital Imaging and Communications in Medicine; (2) Obtain the brain MRI data source, and output the original image data in the DICOM format; (3) Obtain the cerebral CTA data source, and output the original image data in the DICOM format; (4) Register and unify the coordinate systems of the CT, MRI, and CTA examination data of the same patient; (5) Reconstruct a 3D virtual model of the brain, including: Reconstruct the skull, skin, and marking points with CT / CTA; reconstruct the brain tissue and brain tumors with CT / MRI; reconstruct the cerebral arteries with CTA; reconstruct the cerebral arteries / veins with MRA / MRV; perform 3D modeling of the brain functional areas; (6) Publish the modeling results in a 3D scene, realize model imaging and function verification, generate a QR code or import them into a mixed reality glasses for result publication.

2. The novel imaging method for occupying the functional area of brain tumors based on mixed reality technology according to claim 1, characterized in that: In step (1), before the brain CT scan, stick 3 to 6 markers near the scalp area to be operated on the patient, and then perform a thin-slice 3D CT scan of the head. Use a 256-slice CT for spiral scanning, and the scanning parameters are a slice thickness of 1 mm.

3. The novel imaging method for occupying the functional area of brain tumors based on mixed reality technology according to claim 2, wherein: In step (2), use a 3.0T magnetic resonance scanner and an 8-channel phased array head coil to perform magnetic resonance image scanning. The scanning parameters are a slice thickness of 1 mm, and the scanning sequences include plain scan and enhancement; add other sequences according to the tumor situation, and the other sequences are: magnetic resonance angiography MRA, magnetic resonance venography MRV, and functional magnetic resonance imaging fMRI.

4. The novel imaging method for occupying the functional area of brain tumors based on mixed reality technology according to claim 3, wherein: In step (3), before the cerebral CTA scan, stick 3 to 6 markers near the scalp area to be operated on the patient, and use a Philips 256-slice CT for spiral scanning. The scanning parameters are a slice thickness of 0.625 mm.

5. The novel imaging method for occupying the functional area of brain tumors based on mixed reality technology according to claim 4, wherein: In step (4), use the semi-automatic matching module or the automatic matching module of the 3D-slicer software to register and unify the coordinate systems of the CT, MRI, or CTA of the same patient respectively.

6. The novel imaging method for occupying the functional area of brain tumors based on mixed reality technology according to claim 5, wherein: In the CTA reconstruction of cerebral arteriovenous in step (5), load the patient's head CTA data, run the SegmentEditor module, create a new segment1, and draw the intracranial blood vessels using the threshold method; create a new segment 2, and draw the skull using the threshold method; select grow in Margin, click the apply button once, and display segment 1 and segment2 simultaneously to observe the relationship between the blood vessels and the skull; use subtraction in logical operations, subtract segment 2 from segment 1, remove segment 2, and segment 1 shows the blood vessels. View the blood vessels in 3D; use islands to remove relevant bone mass and remove some discontinuous fragments; at the same time, use scissors to cut off some continuous bone mass, and alternate between islands and scissors to remove the remaining bone mass; use the brush and eraser tools to remove the remaining bone mass to obtain blood vessels containing arteriovenous; continue with copying in logical operations to obtain segment 2, and alternate between scissors and islands to subtract the venous system from segment 1 to obtain the artery; continue with subtraction in logical operations, subtract segment 1 from segment2 to obtain the venous system, display the arteriovenous in 3D, export the arteriovenous model by segmentation, and save it in STL format.

7. The novel imaging method for occupying the functional area of brain tumors based on mixed reality technology according to claim 6, wherein: In the CT / CTA reconstruction of the skull, skin, and marker points in step (5), load the patient's head CT data, run the Segment Editor module, create a new segment 1, and draw the skull using the threshold method; use islands to remove fragments and display the skull model in 3D; use the same method to extract the CT skin surface information with 3D-slicer software to realize the reconstruction models of the skin and marker points; export the skull and skin models by segmentation and save them in STL format.

8. The novel imaging method for occupying the functional area of brain tumors based on mixed reality technology according to claim 7, wherein: In the MRI / CT / CTA reconstruction of the brain tissue in step (5), load the patient's head MRI / CT / CTA data, run the Segment Editor module, create a new segment 1, and draw the brain tissue using the threshold method; select the foramen magnum layer, use Erase to erase the single-layer label map, select shrink in Margin, click the apply button twice, use keepselected island in islands to select the brain tissue, select grow in Margin, click the apply button twice, display the brain tissue in 3D, export the brain tissue model by segmentation, and save it in STL format.

9. The novel imaging method for occupying the functional area of brain tumors based on mixed reality technology according to claim 8, wherein: In the MRI reconstruction of tumors in step (5), load the patient's head MRI data, including diffusion tensor imaging DTI data, run the Segment Editor module, create a new segment 1, and draw the tumor using the threshold method. Alternate between islands and scissors to remove the fragments outside the tumor and display the tumor in 3D. Export the tumor model by segmentation and save it in STL format.

10. The novel imaging method for occupying the functional area of brain tumors based on mixed reality technology according to claim 9, characterized in that: In the MRA / MRV reconstruction of arteries / veins in step (5), load the patient's cranial MRA / MRV data, run the Segment Editor module, create a new segment1, draw the arteries / veins using the threshold method, alternately use islands and scissors to remove the debris outside the tumor, display the arteries / veins in 3D, export the arteries / veins model by segmentation, and save it in the STL format.

11. The novel imaging method for occupying the functional area of brain tumors based on mixed reality technology according to claim 10, wherein: In the 3D modeling of the brain functional area in step (5), according to the different brain tissue marks in the MRI data, use the MC algorithm to reconstruct each model: divide the input overall three-dimensional volume data into a series of small cubes; Analyze the vertex states of each cube one by one, use the relationship between the mark values of the 8 vertices of the small cube and the preset threshold to generate a binary index, query the predefined lookup table to determine the corresponding triangular patches; insert isosurfaces on the corresponding edges by linear interpolation, and the algorithm constructs a relatively smooth surface on the cube boundary to realize the 3D modeling of different brain functional areas.

12. The novel imaging method for occupying the functional area of brain tumors based on mixed reality technology according to claim 11, wherein: In step (6), import the generated STL format file into the 3D scene editing mode, observe the modeling results, and adjust the brightness, transparency, contrast, and color of the model, verify the rotation, scaling, hidden surface removal, and stretching functions of the model. After passing the verification, generate a QR code, which can be read by scanning the code with an electronic device or directly imported into the mixed reality glasses for real-time imaging.

Images