Method and system for training a brain glioma segmentation and three-dimensional visualization model

Abstract

The application discloses a brain glioma segmentation and three-dimensional visualization model training method and system based on multi-modal fusion, and comprises the following steps: inputting multi-modal medical image data into a Laplacian pyramid multi-modal fusion learning model to generate a fusion graph, then inputting the fusion graph into a U-net segmentation model comprising an encoder and a decoder to obtain a segmentation mask under the multi-modal fusion graph; and finally, based on the multi-modal fusion graph and the segmentation mask, adopting a Marching cubes (MC) algorithm to perform three-dimensional reconstruction visualization of brain regions and brain tumor region labeling. The brain glioma segmentation and three-dimensional visualization model training method and system based on multi-modal fusion provided by the application optimizes the disadvantages of insufficient information and incomplete modeling under single modal, fully utilizes the advantages of the completeness of multi-modal data, performs multi-modal fusion, segmentation and three-dimensional visualization, can more comprehensively assist surgeons in treatment, and improves the accuracy in the operation process.

Description

Technical Field

[0001] This invention relates to the fields of computers and medicine, and in particular to a method and system for training a glioma segmentation and three-dimensional visualization model based on multimodal fusion. Background Technology

[0002] Glioblastoma multiforme (GBM) is the most common and deadliest type of brain tumor. Statistics show that the median postoperative survival time for glioma patients is only 8 months, and the 5-year survival rate is almost zero. Magnetic resonance imaging (MRI) plays a crucial role in diagnosing brain tumors. Due to the complexity and unique morphology of GBM tumors, single-modal MRI cannot clearly reflect the different tissue structures of GBM. Therefore, doctors often have patients undergo imaging in multiple modalities for comprehensive evaluation. However, comparing images from different modalities is difficult, often requiring years of experience from the doctor to distinguish tumor regions. Furthermore, gliomas in the brain are actually three-dimensional objects, while the radiological images used by doctors are two-dimensional, leading to frequent difficulties in localization and handling of tumor margins. This increases the risk of incomplete tumor resection leading to recurrence, or the removal of normal brain regions causing unnecessary harm to the patient's health.

[0003] To comprehensively utilize medical imaging data from different modalities, researchers have proposed multimodal fusion techniques. These techniques overlay, complement, and fuse imaging data from a single modality, allowing anatomical and functional metabolic images of tissues to be displayed in a single image, helping doctors gain a more comprehensive understanding of lesions. However, most current image fusion algorithms are single-scale algorithms. While single-scale algorithms are characterized by small space requirements and ease of implementation, and are often used in image fusion at different focal lengths, their ability to preserve detail is weak, resulting in blurry fused images, making them unsuitable for the medical field. Furthermore, in 3D visualization, most techniques are also based on single-modality reconstruction of the human brain, and cannot clearly label specific areas of brain tumors, limiting their effectiveness in assisting doctors with surgical treatment.

[0004] In contrast, multi-parameter magnetic resonance imaging (MRI) contains rich information about tissue structure. Therefore, image fusion techniques are needed to combine MRI with multi-parameter MRI to overlay, complement, and fuse single-modality imaging data. This allows anatomical and functional metabolic images of the tissue to be displayed in a single image, helping doctors to gain a more comprehensive understanding of the lesion. In the application of multi-modal image fusion in glioma surgery, the fused data is combined with computer technology to further segment the tumor region using a segmentation model. Then, three-dimensional reconstruction is performed, providing doctors with three-dimensional images of the glioma's spatial location, functional metabolism, and cranial anatomy, thus better guiding the surgeon in tumor resection. Summary of the Invention

[0005] The purpose of this invention is to achieve the fusion of medical image information of glioma patients from different medical devices, and to realize the segmentation and three-dimensional imaging display of glioma based on the fusion image obtained by the fusion algorithm, thereby assisting surgeons in completing glioma resection surgery.

[0006] To achieve the above objectives, one technical solution of the present invention provides a system for glioma segmentation and three-dimensional visualization, characterized in that it includes:

[0007] The sample database stores image data in different modalities, which is used to train the multimodal fusion image generation module, the brain tumor region segmentation module, and the brain region 3D visualization generation module.

[0008] The multimodal fusion map generation module adopts the Laplacian pyramid model structure to extract the representation information of image data under different modalities at multiple scales. Based on the representation information, the maximum fusion strategy and the local gradient energy strategy are used to fuse information and generate a multimodal fusion map.

[0009] The brain tumor region segmentation module preprocesses and augments the multimodal fusion image, and then uses a U-Net network structure to train an encoder and decoder to segment the brain tumor region in the multimodal fusion image to obtain a segmentation mask.

[0010] The brain region 3D visualization generation module aligns the multimodal fusion map and segmentation mask, then uses the MarchingCubes algorithm to reconstruct the brain region in 3D, and annotates the brain tumor region, displaying the 3D image of the brain region with the annotated brain tumor region.

[0011] Preferably, in the sample database, there is a one-to-one correspondence between medical image data of different modalities and patients, and it includes a training set, a test set, and a validation set.

[0012] Another technical solution of the present invention provides a method for training a glioma segmentation and three-dimensional visualization model, characterized by comprising the following steps:

[0013] S100, Sample Data Acquisition: Acquire multimodal radiological images of human gliomas using various commonly used clinical imaging devices, and transmit the multimodal image data to the sample database for storage;

[0014] S200, Multimodal Fusion Map Generation:

[0015] Using the Laplacian pyramid model structure, representational information of image data under different modalities is extracted at multiple scales. Based on the representational information, the maximum fusion strategy and the local gradient energy strategy are used to fuse information and generate a multimodal fusion map.

[0016] S300, Generation of brain tumor segmentation region mask:

[0017] The multimodal fusion map is preprocessed and augmented, and then an encoder and decoder are trained using the U-Net network structure to segment the brain tumor region in the multimodal fusion map and obtain the segmentation mask.

[0018] S400, Generation of 3D Visualization of Brain Regions:

[0019] The multimodal fusion map and segmentation mask are aligned, and then the Marching Cubes algorithm is used to reconstruct the brain region in three dimensions. The brain tumor region is then labeled, and the three-dimensional image of the brain region labeled with the brain tumor region is displayed.

[0020] Preferably, step S200 includes the following steps:

[0021] Step S201: Obtain image data under different modalities, downsample them to generate multiple feature images of different scales, and reduce the scale to half of the original scale each time.

[0022] Step S202: Perform residual calculations on feature maps at different scales to obtain residual information.

[0023] In the Laplace pyramid model structure, the feature map of the previous layer of each layer is upsampled to obtain a feature map with the same scale as the current layer. Then, the feature map of the current layer is subtracted from the upsampled feature map to obtain the residual information between the two feature maps.

[0024] Step S203: Fuse the obtained feature maps and residual information of different scales of different modalities to generate a multimodal fused feature map:

[0025] Feature maps and residual information of different scales are fused using a two-layer maximum fusion strategy and a two-layer local gradient energy strategy, and finally the maximum gradient fusion strategy is used for fusion.

[0026] Preferably, in step S203, in the maximum fusion strategy, the gray values ​​of the two images to be fused at the same position are compared, the larger gray value is selected as the gray value of the new image at that position, and all points in the image are traversed to obtain the fused image;

[0027] In the local gradient energy strategy, the regional gradient energy of the five points surrounding each pixel in the two images (top, bottom, left, and right) is calculated. The regional gradient energies of the five pixels are summed, and the sums are compared. The gray value of the pixel with the larger energy is taken as the gray value of the pixel in the new image. The preceding steps are repeated until all pixels in the image are traversed. The formula for calculating the local gradient energy is as follows:

[0028] GE (i,j) =(I (i+1,j) -I (i,j) ) 2 +(I (i,j+1) -I (i,j) ) 2

[0029] In the formula, (i,j) represent the row index and column index of the pixel, respectively. (i,j) GE represents the value of the pixel at position (i,j). (i,j) This represents the magnitude of the local energy gradient at the pixel at position (i,j).

[0030] Preferably, step S300 includes the following steps:

[0031] Step S301: Perform data preprocessing on the multimodal fusion map generated in step S200, including standardization and background enhancement;

[0032] Step S302: Perform data augmentation using various methods on the data processed in step S301 to improve the data's breadth.

[0033] Step S303: Input the enhanced data into a U-Net network containing an encoder and a decoder to generate a segmentation mask.

[0034] Preferably, step S400 includes the following steps:

[0035] Step S401: Align the multimodal fusion map and segmentation mask obtained in steps S200 and S300:

[0036] Alignment is used to map and label the brain tumor region to the multimodal fusion map and the segmentation mask. The brain tumor region segmented by the segmentation mask is marked and displayed on the corresponding voxel point position of the multimodal fusion map.

[0037] Step S402: The aligned data is plotted from a two-dimensional plane to a three-dimensional stereoscopic image using the Marching Cubes algorithm, outputting a three-dimensional visualized brain map and labeling the tumor region:

[0038] First, define a cubic unit in the aligned data sequence as a voxel. Each voxel consists of 8 vertices. A voxel vertex is represented by two different state variables: first, a value higher than or equal to the potential value indicates it is inside the object's surface; second, a value lower than the potential value indicates it is outside the object's surface. Therefore, a single vertex of a voxel has two possible states, and all 8 vertices of a voxel have 256 possible states.

[0039] Secondly, based on the rotation and mapping invariance properties, the state of voxels is summarized into 15 basic configurations. That is, all 256 states of voxels can be obtained through rotation and symmetry transformations using these 15 basic configurations. Each voxel state contains several triangular facets. The specific position of the vertex of the triangular facet in the voxel is calculated by linear interpolation based on the value of the isosurface and the values ​​of the two vertices of the side it is on.

[0040] Then, a lookup table is created for all 256 states of the voxels to be used when reconstructing the 3D image. The Marching Cubes algorithm finds the triangular facets among all voxels and combines them to form the final 3D visualization model of the brain.

[0041] This invention features a rationally designed structure. It utilizes a multimodal sample database for multimodal fusion learning, feeds the multimodal fusion map into a segmentation network to generate a segmentation mask, and finally uses the multimodal fusion map and segmentation mask to perform three-dimensional visualization modeling of brain regions and annotate glioma areas, assisting surgeons in making diagnoses and greatly improving the accuracy of the diagnostic process.

[0042] Compared with the prior art, the present invention has at least the following advantages:

[0043] (1) This invention proposes a multimodal fusion model based on the Laplace pyramid model, which has the characteristics of multi-scale fusion, making the details of the multimodal fusion graph more complete, the loss of fusion information less, and the fusion accuracy higher.

[0044] (2) The fusion image combines information from medical images in different modalities, enabling doctors to have a more comprehensive understanding of the overall condition of the patient's brain lesion area, which greatly improves the success rate of the doctor's surgery.

[0045] (3) This invention proposes a segmentation model based on the U-net structure consisting of an encoder and a decoder, and uses a variety of data preprocessing and data augmentation methods for training, which makes the model segmentation more accurate and more robust.

[0046] (4) This invention uses the MC algorithm to perform three-dimensional modeling of brain regions, completing the transformation of two-dimensional medical images into three-dimensional models that are more consistent with the actual morphology of human organs. This will be more conducive to assisting the surgical process. Furthermore, we have labeled the brain tumor region by segmentation masking and alignment of multimodal fusion maps, which can help doctors quickly and accurately locate the tumor.

[0047] (5) The present invention proposes a method and system for glioma segmentation and three-dimensional visualization model training based on multimodal fusion, which combines multimodal fusion technology, neural network segmentation technology and three-dimensional imaging technology to complete the integrated operation of the process. It has the characteristics and advantages of speed, convenience and comprehensiveness. Attached Figure Description

[0048] Figure 1 This is an overall framework diagram of a method and system for glioma segmentation and three-dimensional visualization model training based on multimodal fusion according to the present invention;

[0049] Figure 2 This is sample data for an embodiment of the method and system for training a glioma segmentation and three-dimensional visualization model based on multimodal fusion according to the present invention;

[0050] Figure 3 This is a multimodal fusion model framework diagram of a method and system for training a glioma segmentation and 3D visualization model based on multimodal fusion according to the present invention.

[0051] Figure 4 This is a segmentation model framework diagram of a method and system for segmenting and training a three-dimensional visualization model of glioma based on multimodal fusion, as described in this invention.

[0052] Figure 5 This is a 3D visualization diagram illustrating a method and system for training a glioma segmentation and 3D visualization model based on multimodal fusion, as described in this invention. Detailed Implementation

[0053] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0054] The invention is described more specifically by way of example in the following paragraphs with reference to the accompanying drawings. The advantages and features of the invention will become clearer from the following description and claims. It should be noted that the drawings are in a very simplified form and use non-precise proportions, and are only used to facilitate and clarify the illustration of the embodiments of the invention.

[0055] like Figure 1 As shown, the technical solution proposed in this invention achieves accurate three-dimensional reconstruction and visualization of brain regions and annotation of brain tumor regions through artificial intelligence technologies such as multimodal fusion and segmentation, generating a corresponding multimodal fusion three-dimensional visualization model of brain regions and tumors for each patient. Specifically, it includes the following steps:

[0056] Step S100: Multimodal sample data acquisition: Radiographic images of the human brain are acquired using equipment such as MRI and CT scanners, and the multimodal image data is transmitted to the multimodal sample database.

[0057] For details, please refer to Figure 2 The multimodal sample database has a one-to-one correspondence between data of different modalities and patients, and the dataset includes a training set, a test set, and a validation set.

[0058] Step S200, Multimodal Fusion Map Generation: Using the Laplacian pyramid model structure, representation information under different modes is extracted at multiple scales, and information fusion is performed using the maximum fusion strategy and the local gradient energy strategy.

[0059] For details, please refer to Figure 3 The generation of a multimodal fusion graph using the Laplace pyramid model specifically includes the following steps:

[0060] Step S201: Input multimodal medical image data and generate feature maps of different scales for the data in each modality. Specifically, we input data from different modalities, downsample them to generate multiple feature images of different scales, and each downsampling scale is reduced to half of the original.

[0061] Step S202: Perform residual calculation on feature maps at different scales to obtain residual information.

[0062] Specifically, we upsample the feature map of the previous layer to obtain an image with the same scale as the current layer, and then subtract the upsampled image from the current layer image to obtain the residual information between the two images.

[0063] Step S203: The feature maps and residual information of different scales of different modalities learned through the above steps are fused to generate a multimodal fusion feature map.

[0064] Specifically, we fuse feature maps and residual information at different scales using a two-layer maximum fusion strategy and a two-layer local gradient energy strategy, and finally use the maximum gradient fusion strategy. The calculation methods for the two fusion strategies are as follows: For the maximum fusion strategy, we compare the gray values ​​of the two images at the same location, select the larger gray value as the gray value of the new image at that location, and iterate through all points in the image to obtain the fused image. For the local gradient energy strategy, we calculate the regional gradient energy of the five points surrounding each pixel in both images (up, down, left, and right), sum the regional gradient energies of the five pixels, compare the sum of the regional gradient energies, and take the gray value of the point with the larger energy as the gray value of the new image. We repeat the previous steps until all points in the image have been traversed. The formula for calculating the local gradient energy is shown below:

[0065] GE (i,j) =(I (i+1,j) -I (i,j) ) 2 +(I (i,j+1) -I (i,j) ) 2

[0066] In the formula, (i,j) represent the row index and column index of the pixel, respectively. (i,j) GE represents the value of the pixel at position (i,j). (i,j) This represents the magnitude of the local energy gradient at the pixel at position (i,j).

[0067] Step S300: Generation of brain tumor segmentation region mask: The multimodal fusion map is preprocessed and augmented, and then the encoder and decoder are trained using the U-Net network structure to segment the brain tumor region and obtain the segmentation mask.

[0068] For details, please refer to Figure 4 The U-Net network structure is used to train an encoder and decoder to segment brain tumor regions and obtain a segmentation mask. The specific steps include the following:

[0069] Step S301: Perform data preprocessing on the multimodal fusion map generated in step S200, including normalization and background enhancement. Specifically, we prune redundant background voxels at the boundaries of each voxel (making their values ​​zero) because they provide no useful information and can be ignored by the neural network. Subsequently, for each example, the mean and standard deviation are calculated separately for the non-zero regions of each channel. Then, each voxel value is normalized by subtracting the mean and dividing by the standard deviation. Background voxels do not need to be normalized, so their values ​​remain zero. To distinguish between background voxels and voxels with values ​​close to zero, an additional input channel is created using a hot encoding for the foreground voxels and stacked with the input data.

[0070] Step S302: Perform data augmentation using various methods on the data processed in step S301 to improve the data's breadth.

[0071] Specifically, data augmentation is a technique that alleviates overfitting by artificially expanding the dataset during the training phase. The following data augmentations are used during the training phase:

[0072] Random cropping: A patch of size (5, 128, 128, 128) is randomly cropped from the input voxels. Furthermore, with a probability of 0.4, the patch selected via randomized biased cropping ensures that some foreground voxels exist within the cropped region.

[0073] Image scaling: With a probability of 0.15, a random value is uniformly sampled from (1.0, 1.4), and the image size is adjusted to its original size multiplied by the sampled value using cubic interpolation.

[0074] Horizontal flip: With a probability of 0.5, the volume is flipped along each x, y, and z axis.

[0075] Gaussian noise: Enhancement is achieved using random Gaussian noise with zero mean and a standard deviation uniformly sampled from (0, 0.33) with a probability of 0.15.

[0076] Gaussian blur: Apply Gaussian blur to the input voxels with a probability of 0.15. The standard deviation of the Gaussian kernel is uniformly sampled from (0.5, 1.5).

[0077] Brightness adjustment: With a probability of 0.15, a random value is uniformly sampled from (0.7, 1.3), and then the input voxel is multiplied by that value.

[0078] Step S303: Input the enhanced data into a U-Net network containing an encoder and a decoder to generate a segmentation mask.

[0079] Specifically, the U-Net architecture is characterized by a symmetrical U-shape and can be divided into two parts: the encoder and the decoder. The first part is the shrinking path (encoder) that transforms the input voxels into a low-dimensional space. The encoder has a modular structure consisting of repeating convolutional blocks. Each convolutional block has two smaller transformation blocks. The first smaller transformation block reduces the spatial dimension of the input feature map by half through a convolutional layer with a 3x3x3 kernel and a 2x2x2 stride. The other transformation block has a convolutional layer with a 1x1x1 stride, and the next feature map is transformed using almost the same convolutional operation, except for this transformation block's 1x1x1 stride. After the spatial dimension of the feature map is transformed to a size of 2x2x2, the decoder part begins. The decoder also has a modular structure, but its goal is to increase the spatial dimension by reducing the encoder's feature map. A block in the decoder consists of three smaller blocks. The first is a transposed convolution with a 2x2x2 kernel and a 2x2x2 stride, which doubles the spatial dimension of the feature map. Then, the upsampled feature map is concatenated with the encoder feature map from the equivalent spatial level, and then convolution is performed through two identical convolutional blocks (kernel 3x3x3, stride 1x1x1).

[0080] Step S400, Generation of 3D visualization of brain regions: Align the multimodal fusion map and the segmentation mask, then use the Marching Cubes (MC) algorithm to reconstruct the 3D brain regions, and label the brain tumor regions according to the segmentation mask.

[0081] Specifically, its 3D visualization model diagram is available for reference. Figure 5 The specific steps for using the MC algorithm to perform 3D reconstruction of brain regions and labeling brain tumor regions based on segmentation masks are as follows:

[0082] Step S401: Align the multimodal fusion map and segmentation mask obtained in steps S200 and S300:

[0083] Specifically, the alignment mainly involves mapping and labeling the brain tumor region between the multimodal fusion map and the segmentation mask, and making the brain tumor region segmented by the segmentation mask prominently marked and displayed at the corresponding voxel point position in the multimodal fusion map.

[0084] Step S402: The aligned data is plotted from a two-dimensional plane to a three-dimensional stereoscopic image using the MC algorithm, and a three-dimensional visualized brain map is output with the tumor region labeled.

[0085] Specifically, the Monte Carlo (MC) algorithm approximates isosurfaces in a 3D discrete data field using linear interpolation. First, a cubic unit in the aligned data sequence is defined as a voxel, and each voxel consists of 8 vertices. A voxel vertex is represented by two different state variables: 1. A value higher than or equal to the potential value indicates it is inside the object's surface; 2. A value lower than the potential value indicates it is outside the object's surface. Therefore, a vertex of a voxel has two possible states, resulting in a total of 2... 8 =256 states. Based on properties such as rotation and mapping invariance, the states of a voxel can be summarized into 15 basic configurations. That is, all 256 states of a voxel can be obtained through these 15 basic configurations by operations such as rotation and symmetry transformation. Each voxel state contains several triangular faces. The specific positions of the vertices of the triangular faces in the voxel need to be calculated by linear interpolation based on the values ​​of the isosurface and the values ​​of the two vertices of the edge. A lookup table can be created for these 256 states for easy application during reconstruction. By traversing all voxels, the triangular faces are found and combined to form the final 3D visualization model of the brain.

[0086] Specifically, the result output module is connected to a display screen and a printer. By configuring the result output module to connect to a display screen and a printer, the diagnostic report can be displayed on the screen and the document can be printed, facilitating detailed analysis by medical personnel.

[0087] The radiological images are acquired using various medical imaging devices. Magnetic resonance imaging (MRI) plays a crucial role in diagnosing brain tumors. Due to the complexity and unique morphology of tumors, single-modality MRI cannot clearly reflect the different tissue structures of a tumor. In contrast, multi-parameter MRI contains rich tissue structural information. Therefore, image fusion techniques are used in conjunction with multi-parameter MRI to overlay, complement, and fuse single-modality imaging data, allowing anatomical and functional metabolic images of the tissue to be displayed in a single image, helping surgeons to gain a more comprehensive understanding of the lesion.

Claims

1. A system for segmenting and three-dimensional visualization of gliomas, characterized in that, include: The sample database stores image data in different modalities, which is used to train the multimodal fusion image generation module, the brain tumor region segmentation module, and the brain region 3D visualization generation module. The multimodal fusion map generation module adopts the Laplacian pyramid model structure to extract the representation information of image data under different modalities at multiple scales. Based on the representation information, the maximum fusion strategy and the local gradient energy strategy are used to fuse information and generate a multimodal fusion map. The brain tumor region segmentation module preprocesses and augments the multimodal fusion image, and then uses a U-Net network structure to train an encoder and decoder to segment the brain tumor region in the multimodal fusion image to obtain a segmentation mask. The brain region 3D visualization generation module aligns the multimodal fusion map and segmentation mask, then uses the Marching Cubes algorithm to perform 3D reconstruction of the brain region, annotates the brain tumor region, and displays a 3D image of the brain region with the annotated brain tumor region, including: Alignment is used to map and label the brain tumor region to the multimodal fusion map and the segmentation mask. The brain tumor region segmented by the segmentation mask is marked and displayed on the corresponding voxel point position of the multimodal fusion map. The three-dimensional reconstruction of brain regions using the Marching Cubes algorithm includes: First, define a cubic unit in the aligned data sequence as a voxel. Each voxel consists of 8 vertices. A voxel vertex is represented by two different state variables: first, a value higher than or equal to the potential value indicates it is inside the object's surface; second, a value lower than the potential value indicates it is outside the object's surface. Therefore, a single vertex of a voxel has two possible states, and all 8 vertices of a voxel have 256 possible states. Secondly, based on the rotation and mapping invariance properties, the state of voxels is summarized into 15 basic configurations. That is, all 256 states of voxels can be obtained through rotation and symmetry transformations using these 15 basic configurations. Each voxel state contains several triangular facets. The specific position of the vertex of the triangular facet in the voxel is calculated by linear interpolation based on the value of the isosurface and the values ​​of the two vertices of the side it is on. Then, a lookup table is created for all 256 states of the voxels to be used when reconstructing the 3D image. The MarchingCubes algorithm finds the triangular facets among all voxels and combines them to form the final 3D visualization model of the brain.

2. The glioma segmentation and three-dimensional visualization system as described in claim 1, characterized in that, In the sample database, there is a one-to-one correspondence between medical image data of different modalities and patients, and it includes a training set, a test set, and a validation set.

3. A method for segmenting and training a three-dimensional visualization model of gliomas, characterized in that, Includes the following steps: S100, Sample Data Acquisition: Acquire multimodal radiological images of human gliomas using various commonly used clinical imaging devices, and transmit the multimodal image data to the sample database for storage; S200, Multimodal Fusion Map Generation: Using the Laplacian pyramid model structure, representational information of image data under different modalities is extracted at multiple scales. Based on the representational information, the maximum fusion strategy and the local gradient energy strategy are used to fuse information and generate a multimodal fusion map. S300, Generation of brain tumor segmentation region mask: The multimodal fusion map is preprocessed and augmented, and then an encoder and decoder are trained using the U-Net network structure to segment the brain tumor region in the multimodal fusion map and obtain the segmentation mask. S400, Generation of 3D Visualization of Brain Regions: The multimodal fusion map and segmentation mask are aligned, and then the Marching Cubes algorithm is used to reconstruct the brain region in 3D. The brain tumor region is then labeled, and the 3D image of the brain region labeled with the brain tumor region is displayed. The steps include: Step S401: Align the multimodal fusion map and segmentation mask obtained in steps S200 and S300: Alignment is used to map and label the brain tumor region to the multimodal fusion map and the segmentation mask. The brain tumor region segmented by the segmentation mask is marked and displayed on the corresponding voxel point position of the multimodal fusion map. Step S402: The aligned data is plotted from a two-dimensional plane to a three-dimensional stereoscopic image using the Marching Cubes algorithm, outputting a three-dimensional visualized brain map and labeling the tumor region: First, define a cubic unit in the aligned data sequence as a voxel. Each voxel consists of 8 vertices. A voxel vertex is represented by two different state variables: first, a value higher than or equal to the potential value indicates it is inside the object's surface; second, a value lower than the potential value indicates it is outside the object's surface. Therefore, a single vertex of a voxel has two possible states, and all 8 vertices of a voxel have 256 possible states. Secondly, based on the rotation and mapping invariance properties, the state of voxels is summarized into 15 basic configurations. That is, all 256 states of voxels can be obtained through rotation and symmetry transformations using these 15 basic configurations. Each voxel state contains several triangular facets. The specific position of the vertex of the triangular facet in the voxel is calculated by linear interpolation based on the value of the isosurface and the values ​​of the two vertices of the side it is on. Then, a lookup table is created for all 256 states of the voxels to be used when reconstructing the 3D image. The MarchingCubes algorithm finds the triangular facets among all voxels and combines them to form the final 3D visualization model of the brain.

4. The method for segmenting and training a three-dimensional visualization model of a glioma as described in claim 3, characterized in that, Step S200 includes the following steps: Step S201: Obtain image data under different modalities, downsample them to generate multiple feature images of different scales, and reduce the scale to half of the original scale each time. Step S202: Perform residual calculations on feature maps at different scales to obtain residual information. In the Laplace pyramid model structure, the feature map of the previous layer of each layer is upsampled to obtain a feature map with the same scale as the current layer. Then, the feature map of the current layer is subtracted from the upsampled feature map to obtain the residual information between the two feature maps. Step S203: Fuse the obtained feature maps and residual information of different scales of different modalities to generate a multimodal fused feature map: Feature maps and residual information of different scales are fused using a two-layer maximum fusion strategy and a two-layer local gradient energy strategy, and finally the maximum gradient fusion strategy is used for fusion.

5. The method for segmenting and training a three-dimensional visualization model of a glioma as described in claim 4, characterized in that, In step S203, in the maximum fusion strategy, the gray values ​​of the two images to be fused at the same position are compared, and the larger gray value is selected as the gray value of the new image at that position. All points in the image are traversed to obtain the fused image. In the local gradient energy strategy, the regional gradient energy of the five points surrounding each pixel in the two images (top, bottom, left, and right) is calculated. The regional gradient energies of the five pixels are summed, and the sums are compared. The gray value of the pixel with the larger energy is taken as the gray value of the pixel in the new image. The preceding steps are repeated until all pixels in the image are traversed. The formula for calculating the local gradient energy is as follows: In the formula, These represent the row index and column index of the pixel, respectively. Indicates position The value of the pixel at that location, Indicates position The magnitude of the local energy gradient at a pixel.

6. The method for segmenting and training a three-dimensional visualization model of a glioma as described in claim 3, characterized in that, Step S300 includes the following steps: Step S301: Perform data preprocessing on the multimodal fusion map generated in step S200, including standardization and background enhancement; Step S302: Perform data augmentation using various methods on the data processed in step S301 to improve the data's breadth. Step S303: Input the enhanced data into a U-Net network containing an encoder and a decoder to generate a segmentation mask.

Images