A hyperspectral brain tumor surgery guidance segmentation method based on a prototype attention network

Abstract

The application discloses a hyperspectral brain tumor surgery guidance segmentation method based on a prototype attention network. The method is realized through the following steps: collecting hyperspectral image data of brain tumors through an intraoperative hyperspectral imaging platform, and constructing a standard data set for training; designing and constructing a hyperspectral image segmentation model based on a prototype attention network, focusing on important features through self-adaptation, and enhancing the difference between tumors and normal tissues; training the model using the standard data set, optimizing network parameters, thereby improving segmentation accuracy and real-time performance; deploying the trained hyperspectral image segmentation model to the hyperspectral image acquisition platform, processing the collected brain tumor hyperspectral images in real time, outputting the segmentation results and generating surgery guidance information. The application has the advantages of effectively improving the accuracy and efficiency of intraoperative real-time segmentation of brain tumors, and providing accurate guidance for surgery.

Description

Technical Field

[0001] This invention relates to the fields of medical image processing and artificial intelligence technology, and in particular to a hyperspectral brain tumor surgical guidance segmentation method based on a prototype attention network. Background Technology

[0002] Surgical resection of brain tumors is crucial for patient treatment outcomes and survival. However, accurately identifying and removing tumor areas during surgery is extremely challenging due to the blurred boundaries between brain tumors and normal brain tissue. In traditional intraoperative detection techniques, surgeons primarily rely on intraoperative pathology, imaging data, and surgical navigation systems. While these methods offer some assistance, they still have several limitations.

[0003] Current intraoperative pathological examination relies on rapid frozen section technology, allowing pathologists to obtain pathological results from tissue samples within minutes, thus helping surgeons determine the nature and boundaries of tumors. However, this method is affected by the quality of sample processing and depends on the subjective judgment of pathologists, posing a high risk of error. Furthermore, while surgical navigation systems based on MRI, CT, etc., can provide three-dimensional spatial positioning, they struggle to provide sufficient real-time detail, particularly in distinguishing subtle tissue differences.

[0004] In imaging technology, hyperspectral imaging, with its unique high spectral resolution, adds spectral information to spatial resolution, facilitating the detection and analysis of spectral differences between different tissues. The advantage of hyperspectral imaging lies in its ability to provide spectral data for each pixel, enabling better identification of differences between tumors and normal tissues by distinguishing spectral characteristics, thus becoming an emerging tool in the field of medical imaging. However, the multidimensionality and complexity of hyperspectral data present challenges for real-time processing, especially in near-real-time intraoperative detection scenarios, requiring fast and efficient algorithms to extract relevant information.

[0005] Therefore, deep learning technology has been introduced into the field of hyperspectral image processing, achieving significant progress in brain tumor detection. Traditional machine learning methods, such as Support Vector Machines (SVM) and K-Nearest Neighbors (KNN), extract and classify spectral features for brain tumor detection. However, due to their reliance on manual feature extraction, they struggle to fully utilize the multidimensional information of hyperspectral images, resulting in limited processing effectiveness. In recent years, deep learning methods such as Convolutional Neural Networks (CNNs) have demonstrated superior performance in hyperspectral image segmentation and recognition thanks to their automatic feature extraction and classification capabilities.

[0006] Furthermore, the introduction of attention mechanisms has further enhanced the application of deep learning in medical imaging. Attention mechanisms can dynamically adjust feature weights, allowing the network to focus on important regions and reduce background noise interference. In hyperspectral image processing tasks, such methods have been shown to improve segmentation accuracy and target region recognition capabilities. However, these existing methods still face certain limitations in practical applications, such as high computational complexity, insufficient real-time performance, and inadequate utilization of information.

[0007] In summary, existing intraoperative detection methods for brain tumors, while achieving high accuracy, struggle to effectively achieve real-time performance and robustness, and fail to fully exploit the multidimensional information within hyperspectral images. Therefore, developing a more efficient segmentation method for real-time intraoperative detection of hyperspectral images is crucial for improving the accuracy of intraoperative brain tumor detection. Summary of the Invention

[0008] This invention addresses the shortcomings of existing technologies by providing a hyperspectral brain tumor surgical guidance segmentation method based on a prototype attention network. It is particularly suitable for real-time segmentation of brain tumor tissue using hyperspectral images during surgery, assisting surgeons in more accurately identifying and removing tumors while minimizing damage to healthy tissues.

[0009] To achieve the above-mentioned objectives, the technical solution adopted by the present invention is as follows:

[0010] A hyperspectral brain tumor surgically guided segmentation method based on a prototype attention network includes the following steps:

[0011] Step 1: Acquire hyperspectral image data of brain tumors using an intraoperative hyperspectral imaging platform and construct a standard dataset for training;

[0012] Step 2: Design and build a hyperspectral image segmentation model based on a prototype attention network to enhance the difference between tumors and normal tissues by adaptively focusing on important features;

[0013] Step 3: Train the hyperspectral image segmentation model using a standard dataset and optimize the network parameters to improve segmentation accuracy and real-time performance;

[0014] Step 4: Deploy the trained hyperspectral image segmentation model on the hyperspectral image acquisition platform to process the acquired hyperspectral images of brain tumors in real time, output the segmentation results, and generate surgical guidance information.

[0015] Furthermore, step 1 includes the following sub-steps:

[0016] Step 1.1: Acquire high-quality intraoperative hyperspectral images using a hyperspectral acquisition platform, and simultaneously obtain corresponding tissue samples and pathological "gold standard" diagnostic results to ensure the accuracy of dataset annotation;

[0017] Step 1.2: Use the pathological "gold standard" diagnostic results to annotate the hyperspectral images, specifically the isolated brain tumor tissue, to determine the different types of tumors and their boundaries;

[0018] Step 1.3: Divide the hyperspectral data into training data, validation data, and test data to ensure the comprehensiveness of the evaluation;

[0019] Step 1.4: Organize the hyperspectral image data and the corresponding category data, and perform standardization processing to form a standard training dataset for brain tumor segmentation.

[0020] Furthermore, step 2 includes the following sub-steps:

[0021] Step 2.1: A hyperspectral image segmentation model based on a prototype attention network, combining prototype learning and attention mechanisms, enhances the representation ability of tumor features by aggregating similar features;

[0022] Step 2.2: Based on the training dataset for brain tumor segmentation, set the input and output sizes of the hyperspectral image segmentation model and configure the required parameter values, including the learning rate, optimizer function, and loss function.

[0023] Furthermore, step 3 includes the following sub-steps:

[0024] Step 3.1: Set the number of training rounds for the hyperspectral image segmentation model to ensure that the model learns sufficiently and the loss converges;

[0025] Step 3.1: Input the hyperspectral image training dataset into the hyperspectral image segmentation model constructed in Step 2 for training. The hyperspectral image segmentation model continuously adjusts its parameters during training to improve the segmentation effect.

[0026] Step 3.1: After training is complete, save the optimized network parameters and weights file for later use.

[0027] Furthermore, step 4 includes the following sub-steps:

[0028] Step 4.1: Deploy the trained hyperspectral image segmentation model on a standard hyperspectral acquisition platform to achieve near real-time segmentation;

[0029] Step 4.2: Acquire the intraoperative hyperspectral image of the data to be segmented, ensuring that the acquisition method and instrument are consistent with the training data;

[0030] Step 4.3: Process the hyperspectral images to be classified to meet the network input requirements;

[0031] Step 4.4: Input the processed hyperspectral image into the trained prototype attention network;

[0032] Step 4.5: Obtain the segmentation results of the entire hyperspectral image, identify the type of brain tumor, and generate a segmentation report to provide intraoperative decision support for doctors.

[0033] Furthermore, the prototype attention network model combines prototype learning and attention mechanisms, and by focusing on the key features of brain tumors, it improves the difference between tumor regions and normal tissues, thereby enhancing segmentation accuracy.

[0034] Furthermore, the prototype attention network employs a multi-head prototype attention module to enhance the diversity of feature extraction, alleviate overfitting, and improve the robustness of the model.

[0035] Furthermore, the hyperspectral image data includes spectral information of multiple bands and is standardized to meet the training requirements of deep learning models.

[0036] Compared with the prior art, the advantages of the present invention are as follows:

[0037] 1. This invention, by combining prototype learning and attention mechanisms, can effectively focus on the tumor region and enhance the difference between tumors and normal tissues. This enables the segmentation network to exhibit high accuracy in identifying brain tumor boundaries, especially for complex tumor boundaries and regions with subtle differences, where the segmentation results show higher accuracy. Compared with traditional methods, this invention significantly improves the accuracy of tumor region segmentation, reduces background noise interference, and enhances the detail representation of the segmentation.

[0038] 2. This invention, through its efficient network architecture design, can rapidly process intraoperative hyperspectral images and provide tumor segmentation results in real time. This technology is suitable for dynamic and rapidly changing surgical scenarios, providing surgeons with timely segmentation information to help them make more accurate decisions during surgery, thereby improving surgical safety and efficiency.

[0039] 3. By introducing a spatially aware feedforward network module and employing multi-scale depthwise separable convolution, the model can effectively handle features at different spatial scales in hyperspectral images. This module enhances the network's robustness, enabling the model to maintain stable performance when processing complex spatial features and large-scale data, unaffected by image noise and different anatomical features.

[0040] 4. The prototype attention network designed in this invention can fully utilize the multi-band information of hyperspectral images to extract richer features from complex brain tumor images. This technology can not only process single hyperspectral images, but also adapt to the fusion of multimodal data, providing more comprehensive information support for tumor segmentation and improving the accuracy and reliability of segmentation results.

[0041] 5. Traditional brain tumor segmentation methods often rely on manual intervention or threshold adjustments, while this invention employs a deep learning-based automatic segmentation method, significantly reducing the need for manual intervention. The network can automatically complete the high-precision segmentation process, reducing the complexity of manual operations and minimizing the possibility of human error.

[0042] 6. This invention provides an efficient tool for the precise segmentation and localization of brain tumors, helping surgeons to perform more accurate tumor resections during surgery. By accurately identifying tumor boundaries, it can support the development of personalized treatment plans, improving patient outcomes and survival rates.

[0043] 7. A prototype attention mechanism is employed to deeply fuse the spatial and spectral information of hyperspectral images, significantly enhancing the ability to distinguish tumor tissue from normal tissue. Particularly at the tumor's periphery, the network can accurately identify minute lesion areas, ensuring complete tumor resection and avoiding residual tumor. Attached Figure Description

[0044] Figure 1 Flowchart for constructing the dataset in an embodiment of the present invention;

[0045] Figure 2 This is a flowchart illustrating the construction process of the prototype attention network in an embodiment of the present invention.

[0046] Figure 3 This is a flowchart illustrating the training, testing, and deployment process of the network model in an embodiment of the present invention.

[0047] Figure 4 This is a diagram of the hyperspectral brain tumor segmentation network structure based on the prototype attention network in an embodiment of the present invention.

[0048] Figure 5 This is a structural diagram of the prototype attention module of an embodiment of the present invention;

[0049] Figure 6 This is a diagram of the feedforward network module for spatial perception in an embodiment of the present invention;

[0050] Figure 7 The figure shows an example of experimental results from an embodiment of the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and examples.

[0052] like Figure 4 As shown, this invention provides a hyperspectral brain tumor surgically guided segmentation method based on a prototype attention network, comprising the following steps:

[0053] Step 1: Acquire hyperspectral image data of brain tumors using an intraoperative hyperspectral imaging platform and construct a standard dataset for training;

[0054] Step 2: Design and build a hyperspectral image segmentation model based on a prototype attention network. This model enhances the difference between tumors and normal tissues by adaptively focusing on important features.

[0055] Step 3: Train the model using the hyperspectral brain tumor training dataset and optimize the network parameters to improve segmentation accuracy and real-time performance;

[0056] Step 4: Deploy the trained model on a hyperspectral image acquisition platform to process the acquired hyperspectral images of brain tumors in real time, output segmentation results, and generate surgical guidance information.

[0057] Step 1 includes the following sub-steps:

[0058] Step 1.1: Acquire high-quality intraoperative hyperspectral images using a hyperspectral acquisition platform, and simultaneously obtain corresponding tissue samples and pathological "gold standard" diagnostic results to ensure the accuracy of dataset annotation;

[0059] Step 1.2: Use the pathological "gold standard" diagnostic results to annotate the hyperspectral images, specifically the isolated brain tumor tissue, to determine the different types of tumors and their boundaries;

[0060] Step 1.3: To verify the effectiveness of the algorithm and the generalization ability of the model, the standard dataset is randomly divided, with 80% of the hyperspectral data used as training data, 10% as validation data, and 10% as test data, to ensure the comprehensiveness of the evaluation.

[0061] Step 1.4: Organize the hyperspectral image data and the corresponding category data, and perform standardization processing to form a standard training dataset for brain tumor segmentation.

[0062] Step 2 includes the following sub-steps:

[0063] Step 2.1: Design a prototype attention network model, combining prototype learning and attention mechanisms, to enhance the representation ability of tumor features by aggregating similar features;

[0064] Step 2.2: Based on the training dataset for brain tumor segmentation, set the input and output sizes of the network and configure the required parameter values, including the learning rate, optimizer function, and loss function.

[0065] Step 3 includes the following sub-steps:

[0066] Step 3.1: Set the number of training rounds for the prototype attention network to ensure that the model learns sufficiently and the loss converges;

[0067] Step 3.2: Input the hyperspectral image training dataset into the network constructed in Step 2 for training. During the training process, the network continuously adjusts its parameters to improve the segmentation effect.

[0068] Step 3.3: After training is complete, save the optimized network parameters and weights file for later use.

[0069] Step 4 includes the following sub-steps:

[0070] Step 4.1: Deploy on the hyperspectral standard acquisition platform to complete near real-time segmentation;

[0071] Step 4.2: Acquire the intraoperative hyperspectral image of the data to be segmented, ensuring that the acquisition method and instrument are consistent with the training data;

[0072] Step 4.3: Process the hyperspectral images to be classified to meet the network input requirements;

[0073] Step 4.4: Input the processed hyperspectral image into the trained prototype attention network;

[0074] Step 4.5: Obtain the segmentation results of the entire hyperspectral image, identify the type of brain tumor, and generate a segmentation report to provide intraoperative decision support for doctors.

[0075] The following is a sample procedure based on the hyperspectral brain tumor surgical-guided segmentation method described above:

[0076] 1. Data preparation and preprocessing:

[0077] like Figure 1 As shown, hyperspectral image data acquired during brain tumor surgery guidance underwent preprocessing. To ensure data consistency, the spectral data was calibrated using a standard reflector. The spectral range was 400-1000 nm, encompassing 128 bands, with a spatial resolution of 512×512. After processing, the data was standardized and an image dataset suitable for deep learning training was generated.

[0078] 2. Constructing a hyperspectral brain tumor segmentation network based on prototype attention:

[0079] like Figure 2As shown, a hyperspectral brain tumor segmentation network model based on prototype attention is constructed. The entire network is divided into five modules:

[0080] Module 1: Tile Embedding Module

[0081] The patch embedding module is used to extract the spatial-spectral joint embedding features of hyperspectral patches. The convolution kernel size is 64×4×4, the stride is 4, and the resulting feature map is 64×128×128.

[0082] X = conv(I in )

[0083] Module 2: Prototype Attention Module

[0084] like Figure 5 As shown, a prototype attention module is constructed to capture spatial spectral information by modeling the feature relationships between pixels in a hyperspectral image. A learnable prototype Z∈R is introduced. n×d , where n represents the number of prototypes. The key of the prototype K c ∈R N×d Sum V c ∈R N×d They are respectively

[0085] K c =softmax(ZX) T ,dim=0)K

[0086] V c =softmax(ZX) T ,dim=0)V

[0087] Wherein, the expression softmax(ZX) T (dim=0) Similarity scores for all features in the hyperspectral image were calculated. This process is similar to the soft K-means algorithm used to update the prototype. The prototype attention results are then as follows:

[0088]

[0089] The above describes the computation process for a single prototype attention. In the self-attention module, multi-head self-attention exhibits superior performance compared to single-head self-attention by enabling each label to capture information beyond its single context, thus effectively mitigating overfitting. This concept aligns with the principle of "don't concentrate all resources in one region," emphasizing the importance of diverse feature extraction to generate more robust representations. Similarly, we introduce multi-head prototype attention in our method, with h heads. Therefore, Q, K, and V are typically partitioned along the channel dimension, with each part located in R... N×(d / h)To further enhance the diversity of prototype attention, we partition Z along the first dimension, obtaining... Next, we apply Q to the i-th head (1≤i≤h). i K i V i The above prototype attention calculation is applied. The outputs of all heads are then concatenated to generate the final output.

[0090] Module 3: Spatial Awareness Feedforward Network Module

[0091] like Figure 6 As shown, the spatially aware feedforward network module comprises multi-scale depthwise separable convolutions and a feedforward network model. First, the input features are segmented into channels, and then depthwise separable convolutions with different kernel sizes are used for computation.

[0092] I1 = depthconv(I i1 )

[0093] I2 = depthconv(I i2 )

[0094] I3 = depthconv(I i3 )

[0095] Next, the different outputs are combined into I = (I1, I2, I3) and input into the feedforward network:

[0096] out = FC2(ReLU(FC1(I)))

[0097] Here, FC stands for Connective Layer and ReLU stands for Rectified Linear Activation Function. The model's flexibility is enhanced by placing depthwise separable convolutional layers at the beginning and using different kernel sizes. Larger kernels are used for heads with more prototypes, while smaller kernels are used for heads with fewer prototypes, balancing local and global feature information.

[0098] In a hyperspectral brain tumor segmentation network based on prototype attention, the second and third modules are connected to form a new module, and this module is repeated L times.

[0099] Module 4: Classification Module

[0100] Based on the features proposed by the multi-layer prototype attention and spatial awareness feedforward network model, the final segmentation map output is obtained by utilizing two fully connected layers, namely the projection layer and the linear classification layer. Figure 7 ):

[0101] Class = FC4((FC3(I)))

[0102] Model training and deployment:

[0103] like Figure 3 As shown, the training steps include the following aspects:

[0104] First, a standardized hyperspectral image dataset is input into the model. The model's performance is improved through loss function optimization, gradient calculation, and iterative optimization. Finally, the model parameters are saved. Then, the trained model is validated on a test dataset. Finally, the model is deployed to a clinical system to guide brain tumor resection surgery in real time.

[0105] The methods described above according to the invention can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code originally stored on a remote recording medium or a non-transitory machine-readable medium and subsequently stored on a local recording medium, downloaded via a network. Thus, the methods described herein can be stored as software processing on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., RAM, ROM, flash memory, etc.) capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the hyperspectral brain tumor surgically guided segmentation method described herein. Furthermore, when a general-purpose computer accesses the code used to implement the processing shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for performing the processing shown herein.

[0106] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the implementation methods of the present invention, and should be understood that the scope of protection of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of the present invention.

Claims

1. A hyperspectral brain tumor segmentation method based on a prototype attention network, characterized in that, Includes the following steps: Step 1: Acquire hyperspectral image data of brain tumors using an intraoperative hyperspectral imaging platform and construct a standard dataset for training; Step 2: Design and build a hyperspectral image segmentation model based on a prototype attention network to enhance the difference between tumors and normal tissues by adaptively focusing on important features; The hyperspectral image segmentation model includes, in sequence: a patch embedding module, a prototype attention module, a spatial awareness feedforward network module, and a classification module; The tile embedding module is used to extract spatial-spectral joint embedding features from hyperspectral images through convolution operations; The prototype attention module introduces a learnable prototype, calculates the similarity between image features and the prototype to generate attention weights, thereby aggregating similar features. The prototype attention module employs a multi-head prototype attention mechanism, implemented as follows: Divide the learnable prototype Z into the first dimension. , ... , where h is the number of heads; At the same time, the query Q, key K, and value V are divided into h heads in the channel dimension; For the i-th head, 1≤i≤h, the query will be performed. ,key ,value Applied to prototype attention calculation; Finally, the outputs of all h heads are concatenated to generate the final prototype attention output; The spatial perception feedforward network module includes: Multi-scale depthwise separable convolutional units are used to divide input features along channels and perform computations using depthwise separable convolutions with different kernel sizes. A feedforward network is used to transform the merged features output by the multi-scale depth-separable convolutional units; Based on the features proposed by the multi-layer prototype attention and spatial awareness feedforward network model, the classification module uses two fully connected layers, namely the projection layer and the linear classification layer, to obtain the final segmentation map output. Step 3: Train the hyperspectral image segmentation model using a standard dataset and optimize the network parameters to improve segmentation accuracy and real-time performance; Step 4: Deploy the trained hyperspectral image segmentation model on the hyperspectral image acquisition platform to process the acquired hyperspectral images of brain tumors in real time and output the segmentation results.

2. The hyperspectral brain tumor segmentation method according to claim 1, characterized in that: Step 1 includes the following sub-steps: Step 1.1: Acquire high-quality intraoperative hyperspectral images using a hyperspectral acquisition platform, and simultaneously obtain corresponding tissue samples and pathological "gold standard" diagnostic results to ensure the accuracy of dataset annotation; Step 1.2: Use the pathological "gold standard" diagnostic results to annotate the hyperspectral images, specifically the isolated brain tumor tissue, to determine the different types of tumors and their boundaries; Step 1.3: Divide the hyperspectral data into training data, validation data, and test data to ensure the comprehensiveness of the evaluation; Step 1.4: Organize the hyperspectral image data and the corresponding category data, and perform standardization processing to form a standard training dataset for brain tumor segmentation.

3. The hyperspectral brain tumor segmentation method according to claim 1, characterized in that: Step 2 includes the following sub-steps: Step 2.1: A hyperspectral image segmentation model based on a prototype attention network, combining prototype learning and attention mechanisms, enhances the representation ability of tumor features by aggregating similar features; Step 2.2: Based on the training dataset for brain tumor segmentation, set the input and output sizes of the hyperspectral image segmentation model and configure the required parameter values, including the learning rate, optimizer function, and loss function.

4. The hyperspectral brain tumor segmentation method according to claim 1, characterized in that: Step 3 includes the following sub-steps: Step 3.1: Set the number of training rounds for the hyperspectral image segmentation model to ensure that the model learns sufficiently and the loss converges; Step 3.1: Input the hyperspectral image training dataset into the hyperspectral image segmentation model constructed in Step 2 for training. The hyperspectral image segmentation model continuously adjusts its parameters during training to improve the segmentation effect. Step 3.1: After training is complete, save the optimized network parameters and weights file for later use.

5. The hyperspectral brain tumor segmentation method according to claim 1, characterized in that: Step 4 includes the following sub-steps: Step 4.1: Deploy the trained hyperspectral image segmentation model on a standard hyperspectral acquisition platform to achieve near real-time segmentation; Step 4.2: Acquire the intraoperative hyperspectral image of the data to be segmented, ensuring that the acquisition method and instrument are consistent with the training data; Step 4.3: Process the hyperspectral images to be classified to meet the network input requirements; Step 4.4: Input the processed hyperspectral image into the trained prototype attention network; Step 4.5: Obtain the segmentation results of the entire hyperspectral image, identify the brain tumor type, and generate a segmentation report.

6. The hyperspectral brain tumor segmentation method according to any one of claims 1 to 5, characterized in that: The prototype attention network model combines prototype learning and attention mechanisms. By focusing on key features of brain tumors, it improves the difference between tumor regions and normal tissues, thereby enhancing segmentation accuracy.

7. The hyperspectral brain tumor segmentation method according to claim 6, characterized in that: The prototype attention network employs a multi-head prototype attention module to enhance the diversity of feature extraction, alleviate overfitting, and improve the robustness of the model.

8. The hyperspectral brain tumor segmentation method according to claim 1, characterized in that: The hyperspectral image data includes spectral information from multiple bands and is processed using standardization to meet the training requirements of deep learning models.

Images