Hyperspectral brain tumor operation guide segmentation method based on prototype attention network

Abstract

The invention discloses a hyperspectral brain tumor operation guide segmentation method based on a prototype attention network. The method is realized through the following steps: acquiring hyperspectral image data of brain tumors through an intraoperative hyperspectral imaging platform, and constructing a standard data set for training; a hyperspectral image segmentation model based on a prototype attention network is designed and constructed, and the difference between a tumor and a normal tissue is enhanced through adaptive focusing of important features; a standard data set is used for training the model, network parameters are optimized, and therefore segmentation precision and real-time performance are improved; and deploying the trained hyperspectral image segmentation model to a hyperspectral image acquisition platform, processing the acquired brain tumor hyperspectral image in real time, outputting a segmentation result and generating operation guide information. The method has the advantages that the accuracy and efficiency of real-time segmentation in the brain tumor operation are effectively improved, and accurate guidance is provided for the operation.

Description

Technical Field

[0001] The present invention relates to the field of medical image processing and artificial intelligence technology, and in particular to a hyperspectral brain tumor surgery guided segmentation method based on a prototype attention network. Background Art

[0002] Surgical resection of brain tumors is crucial to the treatment effect and survival prognosis of patients. However, due to the blurred boundary between brain tumors and normal brain tissue, it is extremely challenging to accurately identify and remove the tumor area during surgery. In traditional intraoperative detection techniques, surgeons mainly rely on intraoperative pathology, imaging data and surgical navigation systems. Although these methods have certain auxiliary effects, they still have many limitations.

[0003] At present, intraoperative pathology detection relies on rapid frozen section technology, which allows pathologists to obtain pathological results of tissue samples within a few minutes, thereby helping surgeons determine the nature and boundaries of tumors. However, this method is affected by the quality of sample processing and relies on the subjective judgment of pathologists, which poses a high risk of error. In addition, although surgical navigation systems based on MRI, CT, etc. can provide three-dimensional spatial positioning, they are difficult to provide sufficient real-time details, especially in distinguishing subtle differences in tissues.

[0004] In imaging technology, hyperspectral imaging technology, with its unique high spectral resolution, can add spectral information on the basis of spatial resolution, which helps to detect and analyze the spectral differences of different tissues. The advantage of hyperspectral imaging is that it can provide spectral data for each pixel. By distinguishing the spectral characteristics of tissues, it can better identify the difference between tumors and normal tissues, becoming an emerging tool in the field of medical imaging. However, the multidimensionality and complexity of hyperspectral data bring challenges to real-time processing, especially in near-real-time detection scenarios during surgery, which requires fast and efficient algorithms to extract effective information.

[0005] To this end, deep learning technology has been introduced into the field of hyperspectral image processing and has made significant progress in brain tumor detection. Traditional machine learning methods, such as support vector machines (SVM) and K-nearest neighbors (KNN), are applied to brain tumor detection by extracting and classifying spectral features. However, due to reliance on manual feature extraction, it is difficult to fully utilize the multi-dimensional information of hyperspectral images, and the processing effect is limited. In recent years, deep learning methods such as convolutional neural networks (CNNs) have demonstrated better performance in the segmentation and recognition of hyperspectral images by virtue of their automatic feature extraction and classification capabilities.

[0006] In addition, the introduction of the attention mechanism has further improved the application effect of deep learning in medical imaging. The attention mechanism can dynamically adjust the weights of features so that the network can focus on important areas and reduce the interference of background noise. In hyperspectral image processing tasks, such methods have been proven to improve segmentation accuracy and the ability to identify target areas. However, these existing methods still face certain limitations in practical applications, such as high computational complexity, lack of real-time performance, and insufficient use of information.

[0007] In summary, the existing intraoperative brain tumor detection methods are difficult to effectively achieve real-time and robustness while meeting high accuracy, and fail to fully exploit the multidimensional information in hyperspectral images. Therefore, developing a more efficient segmentation method to achieve real-time detection of hyperspectral images is of great significance for improving the accuracy of intraoperative brain tumor detection. Summary of the invention

[0008] In view of the defects of the prior art, the present invention provides a hyperspectral brain tumor surgical guidance segmentation method based on a prototype attention network. It is particularly suitable for real-time segmentation of brain tumor tissues through hyperspectral images during surgery to assist doctors in more accurately identifying and removing tumors and reducing damage to healthy tissues.

[0009] In order to achieve the above invention object, the technical solution adopted by the present invention is as follows:

[0010] A hyperspectral brain tumor surgery-guided segmentation method based on a prototype attention network, comprising the following steps:

[0011] Step 1: Collect hyperspectral image data of brain tumors through an intraoperative hyperspectral imaging platform and construct a standard dataset for training;

[0012] Step 2: Design and build a hyperspectral image segmentation model based on a prototype attention network to enhance the difference between tumors and normal tissues by adaptively focusing on important features;

[0013] Step 3: Use standard datasets to train the hyperspectral image segmentation model and optimize network parameters to improve segmentation accuracy and real-time performance;

[0014] Step 4: Deploy the trained hyperspectral image segmentation model on the hyperspectral image acquisition platform, process the acquired brain tumor hyperspectral images in real time, output the segmentation results and generate surgical guidance information.

[0015] Furthermore, step 1 includes the following sub-steps:

[0016] Step 1.1: Collect high-quality intraoperative hyperspectral images through the hyperspectral acquisition platform, and simultaneously obtain the corresponding tissue samples and pathological "gold standard" diagnosis results to ensure the accuracy of the dataset annotation;

[0017] Step 1.2: Use the pathological "gold standard" diagnosis results to annotate the hyperspectral image and annotate the ex vivo brain tumor tissue to determine the different types of tumors and their boundaries;

[0018] Step 1.3: Divide the hyperspectral data into training data, validation data, and test data to ensure the comprehensiveness of the evaluation;

[0019] Step 1.4: Arrange and standardize the hyperspectral image data and the corresponding category data to form a standard training dataset for brain tumor segmentation.

[0020] Furthermore, step 2 includes the following sub-steps:

[0021] Step 2.1: A hyperspectral image segmentation model based on a prototype attention network, combining prototype learning and attention mechanism, enhances the representation ability of tumor features by aggregating similar features;

[0022] Step 2.2: According to the training dataset for brain tumor segmentation, set the input and output sizes of the hyperspectral image segmentation model and configure the required parameter values, including the learning rate, optimizer function, and loss function.

[0023] Furthermore, step 3 includes the following sub-steps:

[0024] Step 3.1: Set the number of training rounds for the hyperspectral image segmentation model to ensure that the model is fully learned and the loss converges;

[0025] Step 3.1: Input the hyperspectral image training data set into the hyperspectral image segmentation model constructed in step 2 for training. The hyperspectral image segmentation model continuously adjusts parameters during the training process to improve the segmentation effect.

[0026] Step 3.1: After training is completed, save the optimized network parameters and weight files for subsequent use.

[0027] Furthermore, step 4 includes the following sub-steps:

[0028] Step 4.1: Deploy the trained hyperspectral image segmentation model on the hyperspectral standard acquisition platform to complete near real-time segmentation;

[0029] Step 4.2: Obtain the intraoperative hyperspectral image to be segmented and ensure that the acquisition method and instrument are consistent with the training data;

[0030] Step 4.3: Process the hyperspectral image to be classified to meet the network input requirements;

[0031] Step 4.4: Input the processed hyperspectral image into the trained prototype attention network;

[0032] Step 4.5: Obtain the segmentation results of the entire hyperspectral image, identify the type of brain tumor, and generate a segmentation report to provide intraoperative decision support for doctors.

[0033] Furthermore, the prototype attention network model combines prototype learning and attention mechanism, and improves the difference between tumor area and normal tissue by focusing on the key features of brain tumors, thereby improving segmentation accuracy.

[0034] Furthermore, the prototype attention network adopts a multi-head prototype attention module to enhance the diversity of feature extraction, alleviate overfitting and improve the robustness of the model.

[0035] Furthermore, the hyperspectral image data includes spectral information of multiple bands and is standardized to meet the training requirements of the deep learning model.

[0036] Compared with the prior art, the advantages of the present invention are:

[0037] 1. By combining prototype learning and attention mechanism, the present invention can effectively focus on the tumor area and enhance the difference between tumor and normal tissue. This enables the segmentation network to show high accuracy in identifying brain tumor boundaries, especially for complex tumor boundaries and areas with slight differences, the segmentation results have higher accuracy. Compared with traditional methods, the present invention significantly improves the accuracy of tumor area segmentation, reduces the interference of background noise, and improves the detail performance of segmentation.

[0038] 2. The present invention can quickly process hyperspectral images collected during surgery and provide tumor segmentation results in real time through efficient network architecture design. This technology is suitable for dynamic and rapidly changing surgical scenarios, and can provide surgeons with timely segmentation information, helping doctors make more accurate decisions during surgery and improving the safety and efficiency of surgery.

[0039] 3. By introducing a spatially aware feedforward network module and using multi-scale deep separable convolution, the model can effectively process features of different spatial scales in hyperspectral images. This module enhances the robustness of the network, allowing the model to maintain stable performance when processing data with complex spatial features and large-scale data, without being disturbed by image noise and different anatomical features.

[0040] 4. The prototype attention network designed by the present invention can make full use of the multi-band information of hyperspectral images and extract richer features from complex brain tumor images. This technology can not only process single hyperspectral images, but also adapt to the fusion of multimodal data, provide more comprehensive information support for tumor segmentation, and improve the accuracy and reliability of segmentation results.

[0041] 5. Traditional brain tumor segmentation methods often rely on manual intervention or manual threshold adjustment, while the present invention adopts an automatic segmentation method based on deep learning, which greatly reduces the need for manual intervention. The network can automatically complete the high-precision segmentation process, reducing the complexity of manual operation and the possibility of human error.

[0042] 6. The present invention provides an efficient tool for accurate segmentation and positioning of brain tumors, which helps doctors to perform more accurate tumor resection during surgery. By accurately identifying tumor boundaries, it can provide support for the formulation of personalized treatment plans and improve the treatment effect and survival rate of patients.

[0043] 7. The prototype attention mechanism is used to deeply fuse the spatial and spectral information of the hyperspectral image, significantly enhancing the ability to distinguish tumor tissue from normal tissue. Especially in the edge part around the tumor, the network can accurately identify the tiny lesion area to ensure the complete removal of the tumor and avoid residual. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A flow chart is constructed for a data set of an embodiment of the present invention;

[0045] Figure 2 A flowchart for constructing a prototype attention network for an embodiment of the present invention;

[0046] Figure 3 A flowchart of training, testing and deployment of a network model according to an embodiment of the present invention;

[0047] Figure 4 This is a diagram showing the structure of a hyperspectral brain tumor segmentation network based on a prototype attention network according to an embodiment of the present invention;

[0048] Figure 5 is a structural diagram of a prototype attention module of an embodiment of the present invention;

[0049] Figure 6 A module diagram of a feedforward network for spatial perception according to an embodiment of the present invention;

[0050] Figure 7 2 is an example diagram of experimental results of an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and examples.

[0052] like Figure 4 As shown, the present invention provides a hyperspectral brain tumor surgery guided segmentation method based on a prototype attention network, comprising the following steps:

[0053] Step 1: Obtain hyperspectral image data of brain tumors through an intraoperative hyperspectral imaging platform and construct a standard dataset for training;

[0054] Step 2: Design and build a hyperspectral image segmentation model based on a prototype attention network, which enhances the difference between tumors and normal tissues by adaptively focusing on important features;

[0055] Step 3: Train the model using the hyperspectral brain tumor training dataset and optimize the network parameters to improve segmentation accuracy and real-time performance;

[0056] Step 4: Deploy the trained model on the hyperspectral image acquisition platform, process the acquired brain tumor hyperspectral images in real time, output the segmentation results and generate surgical guidance information.

[0057] The step 1 includes the following sub-steps:

[0058] Step 1.1: Collect high-quality intraoperative hyperspectral images through the hyperspectral acquisition platform, and simultaneously obtain the corresponding tissue samples and pathological "gold standard" diagnosis results to ensure the accuracy of the dataset annotation;

[0059] Step 1.2: Use the pathological "gold standard" diagnosis results to annotate the hyperspectral image and annotate the ex vivo brain tumor tissue to determine the different types of tumors and their boundaries;

[0060] Step 1.3: To verify the effectiveness of the algorithm and the generalization ability of the model, the standard data set is randomly divided, with 80% of the hyperspectral data as training data, 10% as validation data, and 10% as test data to ensure the comprehensiveness of the evaluation;

[0061] Step 1.4: Arrange and standardize the hyperspectral image data and the corresponding category data to form a standard training dataset for brain tumor segmentation.

[0062] The step 2 includes the following sub-steps:

[0063] Step 2.1: Design a prototype attention network model, combining prototype learning and attention mechanism to enhance the representation ability of tumor features by aggregating similar features;

[0064] Step 2.2: According to the training dataset for brain tumor segmentation, set the input and output size of the network and configure the required parameter values, including learning rate, optimizer function, and loss function.

[0065] The step 3 includes the following sub-steps:

[0066] Step 3.1: Set the number of training rounds for the prototype attention network to ensure that the model fully learns and the loss converges;

[0067] Step 3.2: Input the hyperspectral image training data set into the network constructed in step 2 for training. During the training process, the network continuously adjusts parameters to improve the segmentation effect.

[0068] Step 3.3: After training is completed, save the optimized network parameters and weight files for subsequent use.

[0069] The step 4 includes the following sub-steps:

[0070] Step 4.1: Deploy on a standard hyperspectral acquisition platform to complete near real-time segmentation;

[0071] Step 4.2: Obtain the intraoperative hyperspectral image to be segmented and ensure that the acquisition method and instrument are consistent with the training data;

[0072] Step 4.3: Process the hyperspectral image to be classified to meet the network input requirements;

[0073] Step 4.4: Input the processed hyperspectral image into the trained prototype attention network;

[0074] Step 4.5: Obtain the segmentation results of the entire hyperspectral image, identify the type of brain tumor, and generate a segmentation report to provide intraoperative decision support for doctors.

[0075] The following is an example process based on the above-mentioned hyperspectral brain tumor surgery guided segmentation method:

[0076] 1. Data preparation and preprocessing:

[0077] like Figure 1 As shown in the figure, the hyperspectral image data collected during the guided brain tumor surgery was preprocessed. To ensure the consistency of the data, the spectral data was corrected using a standard reflector. The spectral range is 400-1000nm, contains 128 bands, and the spatial resolution is 512×512. After processing, the data is standardized and an image dataset suitable for deep learning training is generated.

[0078] 2. Constructing a prototype attention-based hyperspectral brain tumor segmentation network:

[0079] like Figure 2As shown in the figure, a prototype attention-based hyperspectral brain tumor segmentation network model is built. The entire network is divided into five modules:

[0080] Module 1: Tile Embedding Module

[0081] The tile embedding module is used to extract the spatial-spectral joint embedding features of the hyperspectral tiles. The convolution kernel size is 64×4×4 and the stride is 4, resulting in a 64×128×128 feature map:

[0082] X=conv(I in )

[0083] Module 2: Prototype Attention Module

[0084] like Figure 5 As shown in Figure 2, a prototype attention module is constructed to capture the spatial-spectral information by modeling the feature relationship between pixels in the hyperspectral image. n×d , where n is the number of prototypes. The key K of the prototype c ∈R N×d Sum value V c ∈R N×d They are

[0085] K c =softmax(ZX T ,dim=0)K

[0086] V c =softmax(ZX T ,dim=0)V

[0087] Among them, the expression softmax(ZX T , dim=0) calculates the similarity scores of all features in the hyperspectral image. This process is similar to the soft K-means algorithm used to update the prototype. Subsequently, the prototype attention results are as follows:

[0088]

[0089] The above is the calculation process of a single prototype attention. In the self-attention module, compared with the single-head self-attention, the multi-head self-attention shows superior performance by enabling each tag to capture information beyond its single context, thereby effectively alleviating the overfitting phenomenon. This concept is in line with the principle of “don’t concentrate all resources in one area”, emphasizing the importance of diversified feature extraction to generate more robust representations. Similarly, we introduce multi-head prototype attention in our method, and let the number of heads be h. Therefore, Q, K, and V are usually divided in the channel dimension, and each part is located in R N×(d / h)To further enhance the diversity of prototype attention, we divide Z along the first dimension and obtain Next, we set Q i , K i 、V i Applied to the above prototype attention calculation. The outputs of all heads are then concatenated together to generate the final output.

[0090] Module 3: Spatial Perception Feedforward Network Module

[0091] like Figure 6 As shown in the figure, the spatial perception feedforward network module includes multi-scale depth-wise separable convolution and feedforward network models. First, the input features are divided into channels, and then the depth-wise separable convolutions with different kernel sizes are used for calculation:

[0092] I1 = depthconv(I i1 )

[0093] I2 = depthconv(I i2 )

[0094] I3 = depthconv(I i3 )

[0095] Next, the different outputs are combined into I = (I1, I2, I3) and input into the feedforward network:

[0096] out = FC2(ReLU(FC1(I)))

[0097] Among them, FC is the connection layer and ReLU is the linear rectification activation function. The flexibility of the model is improved by placing the depth-separable convolution layer in the front and using different convolution kernel sizes. For heads with more prototypes, larger convolution kernels are used, while heads with fewer prototypes use smaller convolution kernels to balance local and global feature information.

[0098] In the prototype attention based hyperspectral brain tumor segmentation network, the second and third modules are connected into a new module, and the module is repeated L times.

[0099] Module 4: Classification Module

[0100] Based on the features proposed by the multi-layer prototype attention and spatial perception feedforward network models, two fully connected layers, namely the projection layer and the linear classification layer, are used to obtain the final segmentation map output ( Figure 7 ):

[0101] Class = FC4((FC3(I)))

[0102] Model training and deployment:

[0103] like Figure 3 As shown, the training steps include the following aspects:

[0104] First, the standardized hyperspectral image dataset is input into the model, and the model performance is improved by optimizing the loss function, gradient calculation and iterative optimization, and finally the model parameters are saved. Then, the trained model is verified on the test dataset. Finally, the model is deployed to the clinical system to guide brain tumor resection surgery in real time.

[0105] The method according to the present invention described above may be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code that is originally stored in a remote recording medium or a non-transitory machine-readable medium downloaded over a network and will be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, a processor, or hardware, the hyperspectral brain tumor surgery guided segmentation method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the processing shown herein, the execution of the code converts the general-purpose computer into a dedicated computer for executing the processing shown herein.

[0106] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the implementation methods of the present invention, and should be understood that the protection scope of the present invention is not limited to such special statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A hyperspectral brain tumor surgery-guided segmentation method based on a prototype attention network, characterized in that: The following steps are involved: Step 1: Collect hyperspectral image data of brain tumors through an intraoperative hyperspectral imaging platform and construct a standard dataset for training; Step 2: Design and build a hyperspectral image segmentation model based on a prototype attention network to enhance the difference between tumors and normal tissues by adaptively focusing on important features; Step 3: Use standard datasets to train the hyperspectral image segmentation model and optimize network parameters to improve segmentation accuracy and real-time performance; Step 4: Deploy the trained hyperspectral image segmentation model on the hyperspectral image acquisition platform, process the acquired brain tumor hyperspectral images in real time, output the segmentation results and generate surgical guidance information.

2. The hyperspectral brain tumor surgery guided segmentation method according to claim 1, characterized in that: Step 1 includes the following sub-steps: Step 1.1: Collect high-quality intraoperative hyperspectral images through the hyperspectral acquisition platform, and simultaneously obtain the corresponding tissue samples and pathological "gold standard" diagnosis results to ensure the accuracy of the dataset annotation; Step 1.2: Use the pathological "gold standard" diagnosis results to annotate the hyperspectral image and annotate the ex vivo brain tumor tissue to determine the different types of tumors and their boundaries; Step 1.3: Divide the hyperspectral data into training data, validation data, and test data to ensure the comprehensiveness of the evaluation; Step 1.4: Arrange and standardize the hyperspectral image data and the corresponding category data to form a standard training dataset for brain tumor segmentation.

3. The hyperspectral brain tumor surgery guided segmentation method according to claim 1, characterized in that: Step 2 includes the following sub-steps: Step 2.1: A hyperspectral image segmentation model based on a prototype attention network, combining prototype learning and attention mechanism, enhances the representation ability of tumor features by aggregating similar features; Step 2.2: According to the training dataset for brain tumor segmentation, set the input and output sizes of the hyperspectral image segmentation model and configure the required parameter values, including the learning rate, optimizer function, and loss function.

4. The hyperspectral brain tumor surgery guided segmentation method according to claim 1, characterized in that: Step 3 includes the following sub-steps: Step 3.1: Set the number of training rounds for the hyperspectral image segmentation model to ensure that the model is fully learned and the loss converges; Step 3.1: Input the hyperspectral image training data set into the hyperspectral image segmentation model constructed in step 2 for training. The hyperspectral image segmentation model continuously adjusts parameters during the training process to improve the segmentation effect. Step 3.1: After training is completed, save the optimized network parameters and weight files for subsequent use.

5. The hyperspectral brain tumor surgery guided segmentation method according to claim 1, characterized in that: Step 4 includes the following sub-steps: Step 4.1: Deploy the trained hyperspectral image segmentation model on the hyperspectral standard acquisition platform to complete near real-time segmentation; Step 4.2: Obtain the intraoperative hyperspectral image to be segmented and ensure that the acquisition method and instrument are consistent with the training data; Step 4.3: Process the hyperspectral image to be classified to meet the network input requirements; Step 4.4: Input the processed hyperspectral image into the trained prototype attention network; Step 4.5: Obtain the segmentation results of the entire hyperspectral image, identify the type of brain tumor, and generate a segmentation report to provide intraoperative decision support for doctors.

6. The hyperspectral brain tumor surgery guided segmentation method according to any one of claims 1 to 5, characterized in that: The prototype attention network model combines prototype learning and attention mechanism, and improves the difference between tumor areas and normal tissues by focusing on the key features of brain tumors, thereby improving segmentation accuracy.

7. The hyperspectral brain tumor surgery guided segmentation method according to claim 6, characterized in that: The prototype attention network adopts a multi-head prototype attention module to enhance the diversity of feature extraction, alleviate overfitting and improve the robustness of the model.

8. The hyperspectral brain tumor surgery guided segmentation method according to claim 1, characterized in that: The hyperspectral image data includes spectral information of multiple bands and is standardized to meet the training requirements of the deep learning model.

Images