Medical image segmentation method based on spiking neural network

Abstract

This invention discloses a medical image segmentation method based on a spiking neural network, comprising: inputting a medical image into a segmentation model; an input module performing convolutional encoding on the medical image; an encoder extracting multi-scale semantic features of the medical image, mapping the semantic features at each scale to N feature maps of the same size according to the number of channels N corresponding to each scale to obtain the semantic features of the feature maps; a decoder fusing features of the feature maps at multiple scales corresponding to the encoder of the segmentation model, and restoring the resolution of the feature maps to be consistent with the resolution of the medical image; and an output module outputting a result mask image. Specifically, before inputting the medical image into the segmentation model, the segmentation network of the segmentation model is trained. This invention constructs an Analog Spiking U-Net basic model, improving the model's segmentation performance and scalability; and integrates the Analog CBAM module and the Spiking ViT module into this basic model, resulting in good medical image segmentation performance and high accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image segmentation, in particular to a medical image segmentation method based on a spiking neural network. BACKGROUND

[0002] Medical image segmentation refers to the process of separating the region of interest in a medical image from the background. In recent years, deep learning technology has been widely applied in the field of medical image segmentation, which can automatically extract features, reduce the dependence on manually set parameters, and improve the segmentation accuracy. Among them, the spiking network has good spatiotemporal characteristics, which is suitable for processing time series data and event data application scenarios, and can be applied to the field of medical image segmentation.

[0003] In recent years, attention mechanisms have emerged as a powerful tool in the field of artificial intelligence and machine learning. Among them, the Convolutional Block Attention Module (CBAM) can effectively improve the accuracy of convolutional neural networks in image classification, object detection and other tasks; the self-attention mechanism is adapted to the field of computer vision through position encoding and other technologies, producing the ViT (vision Transformer) module, which uses the self-attention mechanism model to process images of various sizes and resolutions flexibly and avoid information loss in traditional convolutional neural networks.

[0004] Integrating attention modules into spiking neural network models to improve medical image segmentation has the following problems: the internal sub-operations of the convolutional block attention module will destroy the input pulse distribution, and the binary multiplication between pulse signals will cause a lot of information loss; the segmentation model needs to extract features, and deep features and shallow features are equally important, so attention modules need to be integrated at shallow and deep positions to help extract corresponding features; the segmentation model needs to ensure energy consumption advantage, which relates to the deployment and implementation of hardware chips. SUMMARY

[0005] To solve the technical problems of integrating attention modules into spiking neural networks for image segmentation, the present application provides a medical image segmentation method based on a spiking neural network.

[0006] The technical scheme is as follows:

[0007] The medical image segmentation method based on the spiking neural network comprises the following steps:

[0008] S1, integrate an attention module into a spiking neural network model, and input a medical image a into the segmentation model;

[0009] S2. Attention modules are integrated in shallow and deep layers to help extract corresponding features. The segmentation model input module performs convolutional encoding on medical image a and outputs multi-channel feature map b.

[0010] S3. Further improve the Spiking U-Net model into the Analog Spiking U-Net model;

[0011] By introducing the Analog Spiking U-Net basic model, information can be transmitted between sub-modules within the model in the form of floating-point signals, and the Analog CBAM module is integrated into the spiking neural network model.

[0012] The Spiking ViT module is also integrated at a deep level in the Analog Spiking U-Net model;

[0013] By introducing the Analog Spiking U-Net basic model as a framework, information can still participate in the calculation in the form of impulse signals within the sub-modules of the model.

[0014] A segmentation model encoder is constructed. Using the segmentation model encoder, multi-scale semantic features of medical image a are extracted with the help of feature map b. According to the number of channels N corresponding to each scale, the semantic features c at that scale are mapped to N feature maps of the same size to obtain feature map d.

[0015] S4. The segmentation model decoder is used to perform feature fusion on the feature map d at multiple scales corresponding to the segmentation model encoder, and the resolution of the feature map d is restored to be consistent with the resolution of the medical image a.

[0016] S5. The segmentation model output module performs output convolution on the feature map d at restored resolution, with 2 output channels, and outputs a mask map e for medical diagnosis.

[0017] Spiking neural networks (ANNs) are a type of neural network model that mimics biological nervous systems. These models use pulse signals to transmit information, a fundamental communication method between neurons in the brain. NAN models possess excellent spatiotemporal characteristics, making them suitable for processing time-series and event-driven data. The medical image segmentation method of this invention achieves better segmentation results with lower energy consumption, and can reach the same segmentation performance as ANN models with corresponding structural modules.

[0018] Preferably, in step S1, before the medical image a is input into the segmentation model, the segmentation network of the segmentation model is trained, and the training steps are as follows:

[0019] S11. Input the medical image a' from the training set in the dataset into the segmentation model to obtain the corresponding training set mask image e';

[0020] S12. Define the loss function and calculate the loss value between the mask image e' in the training set and the mask labels corresponding to the medical image a' in the training set using the loss function.

[0021] S13. Calculate the gradient of the loss value with respect to each parameter using the chain rule, and update the parameters learned inside the segmentation network using gradient descent.

[0022] S14. Input the medical images in the test set into the segmentation model for segmentation, obtain the segmentation result test set mask image, and determine whether the segmentation accuracy of the test set mask image meets expectations.

[0023] If the segmentation accuracy is less than expected, return to step S12 and perform backpropagation training;

[0024] If the segmentation accuracy is greater than or equal to the expected value, the internally learned parameters of the segmentation network are saved, and training is complete.

[0025] Using the above structure, the loss function is used to calculate the segmentation result and the loss of the mask, and then the parameter weights are updated by gradient descent. Finally, the segmentation accuracy is judged, allowing the system to be trained and the parameter weights updated repeatedly. The medical image segmentation model output after repeated training can be directly used for medical diagnosis.

[0026] Preferably, in step S12 of network training, the loss function is as follows:

[0027] (1)

[0028] Where N represents the number of samples, yi represents the actual label of the i-th sample, pi represents the predicted probability of the i-th sample, and α is a hyperparameter, 0≤α≤1, used to balance the weights of Softmax cross-entropy loss and OHEM loss. When α=1, the loss function becomes the traditional Softmax cross-entropy loss, and when α=0, the loss function becomes the OHEM loss.

[0029] After setting the hyperparameter α, the pixels of the training set mask image e' are substituted as the actual labels of the i-th sample, and the mask labels corresponding to the medical image a' in the training set are substituted as the predicted probabilities of the i-th sample. The sum of the differences between the two is calculated pixel by pixel as the loss value.

[0030] Preferably, in step S13 of network training, the gradient of the loss value with respect to each parameter is calculated using the chain rule, and the parameters learned internally by the network are updated using gradient descent, including the following steps:

[0031] (1) Set the number of preheating iterations and the preheating strategy, wherein the preheating strategy is "constant" or "linear", and the strategy is determined by judging whether the number of iterations is greater than the number of preheating iterations;

[0032] (2) When the number of iterations is less than the number of warm-up iterations, the warm-up strategy is used to calculate the learning rate, which helps the model converge better;

[0033] (3) When the number of iterations is greater than or equal to the number of warm-up iterations, the learning rate is calculated using a polynomial learning rate scheduling strategy.

[0034] The industry consensus is that the gradient of the loss value with respect to each parameter is calculated using the chain rule.

[0035] Preferably, in step S2, the input module performs convolutional encoding on the medical image a, as follows:

[0036] (1) By directly replacing the activation function of the artificial neural network model with a neuron, an input module with the layout of "convolution-batch normalization-neuron-convolution-batch normalization" is obtained;

[0037] (2) The medical image a is directly encoded through the input module to obtain the feature map b.

[0038] Using the above structure, the input module encodes the medical image 'a' using convolution, and the output result is consistent with the output result of pulse coding. The input module is constructed by directly replacing the activation function, ensuring that richer original image information is obtained at the input layer and avoiding the loss of a large amount of original data due to pulse coding.

[0039] Preferably, in step S3, the segmentation model encoder is constructed, and the steps are as follows:

[0040] S31. Use the pulse network model construction method to transform the U-Net model into the Analog Spiking U-Net basic model, and take its encoder part as the main body of the segmentation model encoder.

[0041] S32. Construct an Analog CBAM module using the CBAM module, and integrate the Analog CBAM module into the shallow layer of the segmentation model encoder;

[0042] S33. Construct the Spiking ViT module using the ViT module, and integrate the Spiking ViT module into the deep bottleneck part of the segmentation model.

[0043] In step S31, the U-Net model is transformed into the Analog Spiking U-Net basic model using the spiking network model construction method. The steps are as follows:

[0044] (1) Modify the U-Net model of the artificial neural network into the Spiking U-Net model by directly replacing the activation function with the neuron construction method;

[0045] (2) Adjust the neuron positions of all modules of the Spiking U-Net model from the end to the beginning to obtain the Analog Spiking U-Net basic model.

[0046] The Spiking U-Net model includes a Spiking U-Net operation module, which is a basic operation module for obtaining neuron positions via an Analog Spiking U-Net operation module. The layout of the basic operation module is "neuron-convolution-batch normalization-neuron-convolution-batch normalization".

[0047] In step S32, an Analog CBAM module is constructed using the CBAM module, and the Analog CBAM module is integrated into the shallow layer of the segmentation model encoder. The steps are as follows:

[0048] (1) Construct the CBAM module in the artificial neural network, the steps are as follows:

[0049] (2)

[0050] (3)

[0051] Where F is the input feature map of the CBAM module, This is the output feature map after passing through the channel attention submodule. This is the output feature map after passing through the spatial attention submodule. (Symbol) This represents element-wise multiplication. Mc(·) and Ms(·) represent operations on channel and spatial submodules.

[0052] (4)

[0053] in, and These represent the features after average pooling and max pooling in the spatial dimension, respectively.

[0054] Weights W0 and W1 are shared by these two inputs and are used by the MLP. σ represents the sigmoid activation function applied after the MLP computes the weighted sum.

[0055] (5)

[0056] in, and These represent the features after average pooling and max pooling along the channel dimension, respectively. σ represents the sigmoid function. This indicates a convolution operation with a filter size of 7×7.

[0057] (2) Replace the activation function in the CBAM module with a neuron to obtain the Spiking CBAM module.

[0058] (3) Adjust the neuron positions of the Spiking CBAM module to obtain the Analog CBAM module, as shown below:

[0059] (6)

[0060] (7)

[0061] Where T(·) represents the activation of IF neurons.

[0062] (4) The Analog CBAM module is integrated into the shallow layer of the segmentation model encoder using a residual connection method. During integration, the number of input channels of this module is the same as the number of output channels of the previous module. It can be abstracted as follows:

[0063] (8)

[0064] (9)

[0065] Where F is the input feature map of the CBAM module, This is the output feature map after passing through the channel attention submodule. This is the output feature map after passing through the spatial attention submodule. (Symbol) Represents element-wise multiplication. This represents element-wise addition. Mc(·) and Ms(·) represent operations on channel and spatial submodules.

[0066] As a preferred option: In step S33, the Spiking ViT module is constructed using the ViT module, and the Spiking ViT module is integrated into the deep bottleneck part of the segmentation model, as follows:

[0067] (1) Construct the ViT module in the artificial neural network.

[0068] (2) The ViT module is converted into the SpikingViT spiking neural network module by using the method of replacing activation functions.

[0069] (3) Integrate the Spiking ViT module into the deepest bottleneck part of the segmentation model encoder. When integrating, the number of input channels of this module is consistent with the number of output channels of the previous module, and the feature maps of the output and input maintain structural invariance.

[0070] The Spiking ViT module enhances the relevance of deep abstract features of an image, improving the model's expressive power. The Spiking ViT module processes the abstract feature map b as follows:

[0071] (1) Patch Embedding: The input feature map b is first cut into fixed-size patches. Each patch is stretched into a long vector and then linearly transformed through a fully connected layer to obtain a fixed-dimensional vector representation. These vectors are called patch embeddings.

[0072] (2) Position Embedding: Since the self-attention mechanism cannot handle positional information in the sequence, a position embedding is needed to encode the positional information of each small block. This position embedding can be a fixed function, such as a combination of sine and cosine functions.

[0073] (3) Transformer Encoder:

[0074] ①Layer Normalization: This can normalize each patch embedding in the input to ensure that the numerical range of each patch embedding is similar, thereby improving the stability and convergence speed of the model.

[0075] ② Multi-head Self-Attention: This method concatenates patch embeddings and positional encoding vectors to form a sequence. This sequence then undergoes a multi-head self-attention mechanism, where the sequence is divided into several heads, each calculating its own attention weights, and the results from these heads are concatenated. This process extracts contextual information for each position in the sequence.

[0076] ③Layer Normalization: The attention weights calculated for each head are summed in a weighted manner with their corresponding value vectors to obtain a weighted representation. This weighted sum is then passed through a residual connection and a normalization layer to obtain a new representation.

[0077] ④MLP: Perform a feedforward neural network calculation on the new representation to obtain a new vector representation.

[0078] (4) Patch Composition: Multiple patch embeddings are fused together to obtain a vector representation with more global information, while retaining the local information of each patch. This helps the model to better capture the semantic information of medical image a, thereby improving the model's accuracy and generalization ability.

[0079] Using the above structure, the Analog Spiking U-Net base model serves as the framework for the entire segmentation model, and all operations are performed based on this Analog Spiking U-Net base model.

[0080] Using the above structure, the CBAM module is a module used to enhance the performance of convolutional neural networks. The CBAM module consists of two parts: a channel attention module and a spatial attention module. The channel attention module can adaptively learn the importance of corresponding channels, strengthening useful information and suppressing useless information; the spatial attention module can adaptively learn the importance of corresponding positions in the input tensor, improving the model's ability to perceive spatial features. The introduction of the CBAM module can effectively improve the accuracy of convolutional neural networks in tasks such as image classification and object detection. The Analog CBAM module can help the encoder extract shallow semantic features of images;

[0081] The ViT module is a transformer module for image processing. It segments the input image into several small regions, flattens the pixels within each region into a one-dimensional vector, and then inputs this vector into a self-attention mechanism model for processing. Leveraging the advantages of self-attention mechanisms in natural language processing, the ViT module can more flexibly handle images of various sizes and resolutions, and avoids the information loss problem present in traditional convolutional neural networks.

[0082] Preferably, in step S4, a segmentation model decoder is used to perform feature fusion on the feature map d at multiple scales corresponding to the segmentation model encoder, and the resolution of the feature map d is restored to be consistent with the resolution of the medical image a, including the following steps:

[0083] (1) The decoder part of the Analog Spiking U-Net basic model is used as the segmentation model decoder.

[0084] (2) The encoder skips to connect to the decoder, and splices feature maps d of the same scale on the connection channel to achieve feature fusion;

[0085] (3) When convolutional stitching is performed on feature map d, zero padding is used in the convolution step to ensure that the size of the output feature map d remains unchanged. During stitching, cropping and zero padding are used to ensure that the size of the feature map is consistent.

[0086] (4) The resolution of feature map b is restored to the same size as that of medical image a by multiple upsampling.

[0087] Using the above structure, feature maps of the same size in multi-size feature maps are fused, and zero-padding or cropping is used to ensure that the feature maps maintain the same size during the stitching process.

[0088] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0089] 1. This invention successfully improves the Spiking U-Net model into an Analog Spiking U-Net model by using a new spiking neural network model construction method, thereby improving the model's segmentation performance and scalability.

[0090] 2. This invention introduces the Analog Spiking U-Net basic model, which enables information to be transmitted between sub-modules within the model in the form of floating-point signals. This solves the difficulty of integrating convolutional block attention modules in spiking neural networks and successfully integrates the Analog CBAM module into the spiking neural network model, thereby improving the model's segmentation performance.

[0091] 3. This invention integrates the Spiking ViT module at a deep level in the Analog Spiking U-Net model, which makes up for the neglect of existing models in terms of image feature extraction at multiple scales and improves the segmentation performance of the model.

[0092] 4. This invention introduces the Analog Spiking U-Net basic model as a framework, so that information still participates in the calculation in the form of pulse signals within the sub-modules of the model. By utilizing the characteristics of pulse sparsity, the energy consumption of the model is greatly reduced, which is more conducive to its deployment and implementation on hardware mobile chips. Attached Figure Description

[0093] Figure 1 This is a schematic diagram of a medical image segmentation method based on a spiking neural network.

[0094] Figure 2 This is a schematic diagram of the segmentation network structure;

[0095] Figure 3 A schematic diagram illustrating the construction method of the new spiking neural network model;

[0096] Figure 4 This is a schematic diagram of a traditional CBAM module structure;

[0097] Figure 5This is a schematic diagram of the channel attention module and spatial attention module within the Analog CBAM module.

[0098] Figure 6 A schematic diagram illustrating the integration method of the Analog CBAM module;

[0099] Figure 7 This is a schematic diagram of the Spiking ViT module structure. Detailed Implementation

[0100] The present invention will be further described below with reference to the embodiments and accompanying drawings.

[0101] Example 1

[0102] like Figure 1 As shown, the medical image segmentation method based on spiking neural networks is characterized by the following steps:

[0103] S1. Input the medical image a into the segmentation model;

[0104] S2. The segmentation model input module performs convolutional encoding on the medical image a and outputs a multi-channel feature map b.

[0105] S3. Further improve the Spiking U-Net model into the Analog Spiking U-Net model;

[0106] By introducing the Analog Spiking U-Net basic model, information can be transmitted between sub-modules within the model in the form of floating-point signals, and the Analog CBAM module is integrated into the spiking neural network model.

[0107] The Spiking ViT module is also integrated at a deep level in the Analog Spiking U-Net model;

[0108] By introducing the Analog Spiking U-Net basic model as a framework, information can still participate in the calculation in the form of impulse signals within the sub-modules of the model.

[0109] A segmentation model encoder is constructed. Using the segmentation model encoder, multi-scale semantic features of medical image a are extracted with the help of feature map b. According to the number of channels N corresponding to each scale, the semantic features c at that scale are mapped to N feature maps of the same size to obtain feature map d.

[0110] S4. The segmentation model decoder is used to perform feature fusion on the feature map d at multiple scales corresponding to the segmentation model encoder, and the resolution of the feature map d is restored to be consistent with the resolution of the medical image a.

[0111] S5. The segmentation model output module performs output convolution on the feature map d at restored resolution, with 2 output channels, and outputs a mask map e for medical diagnosis.

[0112] In step S1, before the medical image a is input into the segmentation model, the segmentation network of the segmentation model is trained. The training steps are as follows:

[0113] S11. Input the medical image a' from the training set in the dataset into the segmentation model to obtain the corresponding training set mask image e';

[0114] S12. Define the loss function and calculate the loss value between the mask image e' in the training set and the mask labels corresponding to the medical image a' in the training set using the loss function.

[0115] S13. Calculate the gradient of the loss value with respect to each parameter using the chain rule, and update the parameters learned inside the segmentation network using gradient descent.

[0116] S14. Input the medical images in the test set into the segmentation model for segmentation, obtain the segmentation result test set mask image, and determine whether the segmentation accuracy of the test set mask image meets expectations.

[0117] If the segmentation accuracy is less than expected, return to step S12 and perform backpropagation training;

[0118] If the segmentation accuracy is greater than or equal to the expected value, the internally learned parameters of the segmentation network are saved, and training is complete.

[0119] The training medical image dataset of this invention is a comprehensive dataset containing 3,000 512x512 eye RGB images and their corresponding masks for the training set, and 56 512x512 eye RGB images and their corresponding masks for the test set.

[0120] In step S12, the loss function is defined as follows:

[0121] (1)

[0122] Where N represents the number of samples, yi represents the actual label of the i-th sample, pi represents the predicted probability of the i-th sample, and α is a hyperparameter, 0≤α≤1, used to balance the weights of the Softmax cross-entropy loss and the OHEM loss. When α=1, the loss function becomes the traditional Softmax cross-entropy loss; when α=0, the loss function becomes the OHEM loss.

[0123] After setting the hyperparameter α, the pixels of the training set mask image e' are substituted as the actual labels of the i-th sample, and the mask labels corresponding to the medical image a' in the training set are substituted as the predicted probabilities of the i-th sample. The sum of the differences between the two is calculated pixel by pixel as the loss value.

[0124] In step S13, the gradient of the loss value with respect to each parameter is calculated using the chain rule, and the parameters learned internally by the network are updated using gradient descent, including the following steps:

[0125] 1) Set the number of preheating iterations and the preheating strategy, wherein the preheating strategy is either "constant" or "linear", and the strategy is determined by judging whether the number of iterations is greater than the number of preheating iterations;

[0126] (2) When the number of iterations is less than the number of warm-up iterations, the warm-up strategy is used to calculate the learning rate, which helps the model converge better;

[0127] (3) When the number of iterations is greater than or equal to the number of warm-up iterations, the learning rate is calculated using a polynomial learning rate scheduling strategy.

[0128] like Figure 2 As shown, in step S2, the input module performs convolutional encoding on the medical image a, and the steps are as follows:

[0129] (1) The input module layout is obtained by directly replacing the activation function of the artificial neural network model with a neuron. The layout is "convolution-batch normalization-neuron-convolution-batch normalization".

[0130] (2) The medical image a is directly encoded through the input module to obtain a multi-channel feature map with the same effect as pulse coding.

[0131] In step S3, the segmentation model encoder is constructed, and the steps are as follows:

[0132] S31. Use the pulse network model construction method to transform the U-Net model into the Analog Spiking U-Net basic model, and take its encoder part as the main body of the segmentation model encoder.

[0133] S32. Construct an Analog CBAM module using the CBAM module, and integrate the Analog CBAM module into the shallow layer of the segmentation model encoder;

[0134] S33. Construct the Spiking ViT module using the ViT module, and integrate the Spiking ViT module into the deep bottleneck part of the segmentation model.

[0135] In step S31, the U-Net model is transformed into the Analog Spiking U-Net basic model using the spiking network model construction method. The steps are as follows:

[0136] (1) Modify the U-Net model of the artificial neural network into the Spiking U-Net model by directly replacing the activation function with the neuron construction method;

[0137] (2) Adjust the neuron positions of all modules of the Spiking U-Net model from the end to the beginning to obtain the Analog Spiking U-Net basic model.

[0138] like Figure 3 As shown, the Spiking U-Net model includes a Spiking U-Net operation module, which, after obtaining neuron positions, is followed by an Analog Spiking U-Net basic operation module. The Analog Spiking U-Net basic operation module is laid out as "neuron-convolution-batch normalization-neuron-convolution-batch normalization". The remaining modules are adjusted as described above to obtain the model shown. Figure 2 The Analog Spiking U-Net basic model is shown after removing the integrated modules from the structure.

[0139] like Figure 4 , Figure 5 As shown, in step S32, the Analog CBAM module is constructed using the CBAM module, and the Analog CBAM module is integrated into the shallow layer of the segmentation model encoder. The steps are as follows:

[0140] (1) The BAM module constructed in the neural network can be abstracted as follows:

[0141] (2)

[0142] (3)

[0143] Where F is the input feature map of the CBAM module, This is the output feature map after passing through the channel attention submodule. This is the output feature map after passing through the spatial attention submodule. (Symbol) This represents element-wise multiplication. Mc(·) and Ms(·) represent operations on channel and spatial submodules.

[0144] (4)

[0145] in and These represent the features after average pooling and max pooling in the spatial dimension, respectively.

[0146] Weights W0 and W1 are shared by these two inputs and are used by the MLP. σ represents the sigmoid activation function applied after the MLP computes the weighted sum.

[0147] (5)

[0148] in and These represent the features after average pooling and max pooling along the channel dimension, respectively. σ represents the sigmoid function. This indicates a convolution operation with a filter size of 7×7.

[0149] (2) The Spiking CBAM module is obtained by directly replacing the activation function of the CBAM module.

[0150] (3) The invention's construction method adjusts the neuron positions to obtain an Analog CBAM module, which can be abstracted as follows:

[0151] (6)

[0152] (7)

[0153] Where T(·) represents the activation of IF neurons.

[0154] (4) The Analog CBAM module is integrated into the shallow layer of the segmentation model encoder using a residual connection method:

[0155] like Figure 6 As shown, when integrated, the number of input channels of this module is the same as the number of output channels of the previous module, which can be abstracted as follows:

[0156] (8)

[0157] (9)

[0158] In the formula, F is the input feature map of the CBAM module. This is the output feature map after passing through the channel attention submodule. This is the output feature map after passing through the spatial attention submodule. (Symbol) Represents element-wise multiplication. This represents element-wise addition. Mc(·) and Ms(·) represent operations on channel and spatial submodules.

[0159] like Figure 2 As shown, in step S33, the Spiking ViT module is constructed using the ViT module, and the Spiking ViT module is integrated into the deep bottleneck part of the segmentation model. The steps are as follows:

[0160] (1) Construct the ViT module in the artificial neural network.

[0161] (2) The ViT module is converted into the SpikingViT spiking neural network module by using the method of replacing activation functions.

[0162] (3) Integrate the Spiking ViT module into the deepest bottleneck part of the segmentation model encoder. When integrating, the number of input channels of this module is consistent with the number of output channels of the previous module, and the feature maps of the output and input maintain structural invariance.

[0163] like Figure 7 As shown, the Spiking ViT module is integrated into the deep bottleneck part of the segmentation model. The Spiking ViT module processes the abstract feature map input to it in the following steps:

[0164] (1) PatchEmbedding: The input image is first cut into fixed-size patches. Each patch is stretched into a long vector and then linearly transformed through a fully connected layer to obtain a fixed-dimensional vector representation. These vectors are called patch embeddings.

[0165] (2) PositionEmbedding: Since the self-attention mechanism cannot handle positional information in the sequence, a position embedding is needed to encode the positional information of each patch. This position embedding can be a fixed function, such as a combination of sine and cosine functions.

[0166] (3) TransformerEncoder:

[0167] ①LayerNormalization: This can normalize each input patch bedding to ensure that the numerical range of each patch bedding is similar, thereby improving the stability and convergence speed of the model.

[0168] ② Multi-head Self-Attention: This method concatenates the patch beddings and positional encoding vectors to form a sequence. This sequence then undergoes a multi-head self-attention mechanism, where the sequence is divided into several heads, each calculating its own attention weights, and the results from these heads are concatenated. This process extracts contextual information for each position in the sequence.

[0169] ③Layer Normalization: The attention weights calculated for each head are summed in a weighted manner with their corresponding value vectors to obtain a weighted representation. This weighted sum is then passed through a residual connection and a normalization layer to obtain a new representation.

[0170] ④MLP: Perform a feedforward neural network calculation on the new representation to obtain a new vector representation.

[0171] (4) Patch Composition: Multiple patch beddings are combined to obtain a vector representation with more global information, while retaining the local information of each patch. This helps the model to better capture the semantic information of the image, thereby improving the model's accuracy and generalization ability.

[0172] like Figure 2 As shown, in step S4, the segmentation model decoder is used to perform feature fusion on the feature map d at multiple scales corresponding to the segmentation model encoder, and the resolution of the feature map d is restored to be consistent with the resolution of the medical image a. This includes the following steps:

[0173] (1) Use the decoder part of the Analog Spiking U-Net basic model as the segmentation model decoder;

[0174] (2) The encoder skips to connect to the decoder, and splices feature maps of the same scale on the connection channel to achieve feature fusion;

[0175] (3) When convolutional stitching is performed on feature map d, zero padding is used in the convolution step to ensure that the size of the output feature map d remains unchanged. During stitching, cropping and zero padding are used to ensure that the size of the feature map is consistent.

[0176] (4) The resolution of feature map b is restored to the same size as that of medical image a by multiple upsampling.

[0177] The output module of the segmentation model performs an output convolution on the feature map d at restored resolution, with 2 output channels, and the output result is a mask map e.

[0178] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention. Those skilled in the art, under the guidance of the present invention, can make various similar representations without departing from the spirit and claims of the present invention, and such modifications all fall within the protection scope of the present invention.

Claims

1. A method for medical image segmentation based on a spiking neural network, characterized in that, The method comprises the following steps: S1, inputting a medical image a into a segmentation model; S2, the segmentation model input module convolves and encodes the medical image a, and outputs a multi-channel feature atlas b; S3, a segmentation model encoder is constructed, and the segmentation model encoder extracts multi-scale semantic features of the medical image a with the help of the feature atlas b, and maps the semantic features c at this scale to N feature maps of the same size to obtain a feature map d according to the channel number N corresponding to each scale; S4, the segmentation model decoder is used to fuse the features of the feature map d at multiple scales corresponding to the segmentation model encoder, and restore the resolution of the feature map d to be consistent with the resolution of the medical image a; S5, the segmentation model output module outputs the restored resolution feature map d for output convolution, the output channel number is 2, and the output mask graph e is used for medical diagnosis; In step S3, the segmentation model encoder is constructed, and the steps are as follows: S31, the U-Net model is transformed into an Analog Spiking U-Net basic model using a pulse network model construction method, and the encoder part thereof is taken as the main body of the segmentation model encoder; S32, an Analog CBAM module is constructed using a CBAM module, and the Analog CBAM module is integrated into a shallow position of the segmentation model encoder; S33, a Spiking ViT module is constructed using a ViT module, and the Spiking ViT module is integrated into a deep bottleneck part of the segmentation model; In step S31, the U-Net model is transformed into an Analog Spiking U-Net basic model using a pulse network model construction method, and the steps are as follows: (1) the U-Net model of the artificial neural network is modified into a Spiking U-Net model by directly replacing the activation function with a neuron construction method; (2) adjust the position of the neuron of the Spiking U-Net model from the end to the beginning to obtain an Analog Spiking U-Net basic model; Wherein, the Spiking U-Net model comprises a Spiking U-Net operation module, the Spiking U-Net operation module is obtained by adjusting the position of the neuron of the Analog Spiking U-Net basic operation module, and the Analog Spiking U-Net basic operation module is arranged as "neuron-convolution-batch normalization-neuron-convolution-batch normalization"; In step S32, an Analog CBAM module is constructed using a CBAM module, and the Analog CBAM module is integrated into a shallow position of the segmentation model encoder, and the steps are as follows: (1) a CBAM module is constructed in an artificial neural network; (2) the CBAM module is obtained by directly replacing the activation function to obtain a Spiking CBAM module; (3) adjust the position of the neuron of the Spiking CBAM module to obtain an Analog CBAM module; (4) The Analog CBAM module is integrated in a shallow position of the segmentation model encoder by using a residual connection mode; In step S33, the Spiking ViT module is constructed by using the ViT module, and the Spiking ViT module is integrated into the deep bottleneck part of the segmentation model, and the steps are as follows: (1) constructing the ViT module in the artificial neural network; (2) converting the ViT module into the Spiking ViT module by using the construction method of replacing the activation function; (3) integrating the Spiking ViT module into the bottleneck part at the deepest part of the segmentation model encoder. 2.The pulse-based neural network based medical image segmentation method of claim 1, wherein, In step S1, before the medical image a is input into the segmentation model, the segmentation network of the segmentation model is trained, and the training steps are as follows: S11, input the training set medical image a' in the data set into the segmentation model to obtain the corresponding training set mask graph e'; S12, define a loss function, and calculate the loss value between the training set mask graph e' and the mask label corresponding to the training set medical image a' through the loss function; S13, calculate the gradient of the loss value with respect to each parameter by the chain rule, and update the parameters learned inside the segmentation network by using the gradient descent method; S14, input the test set medical image into the segmentation model for segmentation to obtain the segmentation result test set mask graph, and judge whether the segmentation accuracy of the test set mask graph reaches the expectation or not; If the segmentation accuracy is less than the expectation, return to step S12 and perform back propagation training; If the segmentation accuracy is greater than or equal to the expectation, save the internal learning parameters of the segmentation network, and the training is completed.

3. The pulse-based neural network based medical image segmentation method according to claim 2, characterized in that, In step S12 of network training, the loss function is as follows: ； Where N represents the number of samples, yi represents the actual label of the i-th sample, pi represents the prediction probability of the i-th sample, and a is a hyperparameter, 0≤a≤1, used to balance the weights of Softmax cross entropy loss and OHEM loss. When a=1, the loss function becomes the traditional Softmax cross entropy loss, and when a=0, the loss function becomes the OHEM loss; After setting the hyperparameter a, substitute the pixel points of the training set mask graph e' as the actual label of the i-th sample, and substitute the mask label corresponding to the training set medical image a' as the prediction probability of the i-th sample, and calculate the sum of the difference between the two as the loss value. 4.The pulse-based neural network based medical image segmentation method of claim 2, wherein, In step S13 of network training, the gradient of the loss value with respect to each parameter is calculated by the chain rule, and the parameters learned inside the network are updated by using the gradient descent method, including the following steps: (1) set the preheating iteration number and the preheating strategy, wherein the preheating strategy is "constant" or "linear", and the strategy is determined by judging whether the iteration number is greater than the preheating iteration number or not; (2) when the iteration number is less than the preheating iteration number, the learning rate is calculated by using the preheating strategy, which is helpful for the model to converge better; (3) when the iteration number is greater than or equal to the preheating iteration number, the learning rate is calculated by using the polynomial learning rate scheduling strategy. 5.The pulse neural network-based medical image segmentation method of claim 1, wherein, In step S2, the segmentation model input module convolves and encodes the medical image a, and the steps are as follows: (1) using the way of directly replacing the activation function of the artificial neural network model as the neuron to obtain an input module with a layout of "convolution-batch normalization-neuron-convolution-batch normalization"; (2) directly encoding the medical image a through the input module to obtain the feature map b. 6.The pulse neural network-based medical image segmentation method of claim 1, wherein, In step S4, the feature map d is subjected to feature fusion at multiple scales corresponding to the segmentation model encoder by using a segmentation model decoder, and the resolution of the feature map d is restored to be consistent with the resolution of the medical image a, including the following steps: (1) taking the decoder part of the Analog Spiking U-Net basic model as the segmentation model decoder; (2) the encoder is connected to the decoder, and the feature maps d of the same scale are subjected to convolution splicing on the connection channel to realize feature fusion; (3) when the feature map d is subjected to convolution splicing, the feature map d is subjected to zero padding in the convolution step to ensure that the size of the output feature map d is unchanged, and the feature map size is ensured to be consistent by means of cutting and zero padding when splicing; (4) the resolution of the feature map b is restored to be consistent with the resolution of the medical image a by multiple upsampling.

Images