2D brain tumor segmentation method based on fuzzy convolution combined with Transform

Abstract

The invention provides a 2D brain tumor segmentation method based on fuzzy convolution combined with Transform, and belongs to the technical field of medical image processing and deep learning. The technical problem that the existing method is difficult to accurately segment the fuzzy boundary of the brain tumor and reduce the calculation resource quantity at the same time is solved. According to the technical scheme, a 2D brain tumor segmentation model based on fuzzy convolution combined with Transformer is constructed, the model comprises a fuzzy learning module, a convolutional neural network, an encoder, a decoder and a gating weight controller, the fuzzy learning module comprises a fuzzy membership function and a fuzzy rule, the convolutional neural network is inspired by ResNet, and the fuzzy membership function and the fuzzy rule are integrated into the 2D brain tumor segmentation model. The encoder comprises a convolutional layer and a gated axial self-attention module, and the decoder comprises a convolutional layer and a deconvolutional layer. The method has the beneficial effects that the segmentation accuracy of the brain tumor is improved, and meanwhile, the calculation resource quantity is reduced.

Description

Technical Field

[0001] The present invention relates to the field of medical image processing and deep learning technology, and in particular to a 2D brain tumor segmentation method based on fuzzy convolution combined with Transformer. Background Art

[0002] Brain tumor is one of the most deadly diseases in the nervous system. Its early diagnosis and precise treatment are crucial to the survival rate and quality of life of patients. In the field of medical image processing, the task of brain tumor image segmentation aims to automatically extract tumor areas from brain images. First, accurate brain tumor segmentation can greatly assist clinicians in diagnosis and treatment planning. By accurately identifying and separating tumor tissue, doctors can better understand the size, location and spread of the tumor in the brain, thereby providing an important basis for formulating treatment plans such as surgical resection range and radiotherapy plan. Secondly, the morphological characteristics of brain tumors, such as boundary fuzziness and tumor volume, are closely related to the patient's prognosis. Automated image segmentation tools can provide more quantitative and objective tumor data, which helps doctors evaluate the efficacy before and after treatment. Traditional tumor segmentation methods usually rely on manual annotation, which is not only time-consuming and laborious, but also due to the influence of human factors, the results may be subjective and inconsistent. With the rapid development of deep learning, neural network-based tumor segmentation methods have gradually become a research hotspot. These methods can not only significantly improve the efficiency of segmentation, but also achieve more consistent and accurate segmentation results on large-scale data sets.

[0003] Convolutional neural networks are widely used in brain tumor image segmentation tasks, such as U-Net++, Res-UNet, TernausNet, etc. These networks have become standard tools in this field with their powerful feature extraction capabilities and deep understanding of image spatial structure. The inductive bias of the convolution operation is mainly reflected in its locality and weight sharing. However, the inductive bias of convolutional neural networks also brings some limitations. In brain tumor image segmentation, long-range dependencies are crucial for accurate segmentation, and due to the locality of the convolution operation, it is often difficult to directly capture such long-range dependencies. The self-attention mechanism can effectively capture long-range dependencies. Therefore, many researchers have begun to introduce the self-attention mechanism into convolutional neural networks to achieve the effect of combining local important feature capture with global perception. TransUNet extracts key features through convolutional neural networks and uses multi-head self-attention mechanisms to learn global information; SCUNet++ introduces Swin Transformer in the encoder and decoder to fully capture global features, while supplementing the learning of local features through convolutional neural networks. However, due to the small number of medical image samples and the large number of parameters of the self-attention mechanism, its performance is difficult to reach an ideal level. In addition, the self-attention mechanism also requires very high computing resources.

[0004] In the task of brain tumor image segmentation, some problems with the tumor itself will directly affect the accuracy and efficiency of image segmentation. During the image acquisition process, artifacts may be generated due to the influence of various factors such as acquisition equipment, patient movement, scanning conditions, etc., which will interfere with the visual effect of the image and increase the difficulty of image segmentation. In addition, the boundary between the tumor and other brain organs is blurred, making the structure and details in the image difficult to identify. Fuzzy learning is a learning algorithm that processes uncertain information. It describes the uncertain information in the task by establishing a fuzzy mathematical model, thereby achieving better segmentation results. Some researchers have tried to introduce fuzzy learning to process the uncertain information in the image in order to achieve better segmentation results. For example, FTransCNN uses fuzzy integral to suppress the heterogeneity and uncertainty in the fusion features, and uses the fuzzy attention fusion module to deal with the uncertainty of the segmentation boundary. Summary of the invention

[0005] The purpose of the present invention is to provide a 2D brain tumor segmentation method based on fuzzy convolution combined with Transformer, which solves the technical problem that the existing methods are difficult to accurately segment the fuzzy boundaries of brain tumors and reduce the amount of computing resources at the same time, and proposes a novel network structure that improves the accuracy of brain tumor segmentation while reducing the amount of computing resources.

[0006] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is specifically: a 2D brain tumor segmentation method based on fuzzy convolution combined with Transformer, comprising the following steps:

[0007] S1: Brain tumor images exist in both 2D and 3D forms. For 3D images, the programming code slices them into 2D images to obtain the training data set, validation data set, and test data set required for the 2D segmentation task; and for 2D images, the programming code directly divides them into the training data set, validation data set, and test data set.

[0008] S2: Add a series of data augmentation operations such as random cropping, random rotation, and random flipping to the training dataset and the validation dataset;

[0009] S3: Construct a 2D brain tumor segmentation model based on fuzzy convolution combined with Transformer, which includes a fuzzy learning module, a convolutional neural network, an encoder, a decoder, and a gated weight controller. The fuzzy learning module includes a fuzzy membership function and a fuzzy rule. The convolutional neural network is inspired by ResNet and is designed and constructed. The encoder includes multiple convolutional layers and a gated axial self-attention module. The decoder includes multiple convolutional layers and multiple deconvolutional layers. Multiple residual connections are set between the encoder and the decoder.

[0010] S4: Train a 2D brain tumor segmentation model based on fuzzy convolution combined with Transformer, use the Adam optimization algorithm to train the model parameters, set the number of model training times, and set the mean intersection over union (MIoU) and F1 score (F1-score) to measure the performance of the model;

[0011] S5: In a certain round of training, when the above two evaluation indicators reach the threshold value on both the training data set and the validation data set, the parameter model obtained in this round of training is taken as the final parameter model for training to perform 2D segmentation on the images in the test data set to evaluate the authenticity of the model.

[0012] As a further optimization scheme of the 2D brain tumor segmentation method based on fuzzy convolution combined with Transformer provided by the present invention, the step S1 specifically includes the following steps:

[0013] Step S1.1: Retrieve and download the brain tumor image segmentation dataset from the official platform of medical public datasets;

[0014] Step S1.2: When the data set is a 3D image, the programming code slices it into 2D images as an image segmentation data set, and when the data set is a 2D image, it is directly used as an image segmentation data set;

[0015] Step S1.3: Randomly divide 20% of the image segmentation dataset as a validation dataset, 10% as a test dataset, and the remaining 70% as a training dataset.

[0016] As a further optimization scheme of the 2D brain tumor segmentation method based on fuzzy convolution combined with Transformer provided by the present invention, the step S2 specifically includes the following steps:

[0017] Step S2.1: Based on the training data set and the validation data set, perform data augmentation operations such as cropping with random size and random position, rotating with random degrees, and flipping with random degrees on each image to enhance data diversity;

[0018] Step S2.2: The constructed 2D brain tumor segmentation model based on fuzzy convolution combined with Transformer has fixed requirements on the size of the training dataset and the validation dataset, and controls the final size of the training dataset and the validation dataset after the data augmentation operation to be 128×128×1 or 128×128×3.

[0019] As a further optimization scheme of the 2D brain tumor segmentation method based on fuzzy convolution combined with Transformer provided by the present invention, the step S3 specifically includes the following steps:

[0020] Step S3.1: It is composed of two branches, namely, a fuzzy convolutional neural network branch and a gated axial self-attention mechanism branch. The two branches perform model calculations concurrently and fuse the results at the end. After fusion, a convolution operation is performed to obtain the final output result. The size of the convolution kernel is 1×1 and the step size is also 1. The fuzzy convolutional neural network branch includes a fuzzy learning module and a convolutional neural network. The gated axial self-attention mechanism branch includes an encoder, a decoder, and a gated weight controller.

[0021] Step S3.2: The fuzzy learning module described in step S3 includes fuzzy membership functions and fuzzy rules. At the input end of the fuzzy learning module, M fuzzy membership functions are independently applied to each channel of the feature map. Although the fuzzy membership function remains consistent within the same channel, it is flexibly adjusted between different channels according to the characteristics to adapt to the variable textures and structures in the image. For each feature point in each channel, each fuzzy membership function calculates a fuzzy value for it, and the fuzzy value describes the degree to which it belongs to a certain set. All fuzzy membership functions use Gaussian functions in the following form:

[0022]

[0023] Among them, W and H refer to the width and height of the feature map respectively, C refers to the number of channels of the feature map, M is the number of fuzzy membership functions, which is the only hyperparameter that needs to be preset, and (x, y) is the feature point P in channel c. x,y,c The position coordinates, σ m,c and μ m,c is the standard deviation and mean of the mth Gaussian membership function, L x,y,m,c is the fuzzy value of the mth fuzzy output of the feature point at (x, y) in channel c, σ m,c and μ m,c It is the data information that is continuously learned as the model is trained;

[0024] Finally, for each feature point, the M fuzzy outputs are fused using fuzzy logic “AND” to obtain the final fuzzy result L x,y,c , the process is as follows:

[0025]

[0026] Step S3.3: The convolutional neural network described in step S3 is inspired by ResNet and constructed. The downsampling structure in the convolutional neural network includes a three-layer network architecture. The first layer of the network architecture is sequentially composed of a convolutional layer, a batch normalization operation, and a ReLU activation function. The size of the convolution kernel of the convolutional layer is 7×7, and the step size of the movement is 2. The effect of the convolution operation on the size of the feature map is formulated as shown in the following formula:

[0027] W out =(W in -k+2P) / S+1

[0028] Among them, W out is the size of the output feature map, W in is the size of the input medical image, k is the size of the convolution kernel, P is the size of the zero padding, and S is the size of the convolution kernel movement step;

[0029] The fuzzy learning module is set after the first layer of the network architecture in the downsampling structure. The result obtained by the fuzzy learning module is fused with the result obtained by the first layer of the network architecture, and then a batch normalization operation is performed, and then a ReLU activation function is added;

[0030] The second-layer network architecture consists of a maximum pooling layer and three residual modules. After the maximum pooling operation with a pooling operator size of 3×3 and a moving step of 2, the height and width of the feature map are both reduced to half of the original. The maximum pooling operation is understood according to the following formula:

[0031] A [i,j]=max(x[i×stride_h:i×stride_h+h,j×stride_w:j×stride_w+w])

[0032] Among them, A [i,j] Represents the information of the element in the i-th row and j-th column of the output feature map, x represents the input feature map, stride_h and stride_w represent the moving steps in the vertical and horizontal directions respectively, and h and w represent the height and width of the pooling operator;

[0033] The three residual modules in the second-layer network architecture work in series. The working process of each residual module is that the original feature map passes through the convolution layer with a convolution kernel size of 1×1 and a moving step size of 1, the batch normalization function and the ReLU activation function in sequence, and then passes through the convolution layer with a convolution kernel size of 3×3 and a moving step size of 1, the batch normalization function and the ReLU activation function in sequence, and finally passes through the convolution layer with a convolution kernel size of 1×1 and a moving step size of 1 and the ReLU activation function in sequence to obtain the intermediate process feature map, which is residually connected with the original feature map once, and then passes through a ReLU activation function to obtain the final output feature map;

[0034] The third-layer network architecture consists of four consecutive identical residual modules. The module structure of the residual module is almost the same as that of the second-layer network architecture. Only the calculation of the number of channels of the feature map during the three convolution operations is changed.

[0035] The upsampling structure contains three upsampling modules. Each upsampling module performs a deconvolution operation to double the size of the feature map, a convolution operation with a convolution kernel size of 3×3 and a moving step of 1, a batch normalization operation, and a ReLU activation function. Between the downsampling structure and the upsampling structure, the feature map of the same size is skipped and then input into the next upsampling module. After the upsampling structure, there is a batch normalization operation.

[0036] Step S3.4: The encoder described in step S3 includes multiple convolutional layers and a gated axial self-attention module. After a downsampling operation, the size is The feature map of is input into the first encoder constructed by the present invention. In the first encoder, the feature map first undergoes a convolution operation with a convolution kernel size of 1×1 and a moving step size of 1 to increase the number of channels of the input feature map, and then a batch normalization operation is performed. Subsequently, the size is The feature map is sent to the gated axial self-attention calculation, where C1 represents the number of channels. When calculating the similarity of pixels, the self-attention only operates on the row and column of the height axis and the width axis. The calculation method and operation process of the two are exactly the same. The mathematical formula for calculating the gated axial self-attention on the height axis is as follows:

[0037]

[0038] Among them, q ij represents a query at any position i∈{1,...,H} and j∈{1,...,W}, Indicates q ij The transpose of k ih and v ih Represent the keys and values ​​at positions i∈{1,...,H} and h∈{1,...,W} respectively, and then calculate the similarity between the information at position (i,j) and the information at position (i,h) through the softmax() function. Then, the position deviation information in the axial direction is added. and Added learnable weight parameter G Q , G K , G V1 and G V2 ;

[0039] After the gated axial self-attention calculation is completed, a convolution operation with a kernel size of 1×1 and a moving step of 1 and a batch normalization operation are performed again. Then, the information of the current feature map is residually connected with the information of the feature map before being sent to the encoder as the final output of the first encoder, which is also the starting input of the second encoder.

[0040] The calculation method and operation process of the second encoder are similar to those of the first encoder. Only the calculation amount of the gated axial self-attention is changed, that is, the calculation is performed with a step size of 2 on the height axis and the width axis, and the final size is The feature map information of , where C2 represents the number of channels, and the calculation method and operation process of the third encoder are the same as those of the second encoder, so the final size is Feature map information, where C3 represents the number of channels;

[0041] Step S3.5: The decoder described in step S3 first performs a convolution operation with a convolution kernel size of 3×3 and a moving step size of 1, then performs a deconvolution operation to double the size of the feature map, and finally uses a ReLU activation function as a whole decoder. The number of decoders corresponds to the number of encoders. The step size of the convolution operation in the first decoder is 2, and the step sizes of the convolution operations in the second and third decoders are both 1;

[0042] Between the encoder and the decoder, the feature maps of the same size after being processed by the encoder and the decoder are skipped and then input into the next decoder. After the decoder structure, there is a batch normalization operation.

[0043] Step S3.6: The gated weight controller described in step S3 is a first-order parameter that can be continuously learned as the model is trained. It is set after the batch normalization operation, and it dynamically controls the proportion of learning global context information and learning local detail information. The mathematical formula of the working principle of the gated weight controller is as follows:

[0044]

[0045] The z in the equation end Represents the result after the gated weight controller completes the operation, G w It is the door weight controller. It is the parameter information of the feature map after the batch normalization operation.

[0046] As a further optimization scheme of the 2D brain tumor segmentation method based on fuzzy convolution combined with Transformer provided by the present invention, the step S4 specifically includes the following steps:

[0047] Step S4.1: loading the optimization algorithm into the 2D brain tumor segmentation model based on fuzzy convolution combined with Transformer, and using the Adam optimization algorithm to train the model parameters to optimize the model parameters;

[0048] Step S4.2: To further improve the training effect, the learning rate η is gradually reduced during the training process, and a learning rate adjustment strategy based on exponential decay is adopted, and the formula is as follows:

[0049]

[0050] Among them, η t is the learning rate in the tth round of training, η0 is the initial learning rate, γ is the decay rate, and T is the total number of training rounds.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] 1. The present invention combines gated axial self-attention with the convolutional neural network designed by the present invention, which solves the problem of excessive computational complexity when the traditional CNN is combined with the self-attention mechanism, and at the same time performs well in capturing both local detail information and global context information.

[0053] 2. To address the tumor boundary ambiguity and noise problems in brain tumor segmentation images, the present invention embeds the proposed fuzzy learning module into the convolutional neural network designed by the present invention, suppressing these uncertainties and improving the accuracy of segmentation.

[0054] 3. The gated weight controller proposed in the present invention is used to balance the capture of local perception and global context information, thereby improving the overall segmentation effect in the brain tumor image segmentation task.

[0055] Experiments on different data sets show that the present invention has practical application potential and can provide a new solution for the field of brain tumor image segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0057] Figure 1 This is a flowchart of the 2D brain tumor segmentation method based on fuzzy convolution combined with Transformer of the present invention.

[0058] Figure 2 This is a structural diagram of the 2D brain tumor segmentation model based on fuzzy convolution combined with Transformer in the present invention.

[0059] Figure 3 It is a structural diagram of the fuzzy learning module of the present invention.

[0060] Figure 4 The figure is a flow chart of the downsampling structure of the fuzzy convolutional neural network branch of the present invention.

[0061] Figure 5 The present invention is a flowchart of the residual module in the downsampling structure of the fuzzy convolutional neural network branch. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. Of course, the specific embodiments described here are only used to explain the present invention and are not used to limit the present invention.

[0063] Example 1

[0064] The technical solution provided in this embodiment is a 2D brain tumor segmentation method based on fuzzy convolution combined with Transformer, and the method comprises the following steps:

[0065] Step 1: Download the BraTS-GLI 2023 dataset from the official Brain Tumor Segmentation website. This dataset includes four types of MR sequence input images: T1 image (T1n), T1 weighted image (T1c), T2 weighted image (T2w), T2 fluid attenuated inversion recovery sequence (T2f) and their corresponding segmentation label images. Select the T1c sequence image, use Python programming code to slice the input image of the sequence MR and their corresponding segmentation label images, and use Python programming code again to merge the three label masks of the entire tumor area, enhanced tumor area and tumor core area into one label mask, and then save it as a 2D image to make a segmentation dataset;

[0066] Furthermore, 20% of the split data set is randomly divided as a validation data set, 10% as a test data set, and the remaining 70% as a training data set.

[0067] Step 2: Based on the training data set and the validation data set, perform data augmentation operations such as cropping at random size and random position, rotating at random degrees, and randomly flipping on each image to enhance data diversity, including but not limited to cropping at random size and position between 80% and 100% of the original image area, and rotating the image at any degree between -90 degrees and 90 degrees;

[0068] Furthermore, since the constructed 2D brain tumor segmentation model based on fuzzy convolution combined with Transformer has fixed requirements on the size of the training dataset and the validation dataset, the final size of the training dataset and the validation dataset after the data augmentation operation is controlled to be 128×128×1 or 128×128×3.

[0069] Step 3: Construct a 2D brain tumor segmentation model based on fuzzy convolution combined with Transformer, which includes a fuzzy learning module, a convolutional neural network, an encoder, a decoder, and a gated weight controller. The fuzzy learning module includes a fuzzy membership function and a fuzzy rule. The convolutional neural network is inspired by ResNet and is designed and constructed. The encoder includes multiple convolutional layers and a gated axial self-attention module. The decoder includes multiple convolutional layers and multiple deconvolutional layers. Multiple residual connections are provided between the encoder and the decoder.

[0070] Furthermore, the present embodiment is composed of two branches, namely, a fuzzy convolutional neural network branch and a gated axial self-attention mechanism branch. The two branches concurrently perform model calculations and fuse the results at the end. After fusion, a convolution operation is performed to obtain the final output result. The size of the convolution kernel is 1×1, and the step size is also 1. The fuzzy convolutional neural network branch includes a fuzzy learning module and a convolutional neural network. The gated axial self-attention mechanism branch includes an encoder, a decoder, and a gated weight controller.

[0071] Furthermore, the fuzzy learning module includes fuzzy membership functions and fuzzy rules. At the input end of the fuzzy learning module, M fuzzy membership functions are independently applied to each channel of the feature map. Although the fuzzy membership functions remain consistent within the same channel, they are flexibly adjusted between different channels according to the characteristics to adapt to the variable textures and structures in the image. For each feature point in each channel, each fuzzy membership function calculates a fuzzy value for it, and the fuzzy value describes the degree to which it belongs to a certain set. All fuzzy membership functions use Gaussian functions in the following form:

[0072]

[0073] Among them, W and H refer to the width and height of the feature map respectively, C refers to the number of channels of the feature map, M is the number of fuzzy membership functions, which is the only hyperparameter that needs to be preset, and (x, y) is the feature point P in channel c. x,y,c The position coordinates, σ m,c and μ m,c is the standard deviation and mean of the mth Gaussian membership function, L x,y,m,c is the fuzzy value of the mth fuzzy output of the feature point at (x, y) in channel c, σ m,c and μ m,c It is the data information that is continuously learned as the model is trained;

[0074] Finally, for each feature point, the M fuzzy outputs are fused using fuzzy logic “AND” to obtain the final fuzzy result L x,y,c , the process is as follows:

[0075]

[0076] Furthermore, the convolutional neural network is inspired by ResNet and constructed. The downsampling structure in the convolutional neural network includes a three-layer network architecture. The first layer of the network architecture is sequentially composed of a convolution layer, a batch normalization operation, and a ReLU activation function. The size of the convolution kernel of the convolution layer is 7×7, and the moving step is 2. The influence of the convolution operation on the size of the feature map is formulated as shown in the following formula:

[0077] Wout =(W in -k+2P) / S+1

[0078] Among them, W out is the size of the output feature map, W in is the size of the input medical image, k is the size of the convolution kernel, P is the size of the zero padding, and S is the size of the convolution kernel movement step;

[0079] The fuzzy learning module is set after the first layer of the network architecture in the downsampling structure. The result obtained by the fuzzy learning module is fused with the result obtained by the first layer of the network architecture, and then a batch normalization operation is performed, and then a ReLU activation function is added;

[0080] The second-layer network architecture consists of a maximum pooling layer and three residual modules. After the maximum pooling operation with a pooling operator size of 3×3 and a moving step of 2, the height and width of the feature map are both reduced to half of the original. The maximum pooling operation is understood according to the following formula:

[0081] A [i,j] =max(x[i×stride_h:i×stride_h+h,j×stride_w:j×stride_w+w])

[0082] Among them, A [i,j] Represents the information of the element in the i-th row and j-th column of the output feature map, x represents the input feature map, stride_h and stride_w represent the moving steps in the vertical and horizontal directions respectively, and h and w represent the height and width of the pooling operator;

[0083] The three residual modules in the second-layer network architecture work in series. The working process of each residual module is that the original feature map passes through the convolution layer with a convolution kernel size of 1×1 and a moving step size of 1, the batch normalization function and the ReLU activation function in sequence, and then passes through the convolution layer with a convolution kernel size of 3×3 and a moving step size of 1, the batch normalization function and the ReLU activation function in sequence, and finally passes through the convolution layer with a convolution kernel size of 1×1 and a moving step size of 1 and the ReLU activation function in sequence to obtain the intermediate process feature map, which is residually connected with the original feature map once, and then passes through a ReLU activation function to obtain the final output feature map;

[0084] The third-layer network architecture consists of four consecutive identical residual modules. The module structure of the residual module is almost the same as that of the second-layer network architecture. Only the calculation of the number of channels of the feature map during the three convolution operations is changed.

[0085] The upsampling structure contains three upsampling modules. Each upsampling module performs a deconvolution operation to double the size of the feature map, a convolution operation with a convolution kernel size of 3×3 and a moving step of 1, a batch normalization operation, and a ReLU activation function. Between the downsampling structure and the upsampling structure, the feature map of the same size is skipped and then input into the next upsampling module. After the upsampling structure, there is a batch normalization operation.

[0086] Furthermore, the encoder includes multiple convolutional layers and a gated axial self-attention module. After a downsampling operation, the size is The feature map of is input to the first encoder constructed in this embodiment. In the first encoder, the feature map first undergoes a convolution operation with a convolution kernel size of 1×1 and a moving step size of 1 to increase the number of channels of the input feature map, and then a batch normalization operation is performed. Subsequently, the size is The feature map is sent to the gated axial self-attention calculation, where C1 represents the number of channels. When calculating the similarity of pixels, the self-attention only operates on the row and column of the height axis and the width axis. The calculation method and operation process of the two are exactly the same. The mathematical formula for calculating the gated axial self-attention on the height axis is as follows:

[0087]

[0088] Among them, q ij represents a query at any position i∈{1,...,H} and j∈{1,...,W}, Indicates q ij The transpose of k ih and v ih Represent the keys and values ​​at positions i∈{1,...,H} and h∈{1,...,W} respectively, and then calculate the similarity between the information at position (i,j) and the information at position (i,h) through the softmax() function. Then, the position deviation information in the axial direction is added. and Added learnable weight parameter G Q , G K , G V1 and G V2 ;

[0089] After the gated axial self-attention calculation is completed, a convolution operation with a kernel size of 1×1 and a moving step of 1 and a batch normalization operation are performed again. Then, the information of the current feature map is residually connected with the information of the feature map before being sent to the encoder as the final output of the first encoder, which is also the starting input of the second encoder.

[0090] The calculation method and operation process of the second encoder are similar to those of the first encoder. Only the calculation amount of the gated axial self-attention is changed, that is, the calculation is performed with a step size of 2 on the height axis and the width axis, and the final size is The feature map information of , where C2 represents the number of channels, and the calculation method and operation process of the third encoder are the same as those of the second encoder, so the final size is Feature map information, where C3 represents the number of channels;

[0091] Furthermore, the decoder first performs a convolution operation with a convolution kernel size of 3×3 and a moving step size of 1, then performs a deconvolution operation to double the size of the feature map, and finally uses a ReLU activation function as a whole decoder. The number of decoders corresponds to the number of encoders. The step size of the convolution operation in the first decoder is 2, and the step sizes of the convolution operations in the second and third decoders are both 1.

[0092] Between the encoder and the decoder, the feature maps of the same size after being processed by the encoder and the decoder are skipped and then input into the next decoder. After the decoder structure, there is a batch normalization operation.

[0093] Furthermore, the gated weight controller is a first-order parameter that can be continuously learned as the model is trained. It is set after the batch normalization operation, and it dynamically controls the proportion of learning global context information and learning local detail information. The mathematical formula of the working principle of the gated weight controller is as follows:

[0094]

[0095] The z in the equation end Represents the result after the gated weight controller completes the operation, G w It is the door weight controller. Parameter information of the feature map after batch normalization operation

[0096] Step 4: Train a 2D brain tumor segmentation model based on fuzzy convolution combined with Transformer, use the Adam optimization algorithm to train the model parameters, set the number of model training times, and set the mean intersection over union (MIoU) and F1-score (F1-score) to measure the performance of the model.

[0097] In order to further improve the training effect, the learning rate η is gradually reduced during the training process. This embodiment adopts a learning rate adjustment strategy based on exponential decay, and its formula is as follows:

[0098]

[0099] Among them, η t is the learning rate in the tth round of training, η0 is the initial learning rate, γ is the decay rate, and T is the total number of training rounds.

[0100] Step 5: In a certain round of training, when the above two evaluation indicators reach the threshold on both the training data set and the validation data set, the parameter model obtained in this round of training is taken as the final parameter model to perform 2D segmentation on the images in the test data set to evaluate the authenticity of the model.

[0101] The following table compares the segmentation effects of the model of the present invention and other advanced models:

[0102] U-Net++ Medical transformer The present invention MIoU / % 0.6052 0.6716 0.6768 F1-score / % 0.7267 0.7853 0.7892

[0103] It can be seen that the model of the present invention has higher accuracy, better effect and practical application potential.

[0104] Example 2

[0105] The technical solution provided in this embodiment is a 2D brain tumor segmentation method based on fuzzy convolution combined with Transformer, and the method comprises the following steps:

[0106] Step 1: Download the BraTS-GLI 2023 dataset from the official Brain Tumor Segmentation website. The dataset includes four types of MR sequence input images: T1 image (T1n), T1 weighted image (T1c), T2 weighted image (T2w), T2 fluid attenuated inversion recovery sequence (T2f) and their corresponding segmentation label images. Select the T2f sequence image, use Python programming code to slice the input image of the sequence MR and their corresponding segmentation label images, and use Python programming code again to merge the three label masks of the entire tumor area, enhanced tumor area and tumor core area into one label mask, and then save it as a 2D image to make a segmentation dataset;

[0107] Furthermore, 20% of the split data set is randomly divided as a validation data set, 10% as a test data set, and the remaining 70% as a training data set.

[0108] Step 2: Based on the training data set and the validation data set, perform data augmentation operations such as cropping at random size and random position, rotating at random degrees, and randomly flipping on each image to enhance data diversity, including but not limited to cropping at random size and position between 80% and 100% of the original image area, and rotating the image at any degree between -90 degrees and 90 degrees;

[0109] Furthermore, since the constructed 2D brain tumor segmentation model based on fuzzy convolution combined with Transformer has fixed requirements on the size of the training dataset and the validation dataset, the final size of the training dataset and the validation dataset after the data augmentation operation is controlled to be 128×128×1 or 128×128×3.

[0110] Step 3: Construct a 2D brain tumor segmentation model based on fuzzy convolution combined with Transformer, which includes a fuzzy learning module, a convolutional neural network, an encoder, a decoder, and a gated weight controller. The fuzzy learning module includes a fuzzy membership function and a fuzzy rule. The convolutional neural network is inspired by ResNet and is designed and constructed. The encoder includes multiple convolutional layers and a gated axial self-attention module. The decoder includes multiple convolutional layers and multiple deconvolutional layers. Multiple residual connections are provided between the encoder and the decoder.

[0111] Furthermore, the present embodiment is composed of two branches, namely, a fuzzy convolutional neural network branch and a gated axial self-attention mechanism branch. The two branches concurrently perform model calculations and fuse the results at the end. After fusion, a convolution operation is performed to obtain the final output result. The size of the convolution kernel is 1×1, and the step size is also 1. The fuzzy convolutional neural network branch includes a fuzzy learning module and a convolutional neural network. The gated axial self-attention mechanism branch includes an encoder, a decoder, and a gated weight controller.

[0112] Furthermore, the fuzzy learning module includes fuzzy membership functions and fuzzy rules. At the input end of the fuzzy learning module, M fuzzy membership functions are independently applied to each channel of the feature map. Although the fuzzy membership functions remain consistent within the same channel, they are flexibly adjusted between different channels according to the characteristics to adapt to the variable textures and structures in the image. For each feature point in each channel, each fuzzy membership function calculates a fuzzy value for it, and the fuzzy value describes the degree to which it belongs to a certain set. All fuzzy membership functions use Gaussian functions in the following form:

[0113]

[0114] Among them, W and H refer to the width and height of the feature map respectively, C refers to the number of channels of the feature map, M is the number of fuzzy membership functions, which is the only hyperparameter that needs to be preset, and (x, y) is the feature point P in channel c. x,y,c The position coordinates, σ m,c and μ m,c is the standard deviation and mean of the mth Gaussian membership function, L x,y,m,c is the fuzzy value of the mth fuzzy output of the feature point at (x, y) in channel c, σ m,c and μm,c It is the data information that is continuously learned as the model is trained;

[0115] Finally, for each feature point, the M fuzzy outputs are fused using fuzzy logic “AND” to obtain the final fuzzy result L x,y,c , the process is as follows:

[0116]

[0117] Furthermore, the convolutional neural network is inspired by ResNet and constructed. The downsampling structure in the convolutional neural network includes a three-layer network architecture. The first layer of the network architecture is sequentially composed of a convolution layer, a batch normalization operation, and a ReLU activation function. The size of the convolution kernel of the convolution layer is 7×7, and the moving step is 2. The influence of the convolution operation on the size of the feature map is formulated as shown in the following formula:

[0118] W out =(W in -k+2P) / S+1

[0119] Among them, W out is the size of the output feature map, W in is the size of the input medical image, k is the size of the convolution kernel, P is the size of the zero padding, and S is the size of the convolution kernel movement step;

[0120] The fuzzy learning module is set after the first layer of the network architecture in the downsampling structure. The result obtained by the fuzzy learning module is fused with the result obtained by the first layer of the network architecture, and then a batch normalization operation is performed, and then a ReLU activation function is added;

[0121] The second-layer network architecture consists of a maximum pooling layer and three residual modules. After the maximum pooling operation with a pooling operator size of 3×3 and a moving step of 2, the height and width of the feature map are both reduced to half of the original. The maximum pooling operation is understood according to the following formula:

[0122] A [i,j] =max(x[i×stride_h:i×stride_h+h,j×stride_w:j×stride_w+w])

[0123] Among them, A [i,j] Represents the information of the element in the i-th row and j-th column of the output feature map, x represents the input feature map, stride_h and stride_w represent the moving steps in the vertical and horizontal directions respectively, and h and w represent the height and width of the pooling operator;

[0124] The three residual modules in the second-layer network architecture work in series. The working process of each residual module is that the original feature map passes through the convolution layer with a convolution kernel size of 1×1 and a moving step size of 1, the batch normalization function and the ReLU activation function in sequence, and then passes through the convolution layer with a convolution kernel size of 3×3 and a moving step size of 1, the batch normalization function and the ReLU activation function in sequence, and finally passes through the convolution layer with a convolution kernel size of 1×1 and a moving step size of 1 and the ReLU activation function in sequence to obtain the intermediate process feature map, which is residually connected with the original feature map once, and then passes through a ReLU activation function to obtain the final output feature map;

[0125] The third-layer network architecture consists of four consecutive identical residual modules. The module structure of the residual module is almost the same as that of the second-layer network architecture. Only the calculation of the number of channels of the feature map during the three convolution operations is changed.

[0126] The upsampling structure contains three upsampling modules. Each upsampling module performs a deconvolution operation to double the size of the feature map, a convolution operation with a convolution kernel size of 3×3 and a moving step of 1, a batch normalization operation, and a ReLU activation function. Between the downsampling structure and the upsampling structure, the feature map of the same size is skipped and then input into the next upsampling module. After the upsampling structure, there is a batch normalization operation.

[0127] Furthermore, the encoder includes multiple convolutional layers and a gated axial self-attention module. After a downsampling operation, the size is The feature map of is input to the first encoder constructed in this embodiment. In the first encoder, the feature map first undergoes a convolution operation with a convolution kernel size of 1×1 and a moving step size of 1 to increase the number of channels of the input feature map, and then a batch normalization operation is performed. Subsequently, the size is The feature map is sent to the gated axial self-attention calculation, where C1 represents the number of channels. When calculating the similarity of pixels, the self-attention only operates on the row and column of the height axis and the width axis. The calculation method and operation process of the two are exactly the same. The mathematical formula for calculating the gated axial self-attention on the height axis is as follows:

[0128]

[0129] Among them, q ij represents a query at any position i∈{1,...,H} and j∈{1,...,W}, Indicates q ij The transpose of k ih and v ihRepresent the keys and values ​​at positions i∈{1,...,H} and h∈{1,...,W} respectively, and then calculate the similarity between the information at position (i,j) and the information at position (i,h) through the softmax() function. Then, the position deviation information in the axial direction is added. and Added learnable weight parameter G Q , G K , G V1 and G V2 ;

[0130] After the gated axial self-attention calculation is completed, a convolution operation with a kernel size of 1×1 and a moving step of 1 and a batch normalization operation are performed again. Then, the information of the current feature map is residually connected with the information of the feature map before being sent to the encoder as the final output of the first encoder, which is also the starting input of the second encoder.

[0131] The calculation method and operation process of the second encoder are similar to those of the first encoder. Only the calculation amount of the gated axial self-attention is changed, that is, the calculation is performed with a step size of 2 on the height axis and the width axis, and the final size is The feature map information of , where C2 represents the number of channels, and the calculation method and operation process of the third encoder are the same as those of the second encoder, so the final size is Feature map information, where C3 represents the number of channels;

[0132] Furthermore, the decoder first performs a convolution operation with a convolution kernel size of 3×3 and a moving step size of 1, then performs a deconvolution operation to double the size of the feature map, and finally uses a ReLU activation function as a whole decoder. The number of decoders corresponds to the number of encoders. The step size of the convolution operation in the first decoder is 2, and the step sizes of the convolution operations in the second and third decoders are both 1.

[0133] Between the encoder and the decoder, the feature maps of the same size after being processed by the encoder and the decoder are skipped and then input into the next decoder. After the decoder structure, there is a batch normalization operation.

[0134] Furthermore, the gated weight controller is a first-order parameter that can be continuously learned as the model is trained. It is set after the batch normalization operation, and it dynamically controls the proportion of learning global context information and learning local detail information. The mathematical formula of the working principle of the gated weight controller is as follows:

[0135]

[0136] The z in the equationend Represents the result after the gated weight controller completes the operation, G w It is the door weight controller. Parameter information of the feature map after batch normalization operation

[0137] Step 4: Train a 2D brain tumor segmentation model based on fuzzy convolution combined with Transformer, use the Adam optimization algorithm to train the model parameters, set the number of model training times, and set the mean intersection over union (MIoU) and F1 score (F1-score) to measure the performance of the model;

[0138] In order to further improve the training effect, the learning rate η is gradually reduced during the training process. This embodiment adopts a learning rate adjustment strategy based on exponential decay, and its formula is as follows:

[0139]

[0140] Among them, η t is the learning rate in the tth round of training, η0 is the initial learning rate, γ is the decay rate, and T is the total number of training rounds.

[0141] Step 5: In a certain round of training, when the above two evaluation indicators reach the threshold on both the training data set and the validation data set, the parameter model obtained in this round of training is taken as the final parameter model to perform 2D segmentation on the images in the test data set to evaluate the authenticity of the model.

[0142] The following table compares the segmentation effects of the model of the present invention and other advanced models:

[0143] U-Net++ Medical transformer The present invention MIoU / % 0.7155 0.8128 0.8211 F1-score / % 0.8141 0.8871 0.8965

[0144] It can be seen that the model of the present invention has higher accuracy, better effect and practical application potential.

[0145] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A 2D brain tumor segmentation method based on fuzzy convolution combined with Transformer, characterized in that: The following steps are involved: S1: Brain tumor images exist in both 2D and 3D forms. For 3D images, the programming code slices them into 2D images to obtain the training data set, validation data set, and test data set required for the 2D segmentation task. For 2D images, they are directly divided into training data set, verification data set and test data set; S2: Add a series of data augmentation operations such as random cropping, random rotation, and random flipping to the training dataset and the validation dataset; S3: Construct a 2D brain tumor segmentation model based on fuzzy convolution combined with Transformer, which includes a fuzzy learning module, a convolutional neural network, an encoder, a decoder, and a gated weight controller. The fuzzy learning module includes a fuzzy membership function and a fuzzy rule. The convolutional neural network is inspired by ResNet and is designed and constructed. The encoder includes multiple convolutional layers and a gated axial self-attention module. The decoder includes multiple convolutional layers and multiple deconvolutional layers. Multiple residual connections are set between the encoder and the decoder. S4: Train a 2D brain tumor segmentation model based on fuzzy convolution combined with Transformer, use the Adam optimization algorithm to train the model parameters, set the number of model training times, and set the mean intersection over union (MIoU) and F1 score (F1-score) to measure the performance of the model; S5: In a certain round of training, when the above two evaluation indicators reach the threshold value on both the training data set and the validation data set, the parameter model obtained in this round of training is taken as the final parameter model for training to perform 2D segmentation on the images in the test data set to evaluate the authenticity of the model.

2. The 2D brain tumor segmentation method based on fuzzy convolution combined with Transformer according to claim 1, characterized in that: The step S1 comprises the following steps: Step S1.1: Retrieve and download the brain tumor image segmentation dataset from the official platform of medical public datasets; Step S1.2: When the data set is a 3D image, the programming code slices it into 2D images as an image segmentation data set, and when the data set is a 2D image, it is directly used as an image segmentation data set; Step S1.3: Randomly divide 20% of the image segmentation dataset as a validation dataset, 10% as a test dataset, and the remaining 70% as a training dataset.

3. The 2D brain tumor segmentation method based on fuzzy convolution combined with Transformer according to claim 1, characterized in that: The step S2 comprises the following steps: Step S2.1: Based on the training data set and the validation data set, perform data augmentation operations such as cropping with random size and random position, rotating with random degrees, and flipping with random degrees on each image to enhance data diversity; Step S2.2: The constructed 2D brain tumor segmentation model based on fuzzy convolution combined with Transformer has fixed requirements on the size of the training dataset and the validation dataset, and controls the final size of the training dataset and the validation dataset after the data augmentation operation to be 128×128×1 or 128×128×3.

4. The 2D brain tumor segmentation method based on fuzzy convolution combined with Transformer according to claim 1, characterized in that: The step S3 comprises the following steps: Step S3.1: It is composed of two branches, namely, a fuzzy convolutional neural network branch and a gated axial self-attention mechanism branch. The two branches perform model calculations concurrently and fuse the results at the end. After fusion, a convolution operation is performed to obtain the final output result. The size of the convolution kernel is 1×1 and the step size is also 1. The fuzzy convolutional neural network branch includes a fuzzy learning module and a convolutional neural network. The gated axial self-attention mechanism branch includes an encoder, a decoder, and a gated weight controller. Step S3.2: The fuzzy learning module described in step S3 includes fuzzy membership functions and fuzzy rules. At the input end of the fuzzy learning module, M fuzzy membership functions are independently applied to each channel of the feature map. Although the fuzzy membership function remains consistent within the same channel, it is flexibly adjusted between different channels according to the characteristics to adapt to the variable textures and structures in the image. For each feature point in each channel, each fuzzy membership function calculates a fuzzy value for it, and the fuzzy value describes the degree to which it belongs to a certain set. All fuzzy membership functions use Gaussian functions in the following form: Among them, W and H refer to the width and height of the feature map respectively, C refers to the number of channels of the feature map, M is the number of fuzzy membership functions, which is the only hyperparameter that needs to be preset, and (x, y) is the feature point P in channel c. x,y,c The position coordinates, σ m,c and μ m,c is the standard deviation and mean of the mth Gaussian membership function, L x,y,m,c is the fuzzy value of the mth fuzzy output of the feature point at (x, y) in channel c, σ m,c and μ m,c It is the data information that is continuously learned as the model is trained; Finally, for each feature point, the M fuzzy outputs are fused using fuzzy logic "AND" to obtain the final fuzzy result L x,y,c , the process is as follows: Step S3.3: The convolutional neural network described in step S3 is inspired by ResNet and constructed. The downsampling structure in the convolutional neural network includes a three-layer network architecture. The first layer of the network architecture is sequentially composed of a convolutional layer, a batch normalization operation, and a ReLU activation function. The size of the convolution kernel of the convolutional layer is 7×7, and the step size of the movement is 2. The effect of the convolution operation on the size of the feature map is formulated as shown in the following formula: W out =(W in -k+2P) / S+1 Among them, W out is the size of the output feature map, W in is the size of the input medical image, k is the size of the convolution kernel, P is the size of the zero padding, and S is the size of the convolution kernel movement step; The fuzzy learning module is set after the first layer of the network architecture in the downsampling structure. The result obtained by the fuzzy learning module is fused with the result obtained by the first layer of the network architecture, and then a batch normalization operation is performed, and then a ReLU activation function is added; The second-layer network architecture consists of a maximum pooling layer and three residual modules. After the maximum pooling operation with a pooling operator size of 3×3 and a moving step of 2, the height and width of the feature map are both reduced to half of the original. The maximum pooling operation is understood according to the following formula: A [i,j] =max(x[i×stride_h:i×stride_h+h,j×stride_w:j×stride_w+w]) Among them, A [i,j] Represents the information of the element in the i-th row and j-th column of the output feature map, x represents the input feature map, stride_h and stride_w represent the moving steps in the vertical and horizontal directions respectively, and h and w represent the height and width of the pooling operator; The three residual modules in the second-layer network architecture work in series. The working process of each residual module is that the original feature map passes through the convolution layer with a convolution kernel size of 1×1 and a moving step size of 1, the batch normalization function and the ReLU activation function in sequence, and then passes through the convolution layer with a convolution kernel size of 3×3 and a moving step size of 1, the batch normalization function and the ReLU activation function in sequence, and finally passes through the convolution layer with a convolution kernel size of 1×1 and a moving step size of 1 and the ReLU activation function in sequence to obtain the intermediate process feature map, which is residually connected with the original feature map once, and then passes through a ReLU activation function to obtain the final output feature map; The third-layer network architecture consists of four consecutive identical residual modules. The module structure of the residual module is almost the same as that of the second-layer network architecture. Only the calculation of the number of channels of the feature map during the three convolution operations is changed. The upsampling structure contains three upsampling modules. Each upsampling module performs a deconvolution operation to double the size of the feature map, a convolution operation with a convolution kernel size of 3×3 and a moving step of 1, a batch normalization operation, and a ReLU activation function. Between the downsampling structure and the upsampling structure, the feature map of the same size is skipped and then input into the next upsampling module. After the upsampling structure, there is a batch normalization operation. Step S3.4: The encoder described in step S3 includes multiple convolutional layers and a gated axial self-attention module. After a downsampling operation, the size is The feature map of is input into the first encoder constructed by the present invention. In the first encoder, the feature map first undergoes a convolution operation with a convolution kernel size of 1×1 and a moving step size of 1 to increase the number of channels of the input feature map, and then a batch normalization operation is performed. Subsequently, the size is The feature map is sent to the gated axial self-attention calculation, where C1 represents the number of channels. When calculating the similarity of pixels, the self-attention only operates on the row and column of the height axis and the width axis. The calculation method and operation process of the two are exactly the same. The mathematical formula for calculating the gated axial self-attention on the height axis is as follows: Among them, q ij represents a query at any position i∈{1,...,H} and j∈{1,...,W}, Indicates q ij The transpose of k ih and v ih Represent the keys and values ​​at positions i∈{1,...,H} and h∈{1,...,W} respectively, and then calculate the similarity between the information at position (i,j) and the information at position (i,h) through the softmax() function. Then, the position deviation information in the axial direction is added. and Added learnable weight parameter G Q , G K , G V1 and G V2 ; After the gated axial self-attention calculation is completed, a convolution operation with a kernel size of 1×1 and a moving step of 1 and a batch normalization operation are performed again. Then, the information of the current feature map is residually connected with the information of the feature map before being sent to the encoder as the final output of the first encoder, which is also the starting input of the second encoder. The calculation method and operation process of the second encoder are similar to those of the first encoder, only the calculation amount of the gated axial self-attention is changed, that is, the calculation is performed with a step size of 2 on the height axis and the width axis, and the final size is The feature map information of , where C2 represents the number of channels, and the calculation method and operation process of the third encoder are the same as those of the second encoder, so the final size is Feature map information, where C3 represents the number of channels; Step S3.5: The decoder described in step S3 first performs a convolution operation with a convolution kernel size of 3×3 and a moving step size of 1, then performs a deconvolution operation to double the size of the feature map, and finally uses a ReLU activation function as a whole decoder. The number of decoders corresponds to the number of encoders. The step size of the convolution operation in the first decoder is 2, and the step sizes of the convolution operations in the second and third decoders are both 1; Between the encoder and the decoder, the feature maps of the same size after being processed by the encoder and the decoder are skipped and then input into the next decoder. After the decoder structure, there is a batch normalization operation. Step S3.6: The gated weight controller described in step S3 is a first-order parameter that can be continuously learned as the model is trained. It is set after the batch normalization operation, and it dynamically controls the proportion of learning global context information and learning local detail information. The mathematical formula of the working principle of the gated weight controller is as follows: The z in the equation end Represents the result after the gated weight controller completes the operation, G w It is the door weight controller. It is the parameter information of the feature map after the batch normalization operation.

5. The 2D brain tumor segmentation method based on fuzzy convolution combined with Transformer according to claim 1, characterized in that: The step S4 comprises the following steps: Step S4.1: loading the optimization algorithm into the 2D brain tumor segmentation model based on fuzzy convolution combined with Transformer, and using the Adam optimization algorithm to train the model parameters to optimize the model parameters; Step S4.2: To further improve the training effect, the learning rate η is gradually reduced during the training process, and a learning rate adjustment strategy based on exponential decay is adopted, and the formula is as follows: Among them, η t is the learning rate in the tth round of training, η0 is the initial learning rate, γ is the decay rate, and T is the total number of training rounds.

Images