Medical image segmentation method, device and equipment and computer readable storage medium

Abstract

The invention relates to the field of image processing, and discloses a medical image segmentation method, device and equipment and a computer readable storage medium. The method comprises the steps that a to-be-segmented medical image is acquired, the to-be-segmented medical image is input to a deep learning model, and the deep learning model comprises an encoder, an attention multi-scale feature fusion module, a decoder and a tail block; performing down-sampling and tensor reconstruction processing on the to-be-segmented medical image through an encoder to obtain a multi-scale feature image and an output attention tensor; performing feature fusion processing on the multi-scale feature image through an attention multi-scale feature fusion module to obtain a fused feature image; inputting the output attention tensor and the fused feature image into a decoder for up-sampling and tensor reconstruction processing to obtain a segmented image; and performing tensor reconstruction processing on the segmented image through the tail block to output a segmented target medical image. The accuracy of medical image segmentation can be improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a medical image segmentation method, device, equipment and computer-readable storage medium. Background Art

[0002] Medical image segmentation plays a vital role in medical imaging diagnosis and treatment, and its main goal is to separate specific tissues or organs in the image from the background. This process is of great significance in the quantitative analysis of medical images because it can effectively extract the shape and spatial information of the target area and automatically identify the target area in the medical image pixel by pixel through machine vision.

[0003] At present, deep learning has been gradually applied to the field of medical image segmentation. Compared with traditional machine learning methods, deep learning performs better in processing large-scale data and complex tasks. Deep learning reduces human intervention, has the ability to learn more abstract and high-dimensional features, and shows excellent feature learning potential. Among deep learning models, the state space model (SSM) shows potential huge advantages in medical image segmentation. SSM not only shows excellent performance in modeling long-range interdependencies, but also maintains the complexity of linear calculation and storage. However, current SSM-based medical image segmentation methods, such as VM-UNet, do not mix multi-scale features in the jump connection part, resulting in low accuracy of medical image segmentation. Summary of the invention

[0004] The main purpose of the present invention is to provide a medical image segmentation method, device, equipment and computer-readable storage medium, aiming to improve the accuracy of medical image segmentation.

[0005] A first aspect of the present invention provides a medical image segmentation method, the medical image segmentation method comprising:

[0006] Acquire a medical image to be segmented, and input the medical image to be segmented into a deep learning model, wherein the deep learning model includes an encoder, an attention multi-scale feature fusion module, a decoder, and a tail block;

[0007] Downsampling and tensor reconstruction processing are performed on the medical image to be segmented by the encoder to obtain a multi-scale feature image and an output attention tensor;

[0008] Performing feature fusion processing on the multi-scale feature image through the attention multi-scale feature fusion module to obtain a fused feature image;

[0009] Inputting the output attention tensor and the fused feature image into the decoder for upsampling and tensor reconstruction processing to obtain a segmented image;

[0010] The segmented image is subjected to tensor reconstruction processing by the tail block to output a segmented target medical image.

[0011] Optionally, the tensor reconstruction process includes:

[0012] Get the input tensor;

[0013] Scanning the input tensor once in the horizontal direction and once in the vertical direction to obtain a horizontal sequence and a vertical sequence;

[0014] Inputting the horizontal sequence and the vertical sequence into a preset state space model to obtain a horizontal attention sequence and a vertical attention sequence;

[0015] After reshaping and concatenating the horizontal attention sequence and the vertical attention sequence, the resultant sequences are input into a preset hybrid feedforward network;

[0016] The positional relationship between the horizontal attention sequence and the vertical attention sequence is modeled through the hybrid feedforward network, and a first attention tensor is output.

[0017] Optionally, the tensor reconstruction process further includes:

[0018] Performing frequency domain feature extraction on the input tensor to obtain a frequency domain tensor;

[0019] Output a second attention tensor according to the first attention tensor and the frequency domain tensor.

[0020] Optionally, the extracting frequency domain features from the input tensor to obtain a frequency domain tensor includes:

[0021] Performing a real Fourier transform on the input tensor to obtain frequency domain features;

[0022] After matrix multiplication of the frequency domain feature and the preset complex weight parameter, the result feature is obtained by adding the preset complex bias parameter;

[0023] Perform a real inverse Fourier transform on the result features to obtain a frequency domain tensor.

[0024] Optionally, the attention multi-scale feature fusion module includes an attention bridge and a multi-scale feature fusion module, and the step of performing feature fusion processing on the multi-scale feature image through the attention multi-scale feature fusion module to obtain a fused feature image includes:

[0025] Inferring the channel attention and spatial attention of the multi-scale feature image through the attention bridge;

[0026] Obtaining a multi-scale output feature tensor according to the channel attention, the spatial attention and the multi-scale feature image;

[0027] The multi-scale output feature tensor is input into the multi-scale feature fusion module to obtain a fused feature image.

[0028] Optionally, the acquiring a multi-scale output feature tensor according to the channel attention, the spatial attention and the multi-scale feature image includes:

[0029] Performing matrix multiplication on the channel attention and the multi-scale feature image to obtain an intermediate tensor;

[0030] After matrix multiplication of the intermediate tensor and the spatial attention, matrix addition is performed with the multi-scale feature image to obtain a multi-scale output feature tensor.

[0031] Optionally, the step of inputting the multi-scale output feature tensor into the multi-scale feature fusion module to obtain a fused feature image comprises:

[0032] In the multi-scale feature fusion module, a unit tensor is generated according to a preset anchor tensor;

[0033] Performing a size conversion on the multi-scale output feature tensor to obtain a feature tensor of the same size as the anchor tensor;

[0034] Performing deep convolution processing on the feature tensor having the same size as the anchor tensor;

[0035] The feature tensor after deep convolution processing is matrix multiplied with the unit tensor to obtain a fused feature image.

[0036] A second aspect of the present invention provides a medical image segmentation device, the medical image segmentation device comprising:

[0037] An input module, used for acquiring a medical image to be segmented, and inputting the medical image to be segmented into a deep learning model, wherein the deep learning model includes an encoder, an attention multi-scale feature fusion module, a decoder, and a tail block;

[0038] An encoding module, used for performing downsampling and tensor reconstruction processing on the medical image to be segmented through the encoder to obtain a multi-scale feature image and an output attention tensor;

[0039] A fusion processing module, used for performing feature fusion processing on the multi-scale feature image through the attention multi-scale feature fusion module to obtain a fused feature image;

[0040] A decoding module, used for inputting the output attention tensor and the fused feature image into the decoder for upsampling and tensor reconstruction processing to obtain a segmented image;

[0041] The output module is used to perform tensor reconstruction processing on the segmented image through the tail block to output a segmented target medical image.

[0042] Optionally, the encoding module, the decoding module and the output module are further used for:

[0043] Get the input tensor;

[0044] Scanning the input tensor once in the horizontal direction and once in the vertical direction to obtain a horizontal sequence and a vertical sequence;

[0045] Inputting the horizontal sequence and the vertical sequence into a preset state space model to obtain a horizontal attention sequence and a vertical attention sequence;

[0046] After reshaping and concatenating the horizontal attention sequence and the vertical attention sequence, the resultant sequences are input into a preset hybrid feedforward network;

[0047] The positional relationship between the horizontal attention sequence and the vertical attention sequence is modeled through the hybrid feedforward network, and a first attention tensor is output.

[0048] Optionally, the encoding module, the decoding module and the output module are further used for:

[0049] Performing frequency domain feature extraction on the input tensor to obtain a frequency domain tensor;

[0050] Output a second attention tensor according to the first attention tensor and the frequency domain tensor.

[0051] Optionally, the encoding module, the decoding module and the output module are further used for:

[0052] Performing a real Fourier transform on the input tensor to obtain frequency domain features;

[0053] After matrix multiplication of the frequency domain feature and the preset complex weight parameter, the result feature is obtained by adding the preset complex bias parameter;

[0054] Perform a real inverse Fourier transform on the result features to obtain a frequency domain tensor.

[0055] Optionally, the attention multi-scale feature fusion module includes an attention bridge and a multi-scale feature fusion module, and the fusion processing module is further used for:

[0056] Inferring the channel attention and spatial attention of the multi-scale feature image through the attention bridge;

[0057] Obtaining a multi-scale output feature tensor according to the channel attention, the spatial attention and the multi-scale feature image;

[0058] The multi-scale output feature tensor is input into the multi-scale feature fusion module to obtain a fused feature image.

[0059] Optionally, the fusion processing module is further used for:

[0060] Performing matrix multiplication on the channel attention and the multi-scale feature image to obtain an intermediate tensor;

[0061] After matrix multiplication of the intermediate tensor and the spatial attention, matrix addition is performed with the multi-scale feature image to obtain a multi-scale output feature tensor.

[0062] Optionally, the fusion processing module is further used for:

[0063] In the multi-scale feature fusion module, a unit tensor is generated according to a preset anchor tensor;

[0064] Performing a size conversion on the multi-scale output feature tensor to obtain a feature tensor of the same size as the anchor tensor;

[0065] Performing deep convolution processing on the feature tensor having the same size as the anchor tensor;

[0066] The feature tensor after deep convolution processing is matrix multiplied with the unit tensor to obtain a fused feature image.

[0067] A third aspect of the present invention provides a medical image segmentation device, which includes: a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the above-mentioned medical image segmentation method is implemented.

[0068] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the above-mentioned medical image segmentation method when executed by a processor.

[0069] The present invention obtains a medical image to be segmented, and inputs the medical image to be segmented into a deep learning model, wherein the deep learning model includes an encoder, an attention multi-scale feature fusion module, a decoder and a tail block; the encoder performs downsampling and tensor reconstruction processing on the medical image to be segmented to obtain a multi-scale feature image and an output attention tensor; the attention multi-scale feature fusion module performs feature fusion processing on the multi-scale feature image to obtain a fused feature image; the output attention tensor and the fused feature image are input into the decoder for upsampling and tensor reconstruction processing to obtain a segmented image; the tail block performs tensor reconstruction processing on the segmented image to output a segmented target medical image. The present invention performs tensor reconstruction processing on the image through the encoder, decoder and tail block in the deep learning model, and performs feature fusion processing through the attention multi-scale feature fusion module, thereby improving the accuracy of medical image segmentation compared to the segmentation method lacking multi-scale feature fusion in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 A schematic diagram of a flow chart of an embodiment of a medical image segmentation method of the present invention;

[0071] Figure 2 Schematic diagram of a skin lesion segmentation data set in an embodiment of the medical image segmentation method of the present invention;

[0072] Figure 3 A schematic diagram of the structure of a deep learning model in an embodiment of the medical image segmentation method of the present invention;

[0073] Figure 4 A schematic diagram of a BFSS module in an embodiment of a medical image segmentation method of the present invention;

[0074] Figure 5 A schematic diagram comparing the scanning modes of the BFSS module and the existing visual state space module SS2D in the embodiment of the medical image segmentation method of the present invention;

[0075] Figure 6 A schematic diagram of a frequency gating module in an embodiment of a medical image segmentation method of the present invention;

[0076] Figure 7 It is a structural schematic diagram of an attention multi-scale feature fusion module in an embodiment of the medical image segmentation method of the present invention;

[0077] Figure 8 for Figure 7 Schematic diagram of the structure of the mid-channel attention module;

[0078] Fig. 9 for Figure 7 Schematic diagram of the structure of the spatial attention module;

[0079] Fig.10 A schematic diagram of modules of an embodiment of a medical image segmentation device of the present invention;

[0080] Fig.11 A schematic diagram of the structure of an embodiment of a medical image segmentation device of the present invention;

[0081] Fig.12 The figure is a connection state diagram of a computer-readable storage medium and a processor according to an embodiment of the present invention. DETAILED DESCRIPTION

[0082] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0083] The invention provides a medical image segmentation method.

[0084] The medical image segmentation method of the embodiment of the present invention can be widely used in the medical field, including but not limited to:

[0085] Brain tumor segmentation: Extract features from brain tumor images, and then predict new images based on the trained model to obtain a binary image of the tumor area. This helps in the diagnosis and treatment of brain tumors.

[0086] Digital pathology image segmentation: In the diagnosis of benign and malignant grading of breast cancer, prostate cancer, etc., the morphology and target measurement of tissue primitives are important bases for quantitative analysis. Deep learning technology has achieved breakthrough results through methods such as convolutional neural networks, fully convolutional networks, and encoder-decoder models, improving the accuracy and efficiency of pathology image segmentation.

[0087] Cardiac Image Segmentation: Segment cardiac MRI or CT scan images to quantitatively measure cardiac structures (such as ventricles, atria, heart valves, etc.). This is critical for the diagnosis and treatment planning of heart disease.

[0088] Lung disease segmentation: In lung CT images, deep learning can help segment out lung lobes, tumors, infected areas, etc. This is very useful for lung cancer screening and early diagnosis of other lung diseases.

[0089] Skin lesion segmentation: Segment lesions (such as melanoma, squamous cell carcinoma, etc.) in dermoscopic images to help doctors diagnose and treat skin cancer.

[0090] Fundus image segmentation: Through deep learning, the retina, optic disc, macula and other structures in the fundus image can be segmented for the diagnosis and treatment of diabetic retinopathy.

[0091] Kidney and Liver Segmentation: Segment kidney and liver CT or MRI images to assess lesions such as tumors, cysts, inflammation, etc.

[0092] In conclusion, the application of this method in the field of medical image segmentation can not only help improve diagnostic accuracy but also accelerate clinical decision-making and treatment planning.

[0093] Reference Figure 1 , Figure 1 The following is a flow chart of an embodiment of a medical image segmentation method of the present invention, wherein the method comprises:

[0094] Step S10, obtaining a medical image to be segmented, and inputting the medical image to be segmented into a deep learning model, wherein the deep learning model includes an encoder, an attention multi-scale feature fusion module, a decoder, and a tail block;

[0095] The medical image segmentation method of this embodiment can be applied to medical equipment to assist doctors in analyzing the condition of patients. In this embodiment, the medical image to be segmented is a medical image that needs to be segmented, and the image includes specific physiological tissues or organs, and the segmentation goal is to separate the specific physiological tissues or organs from the image background.

[0096] The deep learning model is a pre-trained medical image segmentation model, which includes an encoder, an attention multi-scale feature fusion module, a decoder and a tail block, wherein the encoder is connected to the decoder, the encoder and the decoder are respectively connected to the attention multi-scale feature fusion module, and the decoder is connected to the tail block. The medical image to be segmented is first input to the encoder, and then passes through the attention multi-scale feature fusion module and the decoder, and finally the segmented target medical image is output through the tail block.

[0097] To facilitate understanding, the training process of the deep learning model will be explained using skin images as an example.

[0098] Reference Figure 2 , Figure 2 Schematic diagram of a skin lesion segmentation dataset in an embodiment of the medical image segmentation method of the present invention. The International Federation of Skin Imaging Challenge Datasets 2017 and 2018 (ISIC17, ISIC18) are two publicly available skin lesion segmentation datasets, containing 2150 and 2694 dermoscopic images with segmentation mask labels, respectively. The left side of the figure shows some sample images of ISIC17 and ISIC18, and the right side shows some sample labeled images of ISIC17 and ISIC18.

[0099] During training, the dataset can be divided into a training set and a test set in a ratio of 7:3. Specifically, for the ISIC17 dataset, the training set contains 1,500 images and the test set contains 650 images. For the ISIC18 dataset, the training set contains 1,886 images and the test set contains 808 images. The evaluation indicators for model training may include mean intersection over union (mIoU), dice similarity coefficient (DSC), accuracy (Acc), sensitivity (Sen), and specificity (Spe). Before the image is input into the segmentation network, the data can be augmented, including random rotation and random horizontal flipping. Data augmentation can provide the model with more training data, improve the robustness of the model, prevent the model from overfitting, and improve the generalization ability of the model.

[0100] The loss function of the model can be a Dice-BCE loss function, which combines the Dice loss function and the BEC binary cross entropy loss function, and takes into account both the similarity and accuracy of the prediction results.

[0101] The Dice loss function is used to evaluate the similarity between the predicted segmentation results and the true segmentation results, which is expressed as:

[0102]

[0103] Among them, X and Y are sets, |X∩Y| is the intersection of sets X and Y, and |X| and |Y| are the number of elements in sets X and Y, respectively. For the segmentation task, |X| and |Y| represent the true label and predicted label of the segmentation, respectively.

[0104] The BCE loss function is expressed as:

[0105]

[0106] Where N is the total number of pixels in the image, i is the pixel number, yi is the true category of pixel xi, and p(yi) is the probability of predicting that xi belongs to category 1.

[0107] The Dice-BCE loss function is expressed as:

[0108] DICE-BEC=DiceLoss+λBEC;

[0109] Among them, λ is a preset weight parameter used to control the balance between the two loss functions.

[0110] The model is trained based on the above training set, test set and Dice-BCE loss function to obtain a trained medical image segmentation model.

[0111] Step S20, downsampling and tensor reconstruction processing are performed on the medical image to be segmented by the encoder to obtain a multi-scale feature image and an output attention tensor;

[0112] Reference Figure 3 , Figure 3 It is a schematic diagram of the structure of the deep learning model in the embodiment of the medical image segmentation method of the present invention. The model is U-shaped as a whole, with the encoder stage on the left and the decoder stage on the right. The encoder stage is divided into four layers of networks. In the first layer, the size of the input medical image to be segmented is 3×H×W, where 3 is the number of channels, H is the image height, and W is the image width. After patch embedding, the size becomes C×(H / 2)×(W / 2), where C is the number of channels, and then it is input into the cascade BF block of the first layer for tensor reconstruction to obtain the output tensor of the first layer; in the second layer, the output tensor of the first layer is downsampled after patch aggregation, and the image size becomes 2C×(H / 4)×(W / 4), where 2C is the number of channels, and then it is input into the cascade BF block of the second layer for tensor reconstruction, and so on, until a final output attention tensor is obtained through the fourth layer network.

[0113] Each BF block is used to model the positional relationship between sequences in different directions of the image. The deeper the layer, the more advanced and abstract semantic information will be extracted, while the lower the layer, the more concrete low-level information will be learned. After that, the encoder inputs the final output attention tensor (high-level feature map) into the decoder, and inputs the multi-scale feature image obtained by downsampling each layer of the network into the attention multi-scale feature fusion module for multi-scale feature fusion.

[0114] Step S30, performing feature fusion processing on the multi-scale feature image through the attention multi-scale feature fusion module to obtain a fused feature image;

[0115] In this step, the attention multi-scale feature fusion module is used to perform feature fusion processing on the feature images of different sizes input by each layer of the encoder. Specifically, the spatial attention and channel attention of the input multi-scale feature image can be calculated first, and then the multi-scale feature image, the spatial attention and the channel attention can be fused to highlight the important features of the spatial dimension and the channel dimension and to downplay the useless features. By mixing features of different scales, the medical image segmentation model can have a wide receptive field, so that semantic information at a distance can also be fully paid attention to, thereby improving the final feature expression ability of the model.

[0116] Step S40, inputting the output attention tensor and the fused feature image into the decoder for upsampling and tensor reconstruction processing to obtain a segmented image;

[0117] In the decoder stage, the high-level feature map from the end of the encoder stage is received and combined with the features (fused feature image) transmitted by the multi-scale feature fusion module to create a segmented image.

[0118] Continue to refer to Figure 3 , the decoder stage is also divided into four layers of networks. The output attention tensor generated in the encoder stage is first input into the cascaded BF block of the fourth layer for tensor reconstruction, and then spliced ​​with the fourth layer fusion feature image output by the attention multi-scale feature fusion module in the channel dimension (as shown in the symbol As shown in the figure), the image size is enlarged to 4C×(H / 8)×(W / 8) by upsampling in a patch expansion manner, and then input into the cascaded BF block of the third layer for tensor reconstruction, and then upsampling is performed in a patch expansion manner, and so on, until the segmented image is obtained through four-layer decoding, and the size of the segmented image is C×(H / 2)×(W / 2).

[0119] Step S50: performing tensor reconstruction processing on the segmented image through the tail block to output a segmented target medical image.

[0120] After the four-layer decoder stage, the segmented image output by the decoder is passed to the full-resolution tail block, and the segmented image is tensor reconstructed through the cascaded BF block in the tail block to establish a more refined segmented image, and finally the segmented target medical image is output through the linear layer, expressed as num_classes×H×W, where num_classes is the number of channels. The target medical image can accurately express the morphological structure of the input medical image, where white represents the physiological structure to be segmented and black represents the background.

[0121] In this embodiment, the image is tensor reconstructed through the encoder, decoder and tail block in the deep learning model, and feature fusion processing is performed through the attention multi-scale feature fusion module. Compared with the segmentation method in the prior art that lacks multi-scale feature fusion, the accuracy of medical image segmentation is improved.

[0122] Furthermore, based on the first embodiment of the medical image segmentation method of the present invention, a second embodiment of the medical image segmentation method of the present invention is proposed.

[0123] In this embodiment, the tensor reconstruction process includes: obtaining an input tensor; performing a pixel scan on the input tensor in the horizontal direction and the vertical direction respectively to obtain a horizontal sequence and a vertical sequence; inputting the horizontal sequence and the vertical sequence into a preset state space model to obtain a horizontal attention sequence and a vertical attention sequence; reshaping and splicing the horizontal attention sequence and the vertical attention sequence, and inputting them into a preset hybrid feedforward network; modeling the positional relationship between the horizontal attention sequence and the vertical attention sequence through the hybrid feedforward network, and outputting a first attention tensor.

[0124] In this embodiment, the BF block may include a bidirectional fusion selective scanning module (Bidirectional Fusion Selective Scan, BFSS), and the tensor reconstruction process is implemented by the BFSS module. Figure 4 , Figure 4 The schematic diagram of the BFSS module in the embodiment of the medical image segmentation method of the present invention is as follows: the input tensor is first subjected to horizontal order scanning and vertical sequence scanning at the pixel level to obtain a horizontal sequence and a vertical sequence, and then the horizontal sequence and the vertical sequence are input into a preset state space model (State Space Model, SSM) to obtain a horizontal attention sequence and a vertical attention sequence, and then the horizontal attention sequence and the vertical attention sequence are reshaped and spliced ​​to reconstruct into a two-dimensional tensor, and then the positional relationship between the horizontal attention sequence and the vertical attention sequence of the two-dimensional tensor is modeled by a hybrid feedforward network (Fusion Feed Forward Network, FFFN), and the processing in the hybrid feedforward network includes layer normalization, linear layer processing, channel dimension splicing and linear layer processing in sequence, and finally, the attention tensor output by the hybrid feedforward network is the first attention tensor. Among them, the SSM model can adopt the existing Mamba model, which is a new type of selective structured state space model. By introducing a selective mechanism, it can effectively process long sequence data while maintaining linear time complexity.

[0125] Specifically, the above BFSS module can be expressed by the formula:

[0126] Seq Horisontal = Flatten(X);

[0127] Seq Vertical =Flatten(Transpose(X));

[0128] SeqAtten Horisontal =SSM(Seq Horisontal );

[0129] SeqAtten Vertical =SSM(Seq Vertical );

[0130] Y=Linear(LayerNorm(linear(Concat([SeqAtten Horisontal ,SeqAtten Vertical ]))));

[0131] Among them, X represents the input two-dimensional tensor, Y represents the output two-dimensional tensor, SeqHorisontal represents the horizontal sequence, SeqVertical represents the vertical sequence, SeqAttenHorisontal represents the horizontal attention sequence, SeqAttenVertical represents the vertical attention sequence, Flatten(·) represents the expansion into a one-dimensional sequence, Transpose(·) represents the matrix transpose, SSM(·) represents the state space model, Concat(·) represents the reorder and concatenation, Linear(·) represents the linear layer, and LayerNorm(·) represents the layer normalization.

[0132] It should be noted that, compared with the visual state space module SS2D (SelectiveScan 2D) of the existing VM-UNet model, this embodiment only needs to perform one horizontal pixel scan and one vertical pixel scan on the input tensor. Figure 5 , Figure 5 FIG. 4 is a schematic diagram comparing the scanning modes of the BFSS module in the medical image segmentation method embodiment of the present invention and the existing visual state space module SS2D. Figure 5 As shown in (a) and (b), the existing SS2D module needs to scan four different scanning sequences, namely from upper left to lower right, from lower right to upper left, from upper right to lower left, and from lower left to upper right, while the SS2D module proposed in this embodiment only needs to scan two sequences, namely from upper left to lower right and from upper right to lower left.

[0133] The BFSS module proposed in this embodiment optimizes redundant sequences compared to SS2D in the prior art, and only requires two sequences to fully model the horizontal and vertical spatial information of the image, thereby reducing the parameter size of the model and only requiring about one-half of the parameter storage space and computing resource consumption. Moreover, the BFSS module proposed in this embodiment uses a hybrid feedforward network FFFN to model the positional relationship between different sequences, replacing the SS2D modeling method of directly adding sequences by matrix addition. The use of FFFN can more accurately model the positional relationship between horizontal and vertical sequences, and enhance the model's perception of medical images.

[0134] Furthermore, based on the second embodiment of the medical image segmentation method of the present invention, a third embodiment of the medical image segmentation method of the present invention is proposed.

[0135] In this embodiment, the tensor reconstruction process may further include: performing frequency domain feature extraction on the input tensor to obtain a frequency domain tensor; and outputting a second attention tensor based on the first attention tensor and the frequency domain tensor.

[0136] In this embodiment, the BF block may include a frequency gate in addition to the BFSS module, which is used to extract frequency domain features from the input tensor to obtain a frequency domain tensor. After that, the frequency domain tensor and the first attention tensor output by the BFSS module are concatenated in the channel dimension to finally output the second attention tensor.

[0137] Compared with SS2D in the prior art, this embodiment adds the extraction of frequency domain features, thereby increasing feature richness, which is beneficial to further improve the accuracy of medical image segmentation.

[0138] Furthermore, the frequency domain feature extraction of the input tensor to obtain the frequency domain tensor may include: performing a real Fourier transform on the input tensor to obtain a frequency domain feature; matrix multiplying the frequency domain feature and a preset complex weight parameter, and then adding the matrix with a preset complex bias parameter to obtain a result feature; performing a real Fourier inverse transform on the result feature to obtain a frequency domain tensor.

[0139] Reference Figure 6 , Figure 6 The figure is a schematic diagram of the frequency gating module in the embodiment of the medical image segmentation method of the present invention. First, the input tensor is projected into the frequency domain via a real Fourier transform, where Width represents the height of the frequency domain feature, and Width / 2+1 represents the width of the frequency domain feature. Then, the frequency domain feature and the preset complex weight parameter are matrix multiplied, and then added to the preset complex bias parameter to obtain the result feature, and finally the frequency domain tensor is outputted via a real Fourier inverse transform. The size of the complex weight parameter and the complex bias parameter is adaptive to the size of the input tensor.

[0140] Specifically, the above frequency domain tensor acquisition process can be expressed as follows:

[0141] Tensor Freq =RFFT(X);

[0142] Tensor Atten =Tensor Freq ×Weight Complex +Bias Complex ;

[0143] Y=IRFFT(Tensor Atten );

[0144] Where X represents the input two-dimensional tensor, and Y represents the output two-dimensional tensor. RFFT(·) represents the real Fourier transform, IRFFT(·) represents the real inverse Fourier transform, and WeightComplex and BiasComplex represent the preset complex weight parameters and complex bias parameters, respectively.

[0145] A new type of frequency gating proposed in this embodiment can extract finer-grained frequency domain features by introducing a complex bias term, thereby facilitating further improving the accuracy of medical image segmentation.

[0146] Furthermore, based on the first embodiment of the medical image segmentation method of the present invention, a fourth embodiment of the medical image segmentation method of the present invention is proposed.

[0147] In this embodiment, the attention multi-scale feature fusion module includes an attention bridge and a multi-scale feature fusion module. The above step S30 may further include: inferring the channel attention and spatial attention of the multi-scale feature image through the attention bridge; obtaining a multi-scale output feature tensor according to the channel attention, the spatial attention and the multi-scale feature image; inputting the multi-scale output feature tensor into the multi-scale feature fusion module to obtain a fused feature image.

[0148] Reference Figure 7 , Figure 7 The schematic diagram of the structure of the attention multi-scale feature fusion module in the embodiment of the medical image segmentation method of the present invention is shown in Figure (a) as an attention bridge and Figure (b) as a multi-scale feature fusion module. In the attention bridge stage, for the input feature maps of different scales, the attention bridge infers the attention map along two independent dimensions (channel and space), and then multiplies the attention map by the input feature image for adaptive feature refinement to obtain a multi-scale output feature tensor. In the multi-scale feature fusion module, the multi-scale output feature tensor input by the attention bridge is subjected to feature fusion processing to finally obtain a multi-scale fused feature image.

[0149] Reference Figure 8 , Figure 8 for Figure 7Schematic diagram of the structure of the channel attention module in [1]. For BFSS channel attention, we first aggregate the spatial information of the feature map by using average pooling and maximum pooling operations to generate two different spatial context descriptors, which represent the average pooled features and the maximum pooled features, respectively. Then, both descriptors are forwarded to the shared network to produce the channel attention map. The shared network consists of a multilayer perceptron (MLP) with one hidden layer and shared weights. After applying the shared network to each descriptor, we merge the output feature vectors using element-wise summation and finally get the channel attention through an activation function.

[0150] Reference Fig. 9 , Fig. 9 for Figure 7 Schematic diagram of the structure of the spatial attention module in BFSS. For BFSS spatial attention, we first aggregate the channel information of the feature map by using two pooling operations to generate two 2D feature maps: the channel dimension mean feature map and the channel dimension maximum feature map, which represent the average pooling features and the maximum pooling features in the channel, respectively. Then we concatenate them and perform convolution through a standard convolutional layer to finally generate a 2D spatial attention map.

[0151] Furthermore, obtaining a multi-scale output feature tensor based on the channel attention, the spatial attention and the multi-scale feature image may include: matrix multiplying the channel attention and the multi-scale feature image to obtain an intermediate tensor; matrix multiplying the intermediate tensor and the spatial attention, and then matrix adding the intermediate tensor and the multi-scale feature image to obtain a multi-scale output feature tensor.

[0152] In this embodiment, the method for calculating the multi-scale output feature tensor can be: first matrix multiply the channel attention and the multi-scale feature image to obtain an intermediate tensor, then matrix multiply the intermediate tensor and the spatial attention, and then matrix add them with the multi-scale feature image to finally obtain a multi-scale output feature tensor.

[0153] Furthermore, inputting the multi-scale output feature tensor into the multi-scale feature fusion module to obtain a fused feature image may include: in the multi-scale feature fusion module, generating a unit tensor according to a preset anchor tensor; performing size conversion on the multi-scale output feature tensor to obtain a feature tensor of the same size as the anchor tensor; performing deep convolution processing on the feature tensor of the same size as the anchor tensor; and performing matrix multiplication of the feature tensor after the deep convolution processing and the unit tensor to obtain a fused feature image.

[0154] Continue to refer to Figure 7In the multi-scale feature fusion stage, the module receives features of four different scales and generates a unit tensor of the same size according to the preset anchor tensor. For these four feature tensors of different scales, those with a size larger than the anchor tensor will be average pooled to adapt to the size of the anchor tensor, and those with a size smaller than the anchor tensor will be bilinearly interpolated to adapt to the size of the anchor tensor. After that, the four tensors of the same size will be deep convolved to mix spatial information. In the final stage, the generated unit tensor will perform matrix multiplication on the four feature tensors in turn to fuse features of different scales, and the number of channels will be changed through the linear layer to output to adapt to the tensor of the decoder stage.

[0155] This embodiment integrates multi-scale feature images, spatial attention and channel attention to highlight the important features of spatial and channel dimensions and downplay useless features. It also has a wide receptive field that can give full attention to distant semantic information, further enhancing the feature expression ability of the model and improving the accuracy of medical image segmentation.

[0156] The embodiment of the present invention also provides a medical image segmentation device.

[0157] Reference Fig.10 , Fig.10 Schematic diagram of a module of an embodiment of a medical image segmentation device of the present invention. In this embodiment, the medical image segmentation device comprises:

[0158] An input module 10 is used to obtain a medical image to be segmented, and input the medical image to be segmented into a deep learning model, wherein the deep learning model includes an encoder, an attention multi-scale feature fusion module, a decoder, and a tail block;

[0159] An encoding module 20, configured to perform downsampling and tensor reconstruction processing on the medical image to be segmented through the encoder to obtain a multi-scale feature image and an output attention tensor;

[0160] A fusion processing module 30, configured to perform feature fusion processing on the multi-scale feature image through the attention multi-scale feature fusion module to obtain a fused feature image;

[0161] A decoding module 40, configured to input the output attention tensor and the fused feature image into the decoder for upsampling and tensor reconstruction processing to obtain a segmented image;

[0162] The output module 50 is used to perform tensor reconstruction processing on the segmented image through the tail block to output a segmented target medical image.

[0163] Optionally, the encoding module 20, the decoding module 40 and the output module 50 are further used for:

[0164] Get the input tensor;

[0165] Scanning the input tensor once in the horizontal direction and once in the vertical direction to obtain a horizontal sequence and a vertical sequence;

[0166] Inputting the horizontal sequence and the vertical sequence into a preset state space model to obtain a horizontal attention sequence and a vertical attention sequence;

[0167] After reshaping and concatenating the horizontal attention sequence and the vertical attention sequence, the resultant sequences are input into a preset hybrid feedforward network;

[0168] The positional relationship between the horizontal attention sequence and the vertical attention sequence is modeled through the hybrid feedforward network, and a first attention tensor is output.

[0169] Optionally, the encoding module 20, the decoding module 40 and the output module 50 are further used for:

[0170] Performing frequency domain feature extraction on the input tensor to obtain a frequency domain tensor;

[0171] Output a second attention tensor according to the first attention tensor and the frequency domain tensor.

[0172] Optionally, the encoding module 20, the decoding module 40 and the output module 50 are further used for:

[0173] Performing a real Fourier transform on the input tensor to obtain frequency domain features;

[0174] After matrix multiplication of the frequency domain feature and the preset complex weight parameter, the result feature is obtained by adding the preset complex bias parameter;

[0175] Perform a real inverse Fourier transform on the result features to obtain a frequency domain tensor.

[0176] Optionally, the attention multi-scale feature fusion module includes an attention bridge and a multi-scale feature fusion module, and the fusion processing module 30 is further used for:

[0177] Inferring the channel attention and spatial attention of the multi-scale feature image through the attention bridge;

[0178] Obtaining a multi-scale output feature tensor according to the channel attention, the spatial attention and the multi-scale feature image;

[0179] The multi-scale output feature tensor is input into the multi-scale feature fusion module to obtain a fused feature image.

[0180] Optionally, the fusion processing module 30 is further used for:

[0181] Performing matrix multiplication on the channel attention and the multi-scale feature image to obtain an intermediate tensor;

[0182] After matrix multiplication of the intermediate tensor and the spatial attention, matrix addition is performed with the multi-scale feature image to obtain a multi-scale output feature tensor.

[0183] Optionally, the fusion processing module 30 is further used for:

[0184] In the multi-scale feature fusion module, a unit tensor is generated according to a preset anchor tensor;

[0185] Performing a size conversion on the multi-scale output feature tensor to obtain a feature tensor of the same size as the anchor tensor;

[0186] Performing deep convolution processing on the feature tensor having the same size as the anchor tensor;

[0187] The feature tensor after deep convolution processing is matrix multiplied with the unit tensor to obtain a fused feature image.

[0188] The functional implementation and beneficial effects of each module in the above-mentioned medical image segmentation device correspond to the steps in the above-mentioned medical image segmentation method embodiment, and will not be repeated here.

[0189] The embodiment of the present invention also provides a medical image segmentation device.

[0190] Reference Fig.11 , Fig.11 1 is a schematic diagram of the structure of an embodiment of a medical image segmentation device of the present invention. The medical image segmentation device 100 includes a memory 110 and a processor 120. The memory 110 stores a computer program. When the computer program is executed by the processor 120, the medical image segmentation method of any of the above embodiments is implemented. The medical image segmentation device 100 can be a mobile phone, a camera, a laptop computer, a tablet computer, a smart watch, a monitoring device, a server, a medical service terminal, etc.

[0191] An embodiment of the present invention also provides a computer-readable storage medium.

[0192] Reference Fig.12 , Fig.12The figure is a schematic diagram of the connection state of a computer-readable storage medium and a processor according to an embodiment of the present invention. The computer-readable storage medium 200 may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, and a computer program 210 is stored on the computer-readable storage medium 200. When the computer program 210 is executed by the processor 120, the medical image segmentation method according to any of the above embodiments is implemented.

[0193] Those skilled in the art can understand that the above-mentioned integrated modules or units, if implemented in the form of software functional units and sold or used as independent products, can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0194] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A medical image segmentation method, characterized in that: The medical image segmentation method comprises the following steps: Acquire a medical image to be segmented, and input the medical image to be segmented into a deep learning model, wherein the deep learning model includes an encoder, an attention multi-scale feature fusion module, a decoder, and a tail block; Downsampling and tensor reconstruction processing are performed on the medical image to be segmented by the encoder to obtain a multi-scale feature image and an output attention tensor; Performing feature fusion processing on the multi-scale feature image through the attention multi-scale feature fusion module to obtain a fused feature image; Inputting the output attention tensor and the fused feature image into the decoder for upsampling and tensor reconstruction processing to obtain a segmented image; The segmented image is subjected to tensor reconstruction processing by the tail block to output a segmented target medical image.

2. The medical image segmentation method according to claim 1, characterized in that: The tensor reconstruction process includes: Get the input tensor; Scanning the input tensor once in the horizontal direction and once in the vertical direction to obtain a horizontal sequence and a vertical sequence; Inputting the horizontal sequence and the vertical sequence into a preset state space model to obtain a horizontal attention sequence and a vertical attention sequence; After reshaping and concatenating the horizontal attention sequence and the vertical attention sequence, the resultant sequences are input into a preset hybrid feedforward network; The positional relationship between the horizontal attention sequence and the vertical attention sequence is modeled through the hybrid feedforward network, and a first attention tensor is output.

3. The medical image segmentation method according to claim 2, characterized in that: The tensor reconstruction process further includes: Performing frequency domain feature extraction on the input tensor to obtain a frequency domain tensor; Output a second attention tensor according to the first attention tensor and the frequency domain tensor.

4. The medical image segmentation method according to claim 3, characterized in that: The extracting frequency domain features from the input tensor to obtain a frequency domain tensor comprises: Performing a real Fourier transform on the input tensor to obtain frequency domain features; After matrix multiplication of the frequency domain feature and the preset complex weight parameter, the result feature is obtained by adding the preset complex bias parameter; Perform a real inverse Fourier transform on the result features to obtain a frequency domain tensor.

5. The medical image segmentation method according to claim 1, characterized in that: The attention multi-scale feature fusion module includes an attention bridge and a multi-scale feature fusion module. The multi-scale feature image is subjected to feature fusion processing by the attention multi-scale feature fusion module to obtain a fused feature image, which includes: Inferring the channel attention and spatial attention of the multi-scale feature image through the attention bridge; Obtaining a multi-scale output feature tensor according to the channel attention, the spatial attention and the multi-scale feature image; The multi-scale output feature tensor is input into the multi-scale feature fusion module to obtain a fused feature image.

6. The medical image segmentation method according to claim 5, characterized in that: The obtaining of a multi-scale output feature tensor according to the channel attention, the spatial attention and the multi-scale feature image comprises: Performing matrix multiplication on the channel attention and the multi-scale feature image to obtain an intermediate tensor; After matrix multiplication of the intermediate tensor and the spatial attention, matrix addition is performed with the multi-scale feature image to obtain a multi-scale output feature tensor.

7. The medical image segmentation method according to claim 5, characterized in that: Inputting the multi-scale output feature tensor into the multi-scale feature fusion module to obtain a fused feature image comprises: In the multi-scale feature fusion module, a unit tensor is generated according to a preset anchor tensor; Performing a size conversion on the multi-scale output feature tensor to obtain a feature tensor of the same size as the anchor tensor; Performing deep convolution processing on the feature tensor having the same size as the anchor tensor; The feature tensor after deep convolution processing is matrix multiplied with the unit tensor to obtain a fused feature image.

8. A medical image segmentation device, characterized in that: The medical image segmentation device comprises: An input module, used for acquiring a medical image to be segmented, and inputting the medical image to be segmented into a deep learning model, wherein the deep learning model includes an encoder, an attention multi-scale feature fusion module, a decoder, and a tail block; An encoding module, used for performing downsampling and tensor reconstruction processing on the medical image to be segmented through the encoder to obtain a multi-scale feature image and an output attention tensor; A fusion processing module, used for performing feature fusion processing on the multi-scale feature image through the attention multi-scale feature fusion module to obtain a fused feature image; A decoding module, used for inputting the output attention tensor and the fused feature image into the decoder for upsampling and tensor reconstruction processing to obtain a segmented image; The output module is used to perform tensor reconstruction processing on the segmented image through the tail block to output a segmented target medical image.

9. A medical image segmentation device, characterized in that: The medical image segmentation device comprises: a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the medical image segmentation method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the medical image segmentation method according to any one of claims 1 to 7 is implemented.

Images