A medical image segmentation method based on multi-branch feature fusion refinement

Abstract

The application belongs to the field of medical image processing, and particularly relates to a medical image segmentation method, system and device based on multi-branch feature fusion refinement. The medical image segmentation method comprises the following steps: S1: performing feature extraction on a medical image through a PVT and a ResNext101 model; S2: fusing two features of each corresponding layer to obtain fused features; S3: fully mining global context information in each layer of fused features, and outputting refined features corresponding to each layer of fused features; S4: constructing long-distance correlation between each refined feature, and performing cross-layer refinement operation on four refined features at different levels; S5: taking steps S1-S4 as operations of an encoding end; in an upsampling process of a decoding end, the cross-layer refined features and the refined features of the deepest layer are combined with the decoding layer in a manner similar to a skip connection, and a segmentation result of the medical image is output. The application solves the problems of insufficient accuracy, reliability and robustness of existing medical image segmentation methods.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of medical image processing, and particularly relates to a medical image segmentation method, system and device based on multi-branch feature fusion refinement. BACKGROUND

[0002] With the continuous development of science and technology, medical imaging equipment is increasingly applied to the diagnosis of various diseases. Medical imaging can help doctors observe various physiological pathological states inside the patient's body; thus, it plays an irreplaceable and prominent role in the early diagnosis of many diseases. It may be difficult for doctors to analyze medical images alone, and with the application of image recognition technology in the field of medical images, medical image segmentation technology has also been continuously applied to the auxiliary diagnosis of medical applications. The purpose of medical image segmentation is to extract relevant features and segment parts of medical images that have certain special meanings to assist medical personnel in making more accurate judgments.

[0003] Although medical image segmentation technology is an important step in the field of medical image processing and analysis, medical image segmentation itself is a difficult task. Due to the differences in case characteristics of diseases in different types of medical images, the feature information contained in medical images has high complexity. Specifically, the sizes of multiple segmentation targets are different, the shapes are different, and the spatial distribution has no regularity. At the same time, since medical image datasets need to be manually labeled by professionals, it is difficult to obtain a large amount of data, which greatly limits the application of deep learning methods. How to extract effective features from limited samples and obtain accurate segmentation results is still a challenging problem.

[0004] Medical image segmentation is currently mainly divided into segmentation methods based on CNN network and segmentation methods based on Transformer network. CNN has very good performance, which is mainly due to the convolution operation. In order to obtain better image representation, it collects local features in a hierarchical manner, but this design limits the ability of CNN to capture global information, which is crucial for medical image segmentation tasks. Transformer can capture long-range feature dependencies to obtain global feature representation, but it ignores local details and easily confuses background and foreground at the boundary.

[0005] Therefore, most of the above two types of medical image segmentation schemes cannot take into account the rich local features and wide range of global features in medical images, and thus cannot maintain high segmentation accuracy in most complex medical image segmentation tasks. The traditional medical image segmentation scheme has deficiencies in accuracy, reliability and robustness. SUMMARY

[0006] In order to solve the problem that the existing medical image segmentation method cannot consider the local features and global features of the sample, and thus has defects in accuracy, reliability and robustness, the present application provides a medical image segmentation method, system and device based on multi-branch feature fusion refinement.

[0007] The present application adopts the following technical solutions:

[0008] A medical image segmentation method based on multi-branch feature fusion refinement is used to identify the target in the medical image by combining multi-level feature information, and then segment the image of the more accurate target region.

[0009] The overall idea of the image segmentation method provided by the present application is as follows: first, the PVT pre-training model and the ResNext101 pre-training model are used to extract the features of the input medical image, the outputs of each layer of the two main stems are taken as inputs, and the fusion features are obtained through the newly designed double-branch feature fusion module. And all the fusion features are taken as input, and the cross-layer feature refinement module is designed to analyze the features of different levels, and extract the global information therefrom; the cross-layer refined features are obtained. Finally, the cross-layer refined features and the refined features of the deepest layer are simultaneously input into the decoder for up-sampling, and the feature map is gradually restored to the original resolution, and finally the segmentation result image is output.

[0010] Specifically, the detailed processing process of the medical image segmentation method based on multi-branch feature fusion refinement provided by the present application includes the following steps:

[0011] S1: synchronously extracting features of the input medical image through the pre-trained PVT model and ResNext101 model, respectively obtaining two original features, each including four feature maps of different levels; and performing size unification operation on the two original features of different sources.

[0012] In the size unification operation of step S1, the size of the original feature map output by the ResNext101 model is adjusted according to the shape of the feature map output by the PVT model, so that the sizes of the feature maps output by the two models are matched.

[0013] S2: fusing the features of each corresponding layer in the two size-unified feature maps to obtain corresponding fusion features F. The fusion process is as follows:

[0014] S21: adding the two feature maps F p 、F r extracted by the PVT model and the ResNext101 model.

[0015] S22: extracting a multi-scale channel attention matrix M of each layer feature map after addition through a multi-scale channel attention module.

[0016] S23: Multiply the multi-scale channel attention matrix M with its respective input features, and then add the two feature maps together again.

[0017] S24: The feature maps obtained from the previous steps are subjected to a 3x3 convolution and batch normalization operation, and finally activated by ReLU to obtain the fused feature F. pr .

[0018] In step S2 of this invention, the operation process of calculating the multi-scale channel attention matrix M(X) of the feature map X using the multi-scale channel attention module is described in detail below:

[0019]

[0020] In the above formula, g(X) represents global average pooling of the feature map X, and δ represents the ReLU activation function. PWConv represents batch normalization operation, and PWConv represents point convolution operation. The addition operation between feature maps is represented by X, which represents the result of adding the feature maps of the two branches. G(X) and L(X) represent two different operations on feature X. The G(X) branch uses global average pooling and pays more attention to large objects in the global distribution. The L(X) branch maintains the original feature size and pays more attention to local details, making the segmentation boundary clearer and more accurate while ensuring that small objects are not ignored.

[0021] Furthermore, based on the two original feature maps F extracted from the PVT model and the ResNext101 model... p F r Calculate the fusion feature F pr The calculation formula is as follows:

[0022]

[0023] In the above formula, This represents element-wise multiplication; This represents the original feature map F of the corresponding layer. p F r Add them together after unifying the dimensions.

[0024] S3: Fully explore the fusion features of each layer F pr The global context information is used to output the fused features F of each layer. pr Corresponding refining feature F e Refined Feature F e The generation process is as follows:

[0025] S31: Fuse feature F e After processing through 3×3 convolution and average pooling layers respectively, the feature map F is obtained.c and feature map F a .

[0026] S32: Extracting feature map F c and feature map F a Multi-scale channel attention matrix M c and M a .

[0027] S33: Fusing multi-scale channel attention matrix M c and M a with F c and F a by element-wise multiplication, to obtain feature map F cm and F am .

[0028] S34: Restoring the feature map F am to the same size as F cm by upsampling, and directly fusing the feature maps of the two branches by addition to obtain F cam .

[0029] S35: Fusing refined feature F pr and F cam by a residual structure to obtain refined feature F e .

[0030] In the present application, the operation process of generating the corresponding refined feature F pr based on each layer fusion feature F e is represented in detail as follows:

[0031]

[0032] In the above formula, A(·) represents the average pooling operation; U(·) represents the upsampling operation; Conv represents the convolution operation.

[0033] S4: Constructing the long-distance correlation between each refined feature obtained in the previous step, performing cross-layer refining operation on the four different levels of refined features; and outputting three cross-layer refined features. The process is as follows:

[0034] S41: Assuming that the four different levels of refined features obtained are F′1, F′2, F′3, and F′4, flattening them in the spatial dimension, and then deforming them to the same number of channels.

[0035] S42: Splicing the four refined features in the flattened spatial dimension.

[0036] S43: After the spliced features are normalized through the LN layer, the long-distance correlation between the features is constructed through the self-attention layer and the feedforward neural network MixFFN.

[0037] S44: The output of the feedforward neural network in the previous step is divided into three cross-layer refining features, and the cross-layer refining features are respectively transformed into the shape of the corresponding decoding layer of the decoding end and output to the decoding end.

[0038] In the present application, the processing process of the cross-layer feature refining operation is as follows:

[0039]

[0040] In the above formula, F i represents the refining features obtained according to the fusion features of different layers; i represents the layer mark of each fusion feature, i = 1, 2, 3, 4; tokenF i represents the operation result of transforming the feature map to the same shape; B represents the data amount of each batch; C represents the channel number; Reshape represents the transformation operation; Concatenate represents the feature splicing operation; merge_token represents the operation result of feature splicing; Efficient_Atten represents the effective self-attention module; Atten_token represents the operation result of the effective self-attention module; LN represents the layer normalization operation; res_token represents the operation result of the layer normalization; MixFFN represents the feedforward neural network; FFN i represents the result processed by the feedforward neural network; Split represents the dividing and restoring original shape operation; split_token represents the operation result of dividing and restoring the original shape of res_token; output represents the overall output of the cross-layer feature refining operation.

[0041] S5: Steps S1-S4 are used as the operation of the encoding end; in the upsampling process of the decoding end, the cross-layer refining features and the refining features of the deepest layer are combined with the decoding layers through the way of skip connection, the information lost during the pooling operation is corrected, and finally the segmentation result of the medical image is obtained.

[0042] Specifically, the decoding process of step S5 adopts the following decoding strategy:

[0043] (1) The refining features of the deepest layer are used as the features of the first layer of the decoder to participate in the decoder analysis process;

[0044] (2) The features of the deepest layer of the refining features are transformed to the same shape as the second, third and fourth decoding layers of the decoder through convolution and pooling operation once;

[0045] (3) The three cross-layer refined features output by the cross-layer feature refinement operation are respectively transmitted to the first, second and third decoding layers of the decoder;

[0046] (4) In each decoding layer, the corresponding refined feature is combined with the corresponding decoding layer and cross-layer refined feature to eliminate the encoding and decoding ambiguities and effectively alleviate the loss of detail information.

[0047] (5) The predicted image segmentation result is output by the last decoding layer.

[0048] In particular, the medical image segmentation method based on multi-branch feature fusion refinement designed by the application is an image segmentation processing scheme using a network model with an "encoder-decoder" architecture. In the training phase of the built image segmentation network model, the overall network parameters are optimized using side loss and end-to-end loss.

[0049] During the training process, the label is deformed to the size corresponding to each layer of the decoding layer, and the cross-entropy loss is calculated; the cross-entropy loss is added to the loss function between the side supervision and the segmentation result and the label, so as to optimize the overall network parameters through back propagation.

[0050] In the training phase, the network model is trained using the pre-prepared sample images of the training set until the loss function converges. The segmentation result is predicted on the test set during each training round, and the best model is saved according to the indicators.

[0051] The application also includes a medical image segmentation system based on multi-branch feature fusion refinement, which uses the medical image segmentation method based on multi-branch feature fusion refinement as described above to analyze and process the input medical image, and then outputs the image segmentation result of the recognized and predicted target region. The medical image segmentation system based on multi-branch feature fusion refinement includes a first feature extraction module, a second feature extraction module, a size adjustment module, a feature fusion module, a feature refinement module, a cross-layer feature refinement module, and a decoder.

[0052] The first feature extraction module is the backbone network in one of the branches. The first feature extraction module uses a PVT model as the feature extraction network and is used to obtain the original feature maps of the input medical image at four different levels.

[0053] The second feature extraction module is the backbone network in another branch. The second feature extraction module uses a ResNext101 model as the feature extraction network and is used to obtain the original feature maps of the input medical image at four different levels.

[0054] The size adjustment module is used to adjust the size of the original feature map of the corresponding layer output by the ResNext101 model based on the shape of the feature map output by the PVT model, so that the size of the feature maps output by the two models matches.

[0055] The feature fusion module adds the feature maps of corresponding layers extracted from the two backbone networks. Then, a multi-scale channel attention module calculates the importance of the features to obtain a multi-scale channel attention matrix. This matrix is ​​multiplied by the respective input features to highlight important features and suppress task-irrelevant features. Finally, the two feature maps from each layer are added again, followed by a 3x3 convolution and batch normalization operation, and finally activated by ReLU to obtain the fused feature F. pr .

[0056] The feature refinement module first obtains the fused features from each layer of the feature fusion module. Then, the fused features are processed by 3×3 convolution and average pooling layers respectively to obtain the feature map F. c and feature map F a Next, the feature representation based on multi-scale attention is learned through a multi-scale channel attention module, and the obtained multi-scale channel attention matrix M is then processed. c and M a Through element-wise multiplication and F c and F a The feature map F is obtained by fusion. cm and F am Then F am The feature map is recovered to the value of F by upsampling. cm For the same size, F is obtained by directly fusing the feature maps of the two branches through addition. cam Finally, the fused features F are obtained through the residual structure. pr With F cam Fusion yields refined characteristic F e .

[0057] The cross-layer feature refinement module is used to obtain the four-layer refined features F′1, F′2, F′3, and F′ from the feature refinement module. 41,2,3 ,,4, and use F′1~F′4 as input to the cross-layer feature refinement module. The cross-layer feature refinement module first flattens each feature map in the spatial dimension, then transforms it to the same number of channels, and concatenates four refined features in the flattened spatial dimension. Next, after normalization by an LN layer, it passes through a self-attention layer and a feedforward neural network MixFFN to construct long-distance correlations, obtaining the cross-layer refined features. Finally, the cross-layer refined features are transformed into the shapes of the corresponding decoding layers in the decoder and output to the decoder;

[0058] The decoder is used to obtain the cross-layer refined features and the refined features of the deepest layer, and then combines each feature map with the decoding layer through a jump connection to correct the information lost during the pooling operation, and finally obtains the segmentation result of the medical image. The processing process of the encoder includes: (1) the refined features of the deepest layer are used as the features of the first layer of the decoder to participate in the decoding process. (2) The deepest layer of the refined feature is deformed to the same shape as the second, third and fourth decoding layers of the decoder through convolution and pooling operation. (3) The three cross-layer refined features output by the cross-layer feature refining operation are respectively transmitted to the first, second and third decoding layers of the decoder. (4) In each decoding layer, the corresponding refined features are combined with the corresponding decoding layer and cross-layer refined features. (5) The predicted image segmentation result is output through the last decoding layer.

[0059] The application also includes a medical image segmentation device based on multi-branch feature fusion refinement, which includes a memory, a processor and a computer program stored on the memory and executable on the processor. The image segmentation network model of the medical image segmentation device is constructed according to the architecture of the medical image segmentation system based on multi-branch feature fusion refinement as described above, and the image segmentation network model is pre-trained. When the processor executes the program, the steps of the medical image segmentation method based on multi-branch feature fusion refinement as described above are realized; further, the target in the medical image is identified by combining the multi-level feature information, and the image of the more accurate target region is segmented.

[0060] The technical scheme provided by the application has the following beneficial effects:

[0061] The application provides a medical image segmentation method based on multi-branch feature fusion refinement, and a corresponding data processing system and device. In the scheme of the embodiment, first, a PVT model and a ResNext101 model are used to extract features of different sizes as inputs of the double-branch feature fusion module proposed by the application. Then, through a multi-scale attention mechanism, the features of the two branches are combined to obtain fused features that retain features closely related to the segmentation task. These features not only have good local details and global information, but also enable the model to better adapt to objects of different sizes in medical images. Then, the features are sent to a cross-layer feature refinement module. The module first extracts the global information of the fused features through preliminary refinement, and then generates cross-layer refined features by integrating information from multiple different layers. Finally, deep refined features and cross-layer refined features are obtained.

[0062] Finally, in the decoder upsampling process, the scheme of the present application combines the cross-layer refined features and the refined features of the deepest layer with the decoding layer in a manner similar to a skip connection, corrects the information lost during the pooling operation, and finally obtains a more accurate segmentation result. In addition, in the training stage of the network model of the present application, the labels of different samples are changed to the same size, and the cross-entropy loss is calculated after each decoding layer is activated and the label, as side supervision, is combined with the end-to-end cross-entropy loss to optimize the model parameters, so that the segmentation result is more accurate. BRIEF DESCRIPTION OF DRAWINGS

[0063] The accompanying drawings are included to provide a further understanding of the present application, and constitute a part of the specification, illustrate the present application, and are used to explain the present application, and do not constitute a limitation on the present application. In the drawings:

[0064] Figure 1 A principle framework diagram of a medical image segmentation method based on multi-branch feature fusion refinement provided in embodiment 1 of the present application.

[0065] Figure 2 A step flowchart of a medical image segmentation method based on multi-branch feature fusion refinement provided in embodiment 1 of the present application.

[0066] Figure 3 A module processing flowchart of a feature fusion process used in embodiment 1 of the present application.

[0067] Figure 4 A module structure schematic diagram of a multi-scale channel attention module used in embodiment 1 of the present application.

[0068] Figure 5 A processing flowchart of a feature refinement process used in embodiment 1 of the present application.

[0069] Figure 6 A processing flowchart of a cross-layer feature refinement process used in embodiment 1 of the present application.

[0070] Figure 7 A system architecture diagram of a medical image segmentation system based on multi-branch feature fusion refinement provided in embodiment 2 of the present application.

[0071] Figure 8 Image segmentation instances of the present application and other network models on the Glas dataset in the performance test process.

[0072] Figure 9 Image segmentation instances of the present application and other network models on the PanNuke dataset in the performance test process. EMBODIMENT

[0073] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0074] Embodiment 1

[0075] The embodiment provides a medical image segmentation method based on multi-branch feature fusion refinement, which is used for identifying a target in a medical image in combination with multi-level feature information, and further segmenting a more accurate target region image.

[0076] The principle of the image segmentation scheme provided by the embodiment is as shown in Figure 1 Firstly, PVT pre-training models and ResNext101 pre-training models are used to extract features of the input medical image, the outputs of each layer of the two main stems are taken as inputs, and fusion features are obtained through a newly designed double-branch feature fusion module. And taking all the fusion features as inputs, the cross-layer feature refinement module is designed to analyze the features of different levels, and the global information is extracted; cross-layer refined features are obtained. Finally, the cross-layer refined features and the refined features of the deepest layer are added to the decoder for upsampling, and the feature map is gradually restored to the original resolution, and finally the segmentation result image is output.

[0077] Specifically, as shown in Figure 2 The detailed processing process of the medical image segmentation method based on multi-branch feature fusion refinement provided by the present application includes the following steps:

[0078] S1: The pre-trained PVT model and the ResNext101 model are used to extract features of the input medical image simultaneously, and two original features are obtained, each including four feature maps of different levels; size uniformization operation is performed on the two original features from different sources.

[0079] The PVT model extracts four feature maps of different sizes: pm1, pm2, pm3, and pm4. The ResNext101 model extracts four feature maps of different sizes: rm1, rm2, rm3, and rm4. In the size uniformization operation of step S1, the feature map shape output by the PVT model is used as a reference to adjust the size of the original feature map output by the corresponding layer of the ResNext101 model, so that the sizes of the feature maps output by the two models are matched.

[0080] S2: The features of each corresponding layer in the two size-uniformized feature maps are fused to obtain corresponding fusion features F. As shown in Figure 3 The fusion process is as follows:

[0081] S21: two feature maps F p , F r of each corresponding layer extracted by the PVT model and the ResNext101 model are added; a feature map X 1, is obtained 2, 3, X4.

[0082] S22: a multi-scale channel attention matrix M of each layer feature map after addition is extracted by a multi-scale channel attention module. The network framework of the multi-scale channel attention module used in this embodiment is as shown in Figure 4 .

[0083] S23: after the multi-scale channel attention matrix M is multiplied by the respective input features, the two feature maps obtained are added again.

[0084] S24: the feature map obtained after addition in the previous step is subjected to a 3x3 convolution and a batch normalization operation, and finally a fusion feature F pr is obtained by ReLU activation.

[0085] In step S2 of this embodiment, the operation process of calculating the multi-scale channel attention matrix M(X) of the feature map X by the multi-scale channel attention module is represented in detail as follows:

[0086]

[0087] In the above formula, g(X) represents global average pooling of the feature map X, δ represents the ReLU activation function, represents the batch normalization operation, PWConv represents the point convolution operation, represents the addition operation between feature maps, and X represents the result of adding two branch feature maps; G(X) and L(X) represent two different operations on the X feature respectively; the G(X) branch uses global average pooling and pays more attention to large objects in global distribution; the L(X) branch maintains the original feature size and pays more attention to local detail information, so that the segmentation boundary is clearer and more accurate while ensuring that small objects are not ignored.

[0088] Further, the fusion feature F pr is calculated according to the two original feature maps F p , F r extracted by the PVT model and the ResNext101 model. The calculation formula of the fusion feature F pr is as follows:

[0089]

[0090] In the above formula, represents element-wise multiplication; represents the original feature map F p , Fr After size unification, addition is performed.

[0091] S3: fully excavate global context information in each layer fusion feature F pr , and then output each layer fusion feature F pr corresponding refined feature F e . As shown in Figure 5 , the generation process of refined feature F e is as follows:

[0092] S31: after the fusion feature F e is respectively processed by a 3x3 convolution and an average pooling layer, feature map F c and feature map F a are obtained.

[0093] S32: multi-scale channel attention matrices M c and M a of feature map F c and feature map F a are extracted by a multi-scale channel attention module.

[0094] S33: the multi-scale channel attention matrices M c and M a are fused with F c and F a by element-wise multiplication, to obtain feature map F cm and F am .

[0095] S34: the feature map of F am is restored to the same size as F cm by upsampling, and the feature maps of the two branches are directly fused by addition to obtain F cam .

[0096] S35: the fusion feature F pr is fused with F cam by a residual structure to obtain refined feature F e .

[0097] In the present application, the operation process of generating corresponding refined feature F pr based on each layer fusion feature F e is specifically represented as follows:

[0098]

[0099] In the above formula, A(·) represents an average pooling operation; U(·) represents an upsampling operation; and Conv represents a convolution operation.

[0100] S4: build long-range correlation between each refined feature obtained in the previous step, and perform cross-layer refinement operation on four different levels of refined features; and output three cross-layer refined features. As shown in Figure 6 the processing flow of the cross-layer feature refinement model includes the following steps:

[0101] S41: assuming that the four different levels of refined features obtained are F'1, F'2, F'3, and F'4, flatten them in the spatial dimension, and then deform them to the same number of channels.

[0102] S42: splice the four refined features in the flattened spatial dimension.

[0103] S43: after the spliced features are normalized by the LN layer, they are processed by the self-attention layer and the feedforward neural network MixFFN to build the long-range correlation between the features.

[0104] S44: divide the output of the feedforward neural network in the previous step into three cross-layer refined features, and deform the cross-layer refined features into the shape of the corresponding decoding layer at the decoding end, and output to the decoding end.

[0105] In the present application, the processing process of the cross-layer feature refinement operation is as follows:

[0106]

[0107] In the above formula, F i represents the refined feature obtained according to the fusion feature of different levels; i represents the level mark of each fusion feature, i = 1, 2, 3, 4; tokenF i represents the operation result of deforming and reshaping the feature map to the same shape; B represents the data amount of each batch; C represents the number of channels; Reshape represents the deformation operation; Concatenate represents the feature splicing operation; merge_token represents the operation result of feature splicing; Efficient_Atten represents the effective self-attention module; Atten_token represents the operation result of the effective self-attention module; LN represents the layer normalization operation; res_token represents the operation result of the layer normalization; MixFFN represents the feedforward neural network; FFN i represents the result processed by the feedforward neural network; Split represents the operation of dividing and restoring the original shape; split_token represents the operation result of dividing and restoring the original shape of res_token; output represents the overall output of the cross-layer feature refinement operation.

[0108] S5: Steps S1-S4 are taken as the operation of the encoding end; in the upsampling process of the decoding end, the cross-layer refined features and the refined features of the deepest layer are combined with the decoding layer through a jump connection to correct the information lost during the pooling operation, and finally the segmentation result of the medical image is obtained.

[0109] Specifically, the decoding process of step S5 adopts the following decoding strategy:

[0110] (1) The refined features of the deepest layer are taken as the features of the first layer of the decoder to participate in the decoder analysis process;

[0111] (2) The deepest layer of the refined features is deformed to the same shape as the second, third and fourth decoding layers of the decoder through convolution and pooling operation once;

[0112] (3) The three cross-layer refined features output by the cross-layer feature refinement operation are respectively transmitted to the first, second and third decoding layers of the decoder;

[0113] (4) In each decoding layer, the corresponding refined features are combined with the corresponding decoding layer and cross-layer refined features to eliminate the ambiguity between the encoding end and the decoding end and effectively alleviate the loss of detailed information;

[0114] (5) The predicted image segmentation result is output through the last decoding layer.

[0115] In particular, the medical image segmentation method based on multi-branch feature fusion refinement designed by the application is an image segmentation processing scheme using a network model with an "encoder-decoder" architecture. In the training phase of the built image segmentation network model, the overall network parameters are optimized using side loss and end-to-end loss.

[0116] During the training process, the label is deformed to the size corresponding to each layer of the decoding layer, and the cross-entropy loss is calculated; the cross-entropy loss is added to the loss function between the side supervision and the segmentation result and the label, so as to optimize the overall network parameters through back propagation.

[0117] In the training phase, the network model is trained using pre-prepared sample images of the training set until the loss function converges. The segmentation result is predicted on the test set during each training, and the best model is saved according to the index.

[0118] Embodiment 2

[0119] The embodiment provides a medical image segmentation system based on multi-branch feature fusion refinement, which adopts the medical image segmentation method based on multi-branch feature fusion refinement as described above to analyze and process the input medical image, and then output the image segmentation result of the recognized and predicted target region.Figure 7 As shown in the figure, the type of medical image segmentation system based on multi-branch feature fusion refinement includes a first feature extraction module, a second feature extraction module, a size adjustment module, a feature fusion module, a feature refinement module, a cross-layer feature refinement module, and a decoder.

[0120] The first feature extraction module is used as the backbone network in one of the branches. The first feature extraction module uses the PVT model as the feature extraction network and is used to obtain the original feature maps of the input medical image at four different levels.

[0121] The second feature extraction module is used as the backbone network in the other branch. The second feature extraction module uses the ResNext101 model as the feature extraction network and is used to obtain the original feature maps of the input medical image at four different levels.

[0122] The size adjustment module is used to adjust the size of the original feature maps output by the ResNext101 model according to the shape of the feature maps output by the PVT model, so that the sizes of the feature maps output by the two models match.

[0123] The network framework of the feature fusion module is as shown in the figure Figure 3 The feature fusion module is used to add the feature maps of the corresponding layers extracted by the two backbone networks; then the importance of the features is calculated by the multi-scale channel attention module to obtain a multi-scale channel attention matrix, which is multiplied by the input features to highlight important features and suppress features unrelated to the task; then the two feature maps obtained at each layer are added again, and a 3x3 convolution and batch normalization operation is performed, and finally the fusion features F pr are obtained by ReLU activation.

[0124] The network framework of the feature refinement module is as shown in the figure Figure 5 The feature refinement module is used to obtain the fusion features output by the feature fusion module, then process the fusion features through 3x3 convolution and average pooling layer to obtain feature maps F c and F a . Next, learn the multi-scale attention-based feature expression through the multi-scale channel attention module, and obtain the multi-scale channel attention matrices M c and M a . Through element-wise multiplication, F c and F a are fused to obtain feature maps F cm and F am ; then the feature map of F am is restored to the same size as F cm by upsampling, and the feature maps of the two branches are directly fused by addition to obtain F cam . Finally, the fused features Fpr With F cam Fusion yields refined characteristic F e .

[0125] The network framework of the cross-layer feature refinement module is as follows: Figure 6 As shown, the four-layer refined features F′1, F′2, F′3, and F′ are used to obtain the output of the feature refinement module. 41,2,3 ,,4, and use F′1~F′4 as input to the cross-layer feature refinement module. The cross-layer feature refinement module first flattens each feature map in the spatial dimension, then transforms it to the same number of channels, and concatenates four refined features in the flattened spatial dimension. Next, after normalization by an LN layer, it passes through a self-attention layer and a feedforward neural network MixFFN to construct long-distance correlations, obtaining the cross-layer refined features. Finally, the cross-layer refined features are transformed into the shapes of the corresponding decoding layers in the decoder and output to the decoder;

[0126] Combination Figure 1 It can be seen that the decoder is used to obtain cross-layer refined features and the deepest refined features, and then combines each feature map with the decoding layer through skip connections to correct the information lost during the pooling operation, and finally obtain the segmentation result of the medical image. The processing of the encoder includes: (1) taking the deepest refined features as the features of the first layer of the decoder and participating in the decoder analysis process. (2) transforming the deepest refined features into the same shape as the second, third and fourth decoding layers of the decoder through convolution and pooling operations. (3) passing the three cross-layer refined features output by the cross-layer feature refinement operation to the first, second and third decoding layers of the decoder respectively. (4) combining the corresponding refined features with the corresponding decoding layer and cross-layer refined features in each decoding layer. (5) outputting the predicted image segmentation result through the last decoding layer.

[0127] Example 3

[0128] This embodiment provides a medical image segmentation device based on multi-branch feature fusion and refinement, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The medical image segmentation device constructs an image segmentation network model corresponding to the architecture of the aforementioned medical image segmentation system based on multi-branch feature fusion and refinement, and the image segmentation network model is pre-trained. When the processor executes the program, it implements the steps of the aforementioned medical image segmentation method based on multi-branch feature fusion and refinement; then, it combines multi-level feature information to identify targets in the medical image and segment more accurately the target regions.

[0129] The computer device can be a smartphone, a tablet computer, a notebook computer, a desktop computer, a rack server, a blade server, a tower server, or a cabinet server (including a standalone server or a server cluster composed of multiple servers), etc. that can execute programs. The computer device of the embodiment at least includes, but is not limited to, a memory and a processor that can be connected to each other in communication through a system bus.

[0130] In the embodiment, the memory (i.e., a readable storage medium) includes a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory can be an internal storage unit of the computer device, such as a hard disk or a memory of the computer device. In other embodiments, the memory can also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device. Of course, the memory can include both the internal storage unit and the external storage device of the computer device. In the embodiment, the memory is generally used to store an operating system and various application software installed on the computer device, etc. In addition, the memory can also be used to temporarily store various data that have been output or will be output.

[0131] The processor in some embodiments can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips. The processor is generally used to control the overall operation of the computer device.

[0132] Performance test

[0133] In order to verify the effectiveness of the technical solutions provided in Embodiments 1-3, the technical personnel carried out corresponding simulation experiments, built a corresponding network model, and trained and tested the network model.

[0134] In the simulation experiment, the experimental environment is based on Windows 10, the PyTorch 1.8.1 version deep learning framework is used, the programming environment is python 3.7, the hardware device used in the experiment is an Nvidia 3080 graphics card with 10 GB of video memory, the CPU uses Intel(R) Core(TM) i7-10700 model, and the data set uses GlaS, PanNuke, and TNBC medical image data sets.

[0135] In the simulation example, the skilled person in the art also takes the segmentation scheme of the traditional CNN network and multiple different image segmentation networks based on the Transformer network as the control group to perform performance comparison, in the performance comparison, Figure 8 The comparison results of the embodiment method with other advanced methods on the GlaS data set are shown in the following table. Figure 9 The comparison results of the embodiment method with other advanced methods on the PanNuke data set are shown in the following table.

[0136] The evaluation similarity coefficient mDice and the evaluation index Miou of semantic segmentation are selected as the performance evaluation indexes of the scheme. The measurement results of different image segmentation schemes are shown in the following table:

[0137] Table 1: Performance comparison results of each network model in the example and the control group

[0138]

[0139] From the analysis of the above table data, the values of mDice and Miou of the scheme provided in the embodiment are better than those of other traditional schemes in the GlaS, PanNuke and TNBC three different medical image data sets, which reflects that the scheme provided in the embodiment has very outstanding image segmentation performance.

[0140] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A medical image segmentation method based on multi-branch feature fusion refinement, which is used to identify a target in a medical image in combination with multi-level feature information, and further segment a more accurate target region image; characterized in that, The medical image segmentation method based on multi-branch feature fusion refinement comprises the following steps: S1: synchronously extracting features of an input medical image by a pre-trained PVT model and a ResNext101 model to obtain two original features, each comprising four feature maps of different levels; and performing size unification on the two original features of different sources; S2: fusing the features of each corresponding layer in the two size-unified feature maps to obtain corresponding fused features F, and the fusion process is as follows: S21: add the two feature maps F p , r ​ S22: extracting a multi-scale channel attention matrix M of each layer feature map after addition by a multi-scale channel attention module; S23: multiplying the multi-scale channel attention matrix M with the respective input features, and then adding the obtained two feature maps again; S24: The feature map after the addition in the previous step is subjected to a 3x3 convolution and batch normalization operation, and finally a ReLU activation is performed to obtain the fused feature F pr ; S3: fully excavate the global context information in each layer fusion feature F pr , and then output each layer fusion feature F pr Corresponding refined feature F e ; the generation process of the refined feature F e is as follows: S31: fusing the features F e After being processed by 3x3 convolution and average pooling layer respectively, the feature maps F c and F a are obtained. S32: extract the feature map F through the multi-scale channel attention module c and the multi-scale channel attention matrix of the feature map F a ​ M c and M a ; S33: obtaining a multi-scale channel attention matrix M c and M a by element-wise multiplication with F c and F a fusion, to obtain a feature map F cm and F am ; S34: F am is recovered to the same size as F cm by upsampling, and F cam is obtained by directly fusing the feature maps of the two branches through addition. S35: obtain refined features F by fusing features F and residual structure R pr with F cam fusion, obtaining refined features F e ; S4: constructing long-distance correlations among the refined features obtained in the previous step, performing cross-layer refinement on the four refined features of different levels, and outputting three cross-layer refined features; the process is as follows: S41: assuming that the four refined features of different levels obtained are F'1, F'2, F'3, and F'4, flattening them in the spatial dimension, and then deforming them to the same channel number; S42: splicing the four refined features in the flattened spatial dimension; S43: normalizing the spliced features by an LN layer, and then processing them by a self-attention layer and a feedforward neural network MixFFN to construct long-distance correlations among the features; S44: dividing the output of the feedforward neural network in the previous step into three cross-layer refined features, deforming the cross-layer refined features into the shapes of the corresponding decoding layers of the decoding end, and outputting them to the decoding end; S5: taking steps S1-S4 as the operation of the encoding end; in the upsampling process of the decoding end, the cross-layer refined features and the refined feature of the deepest layer are combined with the decoding layers by a skip connection to correct the information lost during the pooling operation, and finally the segmentation result of the medical image is obtained. 2.The medical image segmentation method based on multi-branch feature fusion refinement of claim 1, wherein: In the size unification operation of step S1, the size of the original feature map of the corresponding layer output by the ResNext101 model is adjusted according to the shape of the feature map output by the PVT model, so that the sizes of the feature maps output by the two models match. 3.The medical image segmentation method based on multi-branch feature fusion refinement of claim 2, wherein: In step S2, the operation process of calculating the multi-scale channel attention matrix M(X) of the feature map X by the multi-scale channel attention module is as follows: In the above formula, g(X) represents global average pooling on feature map X, and δ represents the ReLU activation function, represents a batch normalization operation, and PWConv represents a point convolution operation, represents an addition operation between feature maps, and X represents the result of adding two branch feature maps; G(X) and L(X) represent two different operations on the X feature, respectively; the G(X) branch uses global average pooling and pays more attention to large objects in the global distribution; the L(X) branch maintains the original feature size and pays more attention to local detailed information, so that the segmentation boundary is clearer and more accurate while ensuring that small objects are not ignored. 4.The medical image segmentation method based on multi-branch feature fusion refinement of claim 3, wherein: In step S2, two original feature maps F p , F r are extracted according to the PVT model and the ResNext101 model. The calculation formula of the fusion feature F pr is as follows: In the above formulae, denotes an element-wise multiplication; denotes adding the original feature maps F p , F r after size unification. 5.The medical image segmentation method based on multi-branch feature fusion refinement of claim 4, wherein: In step S3, the fusion features F pr corresponding refined features F e are generated based on the fusion features F of each layer. The operation process is represented in detail as follows: In the above formula, A(·) represents the average pooling operation; U(·) represents the upsampling operation; and Conv represents the convolution operation. 6.The medical image segmentation method based on multi-branch feature fusion refinement of claim 1, wherein: In step S4, the processing process of the cross-layer feature refinement operation is as follows: In the above formula, F i denotes the refined features obtained according to the fusion features of different levels; i denotes a level mark of each fusion feature, i = 1, 2, 3, 4; tokenF i denotes the operation result of deforming and reshaping the feature map to the same shape; B denotes the data amount of each batch; C denotes the channel number; Reshape denotes a deformation operation; Concatenate denotes a feature splicing operation; merge_token represents the operation result of feature splicing; Efficient_Atten represents an effective self-attention module; Atten_token represents the operation result of the effective self-attention module; LN represents a layer normalization operation; res_token represents the operation result of the layer normalization; MixFFN represents a feedforward neural network; FFN i represents the result processed by the feedforward neural network; Split represents a division and original shape restoration operation; split_token represents the operation result of dividing and restoring the original shape of res_token; output represents the overall output of the cross-layer feature refinement operation. 7.The medical image segmentation method based on multi-branch feature fusion refinement of claim 1, wherein: The decoding process of step S5 adopts the following decoding strategy: (1) taking the refined feature of the deepest layer as the feature of the first layer of the decoder to participate in the decoder analysis process; (2) deforming the feature of the deepest layer of the refined feature to the same shape as the second, third, and fourth decoding layers of the decoder by convolution and pooling operation once; (3) transmitting the three cross-layer refined features output by the cross-layer feature refinement operation to the first, second, and third decoding layers of the decoder, respectively; (4) In each decoding layer, the corresponding refined feature is combined with the corresponding decoding layer and the cross-layer refined feature to eliminate the ambiguity of the encoding end and the decoding end, effectively alleviating the loss of detailed information; (5) The predicted image segmentation result is output through the last decoding layer. 8.The medical image segmentation method based on multi-branch feature fusion refinement of claim 1, wherein: The medical image segmentation method based on multi-branch feature fusion refinement is a network model using an "encoder-decoder" architecture for image segmentation processing. In the training stage of the built image segmentation network model, the overall network parameters are optimized using side loss and end-to-end loss. The label is deformed to the size corresponding to each layer of the decoding layer, and the cross-entropy loss is calculated. The cross-entropy loss is added to the loss function between the side supervision and the label to optimize the overall network parameters through back propagation. In the training stage, the network model is trained using pre-prepared sample images in the training set until the loss function converges. The segmentation result is predicted on the test set during each training, and the best model is saved according to the index. 9.A medical image segmentation system based on multi-branch feature fusion refinement, characterized in that, It adopts the medical image segmentation method based on multi-branch feature fusion refinement as claimed in any one of claims 1-8 to analyze and process the input medical image, and then output the image segmentation result of the recognized and predicted target region. The medical image segmentation system based on multi-branch feature fusion refinement comprises: A first feature extraction module as the backbone network in one of the branches; the first feature extraction module uses a PVT model as a feature extraction network and is used to obtain four different levels of original feature maps of the input medical image; A second feature extraction module as the backbone network in another branch; the second feature extraction module uses a ResNext101 model as a feature extraction network and is used to obtain four different levels of original feature maps of the input medical image; A size adjustment module for adjusting the size of the original feature map output by the ResNext101 model according to the shape of the feature map output by the PVT model, so that the sizes of the feature maps output by the two models match; The feature fusion module is configured to add the feature maps of the corresponding layers extracted by the two backbone networks, calculate the importance of the features in the added feature maps by a multi-scale channel attention module to obtain a multi-scale channel attention matrix, multiply the multi-scale channel attention matrix with the input features to highlight important features and suppress features irrelevant to the task, add the two feature maps obtained at each layer again, perform a 3x3 convolution and a batch normalization operation, and finally obtain the fusion features F by ReLU activation pr ​ a feature refining module, which is used to first obtain the fused features of each layer output by the feature fusion module, and then obtain a feature map F after the fused features are respectively processed by a 3*3 convolution and an average pooling layer c and the feature map F a ; then, a multi-scale channel attention module is used to learn multi-scale attention-based feature expression, and a multi-scale channel attention matrix M c and M a is obtained c is obtained by element-wise multiplication with F a and F cm is fused to obtain a feature map F am and F am ; the feature map of F cm is restored to the same size as F cam by upsampling, and the feature maps of the two branches are directly fused by addition to obtain F pr ; finally, the fused features F cam and F e are fused by a residual structure to obtain refined features F A cross-layer feature refinement module for obtaining four layers of refined features F'1, F'2, F'3, F'4 output by the feature refinement module and taking F'1-F'4 as inputs of the cross-layer feature refinement module; the cross-layer feature refinement module first flattens each feature map in the spatial dimension, then deforms it to the same number of channels, and splices the four refined features in the flattened spatial dimension; then, after LN layer normalization, it is constructed through a self-attention layer and a feedforward neural network MixFFN to build long-distance correlation to obtain cross-layer refined features; finally, the cross-layer refined features are deformed into the shape of the corresponding decoding layer in the decoder and output to the decoder. The decoder is used to obtain the cross-layer refined features and the refined features of the deepest layer, and then combine each feature map with the decoding layer through the way of jump connection, correct the information lost during the pooling operation, and finally obtain the segmentation result of the medical image. The processing process includes: (1) taking the refined features of the deepest layer as the features of the first layer of the decoder to participate in the analysis process of the decoder; (2) deforming the deepest layer of the refined features to the same shape as the second, third and fourth decoding layers of the decoder through convolution and pooling operation once; (3) transmitting the three cross-layer refined features output by the cross-layer feature refining operation to the first, second and third decoding layers of the decoder respectively; (4) combining the corresponding refined features with the corresponding decoding layer and cross-layer refined features in each decoding layer; (5) outputting the predicted image segmentation result through the last decoding layer. 10.A medical image segmentation device based on multi-branch feature fusion refinement, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: The medical image segmentation device is constructed with an image segmentation network model corresponding to the architecture of the medical image segmentation system based on multi-branch feature fusion refinement according to claim 9, and the image segmentation network model is pre-trained; The processor executes the program to realize the steps of the medical image segmentation method based on multi-branch feature fusion refinement according to any one of claims 1 to 8; further combining multi-level feature information to identify the target in the medical image and segment the more accurate target region image.

Images