Placenta image segmentation method based on learnable visual center mechanism and multilayer perceptron

Abstract

The invention relates to a placenta image segmentation algorithm based on a learnable visual center mechanism and a multilayer perceptron. Automatic segmentation of a placenta can be realized by using fetal magnetic resonance imaging data. According to the algorithm, an encoder-decoder architecture is used, and a marked multilayer perceptron module is introduced to capture a long-distance remote dependency relationship and a local corner region of global features, so that complementation of global and local feature information is realized, and the defects of traditional convolution operation in the aspect are overcome; in addition, the model also uses a learnable visual center mechanism, all front superficial layer features are regulated and controlled by aggregating local key regions of an input image and utilizing context information of an interested region obtained by deep layer features, and feature fusion between levels from top to bottom is realized. Experimental results show that the provided algorithm achieves a good effect on a placenta image segmentation task.

Description

Technical Field

[0001] The present invention belongs to the field of medical image processing, in particular to image segmentation and other related technologies, and is a method for achieving better global and local feature information interaction and high-precision placenta image segmentation. Background Art

[0002] With the development of science and technology, Convolutional Neural Network (CNN), as a deep learning technology, is increasingly being applied to medical image analysis to help doctors reduce their workload, improve the accuracy of doctors' diagnosis, and make more consistent treatment plans. In the field of placental image segmentation, the automatic segmentation algorithm based on convolutional neural network replaces the tedious and boring image segmentation work of manual work, and has a unified training standard, which enables computers to assist doctors to quickly make more accurate treatment plans. At the same time, it also helps pregnant women to have a more accurate and intuitive understanding of the growth and development of the fetus in the abdomen, provides a basis for the prediction and prevention of placental diseases during pregnancy, and can also reduce the consumption of manpower and material resources for hospitals. However, in current research, the results of computer placental image segmentation can only be used as a reference for doctors, and cannot be used as a basis for diagnosis. The segmentation performance of the algorithm is far from reaching the segmentation level of professional physicians.

[0003] With the development of research, the algorithms used for placental image segmentation have become more and more varied, but the segmentation results have never reached the level of human segmentation and can only be used as a reference for doctors' diagnosis. Therefore, it is of great significance to develop a high-precision placental image segmentation algorithm. In recent years, many experts and scholars have conducted extensive research on medical image segmentation technology. With the development of high-performance computers, medical image segmentation-related technologies have also been further improved. At present, the commonly used methods for medical image segmentation include traditional machine learning methods and methods based on convolutional neural networks, among which the methods based on convolutional neural networks perform better. In medical data, different tissue cells have different shapes and functional characteristics, so the feature information extracted from different tissue images is also different. Researchers usually use convolutional neural networks to extract feature information of the region of interest, and then use deconvolution operations to reconstruct tissue and organ images. The boundary features of the reconstructed tissue and organ images are relatively clear, so that the tissues can be separated to achieve the purpose of segmentation.

[0004] However, the simple convolutional network ignores the effective extraction of global information, which is an inherent defect of convolution. Due to the uncertainty of the position, size and shape of the placenta, a single feature scale cannot meet the requirements of high-precision segmentation of placental images. Feature pyramid methods (such as SSD and FFP) have effectively achieved satisfactory results. The unifying principle behind these methods is to assign a region of interest with appropriate contextual information to each object of different sizes and enable these objects to be identified in different feature layers. The interaction between pixels or objects is crucial in the task of image feature extraction. For example, FPN proposes a top-down inter-layer feature interaction mechanism that enables shallow features to obtain the global contextual information and semantic representation of deep features. NAS-FPN attempts to learn the network structure of the feature pyramid part through a network architecture search strategy to obtain scalable feature representation. Although the above methods have achieved certain success, they are all limited by the small receptive field inherent in the CNN backbone. Summary of the invention

[0005] The purpose of the present invention is to propose a placenta image segmentation algorithm based on a learnable visual center mechanism (LVC) and a multi-layer perceptron (MLP). Based on the proposed new algorithm, the features extracted by convolution can be effectively labeled and projected through a labeled MLP block. The shift operation of the labeled MLP greatly reduces the number of parameters and the computational complexity, and can generate better feature representations to help segmentation, so that not only the global long-term dependencies can be captured, but also the feature representations of local details can be effectively obtained, and finally high-precision placenta image segmentation can be achieved.

[0006] To achieve the above purpose, the present invention provides the following technical solution: a placenta image segmentation algorithm based on learning visual center mechanism and multi-layer perceptron, the solution is:

[0007] S1, preprocessing the original placenta image data to obtain a preprocessed placenta image data set;

[0008] S2. Construct an image segmentation network based on a learnable visual center mechanism and a labeled multi-layer perceptron;

[0009] In step S2, the proposed network consists of LVC and tokenized MLP; the MLP module is used to capture the long-range remote dependencies of global features, including dividing the features into H different partitions in the shift MLP stage and shifting them by j = 5 positions according to the specified axis; the features are modeled in the tokenized MLP stage, the features from the shallow separable convolution are converted into abstract tags, and then MLP is used to learn meaningful information for segmentation; at the same time, the spatial information of the input feature map is first compressed through the Squeeze Excitation (SE) module, and then the learned channel attention information is combined with the input feature map to finally obtain a feature map with channel attention; the LVC module is used to capture the local corner areas of the input image, and a bottom-up inter-layer feature interaction mechanism is realized.

[0010] In the MLP module, the shifted MLP first shifts the channel axis of the transformed features and then labels them, which helps the MLP to focus only on certain locations of the convolutional features, thereby inducing locality in the block; the labeled MLP block has 2 shift operations, shifting the features across width in one block and across height in another block; the features are modeled through the tokenized MLP, and the specific implementation steps are as follows: first, the features are transferred and projected into tokens. For tokenization, a kernel size of 3 is first used and the number of channels is changed to E, where E is the embedding dimension, which is a hyperparameter; then, these tokens are passed across width to the shifted MLP, where the hidden dimension of the MLP is the hyperparameter H; next, these features are passed through a depth-wise separable convolutional layer, and after the convolution, the result is activated using GELU, which has been shown to improve the results in the latest architectures such as ViT and BERT; then, the features are passed across height through another shifted MLP, which converts the dimension from H to O; here the original tags are used as residuals, residual connections are used, and then layer normalization is applied (Layer Normalization, LN) and pass the output features to the next block; the calculations in the tokenization MLP block can be summarized as:

[0011] X shift =Shift w (X); T w =Tokenize(X shift ),

[0012] Y=f(DWConv((MLP(T W )))),

[0013] Y shift =Shift H (Y); T H =Tokenize(Yshift ),

[0014] Y=f(LN(T W +MLP(GELU(T H ))))

[0015] In the LVC module, the input feature X with shape C×H×W is in Mapped into a set of C-dimensional features X in ={x1, x2, ..., x N}, where N = H × W is the total number of input features. Next, X in A unique codebook B = {b1, b2, ..., b K} contains two important components: K codewords; a set of smoothing factors S = {s1, s2, ..., s k} is the visual center; the specific implementation steps are: input feature X in First, it is encoded by a combination of a set of convolutional layers (consisting of 1×1 convolution, 3×3 convolution and 1×1 convolution), where the first 1×1 is used to reduce the channels of the input features, the last 1×1 is used to increase the channels of the output features, and the middle 3×3 has a smaller number of channels for input and output features; then the encoded features are passed through CBR (Con v +BN+ReLU) block, which consists of a 3×3 convolution with a batch normalization (BN) layer and a ReLU activation function, and the learned smoothing factor s is input into the codebook. k (i=1, 2, ..., K) can be used in the codebook to match X i (i=1,2,...,N) and the corresponding b K (i=1,2,...,K), and then generate a learnable weight based on the difference between the two, and multiply the weight by the difference to sum it up, and finally output it as a C-dimensional vector e. The information of the entire image relative to the kth codeword can be calculated as:

[0016]

[0017] where x i -b k is the difference between N C-dimensional feature vectors and K codeword vectors, at x i -b k Generated learnable weights:

[0018]

[0019] Among them, ||·|| represents the norm operation, s k represents the smoothing factor of the Kth codeword, sk ||x i -b k || 2 is the output value of the Kth codeword, then multiply the weight by x i -b k , to obtain information about the position of the pixel relative to the codeword, and then add these N results to get e k , which is the information of the entire image relative to the kth codeword; then, use To integrate all e k , where e k It contains batch normalization layer, ReLU activation function and average layer; finally, the K results are summed to get e, which is all the information about the entire image of K codewords, e k and e are both C-dimensional vectors, and the formula for e is as follows

[0020]

[0021] Then predict a set of influencing factors to highlight the categories that need to be highlighted. First, use full connection to map the feature e to a size of C×1×1 as the influencing factor; then, use the input feature X in Channel multiplication is performed between the influence factor coefficient δ(·); the characteristic representation obtained after highlighting is as follows:

[0022]

[0023] Where FC is the fully connected layer, δ(·) is the sigmoid function, It is channel-by-channel multiplication; finally, using the input feature X in Channel addition is performed between the local corner area feature Z, and the formula is:

[0024]

[0025] in It is a channel-by-channel addition.

[0026] In step S2, the SE module is a channel attention module, which can enhance the channel features of the input feature map without changing the size of the input feature map. The specific implementation steps are as follows: the dimension of the SE module input feature map is H×WXC, the feature map is first spatially compressed, and the spatial dimension is obtained by F sq To implement global average pooling, C feature maps are compressed into one value, so that the generated channel-level feature map contains context information, alleviating the problem of channel dependency; then, in order to utilize the information gathered in the compression operation, two layers of full connection F ex(·, W) for learning. The first fully connected layer compresses C channels into C / r channels to reduce the amount of calculation, where r is the compression ratio. Then, through a ReLU nonlinear activation layer, the second fully connected layer restores the number of channels to C channels, and then through Sigmoid activation to obtain the weight S. The dimension of S is 1×1×C, which is used to characterize the weights of C feature maps in the input features. Finally, the feature map of channel attention 1×1×C and the feature map of the original input H×W×C are multiplied by the weight coefficient channel by channel to obtain the output of the SE module with channel attention.

[0027] S3, inputting the preprocessed placenta image into the proposed image segmentation network for training to obtain a trained placenta image segmentation network model;

[0028] S4. Use the trained network model to segment the preprocessed placenta image to obtain the placenta segmentation result.

[0029] Compared with the prior art, the present invention has the following significant advantages: the network uses a tokenized MLP to obtain global long-distance dependencies, and the computational burden of the tokenized MLP is reduced by introducing a shift operation, thereby reducing the model complexity; at the same time, in order to improve the model's feature extraction capability in local corner areas and accelerate network convergence, LVC is further introduced to implement a bottom-up pyramid-like inter-layer feature fusion mechanism, and the deepest layer feature information is used to adjust all shallow features. The combination of the two is more comprehensive for feature extraction, and in order to highlight the relationship between different channels, the SE module is used to improve the modeling capability of the model. Experiments have shown that this method obtains better segmentation results than other methods, effectively realizes the complementarity of global and local features, and has a lightweight model size. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is an implementation flow chart of the present invention.

[0031] Figure 2 It is a diagram of the segmentation network structure of the present invention.

[0032] Figure 3 is a flowchart of the tokenized MLP calculation of the present invention.

[0033] Figure 4 It is a schematic diagram of the LVC framework of the present invention. DETAILED DESCRIPTION

[0034] The purpose of the present invention is to propose a placenta image segmentation algorithm based on a learnable visual center mechanism and a multi-layer perceptron, comprising the following steps:

[0035] refer to Figure 1The steps for implementing the invention include: (1) preprocessing the original placenta image data to obtain a preprocessed placenta image data set; (2) constructing an image segmentation network based on a learnable visual center mechanism and a labeled multi-layer perceptron; (3) inputting the preprocessed placenta image into the proposed image segmentation network for training to obtain a trained placenta image segmentation network model; (4) using the trained network model to segment the preprocessed placenta image to obtain a placenta segmentation result.

[0036] refer to Figure 2 , is the proposed network structure diagram, which consists of LVC and tokenized MLP; the MLP module is used to capture the long-range remote dependencies of global features, including dividing the features into H different partitions in the shift MLP stage and shifting them by i = 5 positions according to the specified axis; the features are modeled in the tokenized MLP stage, the features from the shallow separable convolution are converted into abstract tags, and then MLP is used to learn meaningful information for segmentation; at the same time, the spatial information of the input feature map is first compressed by the squeeze excitation (SE) module at the fourth layer jump connection, and then the learned channel attention information is combined with the input feature map to finally obtain a feature map with channel attention; the LVC module is used to capture the local corner areas of the input image, and at the same time, a bottom-up inter-layer feature interaction mechanism is realized.

[0037] refer to Figure 3 , is the flowchart of tokenized MLP calculation. In the MLP module, the shift MLP first shifts the channel axis of the transformed feature and then marks it, which helps the MLP to focus only on certain positions of the convolutional features, thereby inducing the locality of the block; the labeled MLP block has 2 shift operations, moving the features across the width in one block and across the height in another block; the features are modeled by the tokenized MLP. The specific implementation steps are: first transfer the features and project them into tokens. For tokenization, first use a kernel size of 3 and change the number of channels to E, where E is the embedding The tokens are then passed across width to a shifted MLP, where the hidden dimension of the MLP is the hyperparameter H. Next, the features are passed through a depth-wise separable convolutional layer, after which the result is activated using GELU, which has been shown to improve results in recent architectures such as ViT and BERT. The features are then passed across height through another shifted MLP, which converts the dimension from H to O. Here, the original tokens are used as residuals, residual connections are used, and then layer normalization (LN) is applied and the output features are passed to the next block. The calculations in the tokenized MLP block can be summarized as:

[0038] X shift =Shift w (X); T w =Tokenize(X shift ),

[0039] Y=f(DWConv((MLP(T W )))),

[0040] Y shift =Shift H (Y); T H =Tokenize(Y shift ),

[0041] Y=f(LN(T W +MLP(GELU(T H ))))

[0042] refer to Figure 4 , which is a schematic diagram of the LVC framework. In the LVC module, the input feature X with a shape of C×H×W is in Mapped into a set of C-dimensional features X in ={x1, x2, ..., x N}, where N = H × W is the total number of input features. Next, X in A unique codebook B = {b1, b2, ..., b K} contains two important components: K codewords; a set of smoothing factors S = {s1, s2, ..., s k} is the visual center; the specific implementation steps are: input feature X in First, it is encoded by a combination of a set of convolutional layers (consisting of 1×1 convolution, 3×3 convolution and 1×1 convolution), where the first 1×1 is used to reduce the channels of the input features, the last 1×1 is used to increase the channels of the output features, and the middle 3×3 has a smaller number of channels for input and output features; then the encoded features are processed by the CBR (Conv+BN+ReLU) block, and the encoded features are input into the codebook. The block consists of a 3×3 convolution with a batch normalization (Batch Normal, BN) layer and a ReLU activation function; the learned smoothing factor s k (i=1, 2, ..., K) can be used in the codebook to match X i (i=1,2,...,N) and the corresponding b K (i=1,2,...,K), and then generate a learnable weight based on the difference between the two, and multiply the weight by the difference to sum it up, and finally output it as a C-dimensional vector e. The information of the entire image relative to the kth codeword can be calculated as:

[0043]

[0044] where x i -b k is the difference between N C-dimensional feature vectors and K codeword vectors, at x i -b k Generated learnable weights:

[0045]

[0046] Among them, ||·|| represents the norm operation, s k represents the smoothing factor of the Kth codeword, s k ||x i -b k || 2 is the output value of the Kth codeword, then multiply the weight by x i -b k , to obtain information about the position of the pixel relative to the codeword, and then add these N results to get e k , which is the information of the entire image relative to the kth codeword; then, use To integrate all e k , where e k It contains batch normalization layer, ReLU activation function and average layer; finally, the K results are summed to get e, which is all the information about the entire image of K codewords, e k and e are both C-dimensional vectors, and the formula for e is as follows

[0047]

[0048] Then predict a set of influencing factors to highlight the categories that need to be highlighted. First, use full connection to map the feature e to a size of C×1×1 as the influencing factor; then, use the input feature x in Channel multiplication is performed between the influence factor coefficient δ(·); the characteristic representation obtained after highlighting is as follows:

[0049]

[0050] Where FC is the fully connected layer, δ(·) is the sigmoid function, It is channel-by-channel multiplication; finally, using the input feature X in Channel addition is performed between the local corner area feature Z, and the formula is:

[0051]

[0052] in It is a channel-by-channel addition.

[0053] In step S2, the SE attention module is a channel attention module. The SE module can enhance the channel features of the input feature map without changing the size of the input feature map. The specific implementation steps are as follows: the dimension of the SE module input feature map is H×W×C, the feature map is first spatially compressed, and the spatial dimension is obtained by F sq To implement global average pooling, C feature maps are compressed into one value, so that the generated channel-level feature map contains context information, alleviating the problem of channel dependency; then, in order to utilize the information gathered in the compression operation, two layers of full connection F ex (·, W) for learning. The first fully connected layer compresses C channels into C / r channels to reduce the amount of calculation, where r is the compression ratio. Then, through a ReLU nonlinear activation layer, the second fully connected layer restores the number of channels to C channels, and then through Sigmoid activation to obtain the weight S. The dimension of S is 1×1×C, which is used to characterize the weights of C feature maps in the input features. Finally, the feature map of channel attention 1×1×C and the feature map of the original input H×W×C are multiplied by the weight coefficient channel by channel to obtain the output of the SE module with channel attention.

[0054] The above contents are further detailed descriptions of the present invention in combination with the best embodiment, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. It should be understood by those skilled in the art that various modifications can be made in the details without departing from the scope of the appended claims, and all should be deemed to belong to the protection scope of the present invention.

Claims

1. A placenta image segmentation algorithm based on a learnable visual center mechanism (LVC) and a multi-layer perceptron (MLP), characterized by: S1, preprocessing the original placenta image data to obtain a preprocessed placenta image data set; S2. Construct an image segmentation network based on a learnable visual center mechanism and a labeled multi-layer perceptron; S3, inputting the preprocessed placenta image into the proposed image segmentation network for training, and obtaining a trained placenta image segmentation network model; S4. Use the trained network model to segment the preprocessed placenta image to obtain the placenta segmentation result.

2. The placenta image segmentation algorithm based on the learnable visual center mechanism and multi-layer perceptron according to claim 1 is characterized in that: In step S2, the proposed network is composed of LVC and tokenized MLP; The MLP module is used to capture the long-range dependencies of global features, including dividing the features into H different partitions in the shift MLP stage and shifting them by j = 5 positions according to the specified axis; modeling the features in the tokenization MLP stage, converting the features from the shallow separable convolution into abstract tags, and then using MLP to learn meaningful information for segmentation; at the same time, the spatial information of the input feature map is first compressed through the squeeze excitation (SE) module, and then the learned channel attention information is combined with the input feature map to finally obtain a feature map with channel attention; The LVC module is used to capture the local corner areas of the input image and realize the bottom-up inter-layer feature interaction mechanism.

3. The placenta image segmentation algorithm based on the learnable visual center mechanism and multi-layer perceptron according to claim 2 is characterized in that: In the MLP module, the shifted MLP first shifts the channel axis of the transformed features and then labels them, which helps the MLP focus only on certain positions of the convolutional features, thereby inducing locality of the block; the labeled MLP block has 2 shift operations, shifting the features across width in one block and across height in the other block.

4. The placenta image segmentation algorithm based on the learnable visual center mechanism and multi-layer perceptron according to claim 2 is characterized in that: When modeling features through tokenized MLP, the specific implementation steps are as follows: first transfer the features and project them into tokens. For tokenization, first use a kernel size of 3 and change the number of channels to E, where E is the embedding dimension, which is a hyperparameter. These tokens are then passed across width to a shifted MLP where the hidden dimension of the MLP is a hyperparameter H. Next, the features are passed through a depthwise separable convolutional layer, after which the result is activated using a GELU, which has been shown to improve results in recent architectures such as ViT and BERT. The features are then passed across height through another shifted MLP, which converts the dimension from H to O. Here the original tokens are used as residuals, residual connections are used, and then layer normalization (LN) is applied and the output features are passed to the next block. The computations in the tokenized MLP block can be summarized as: X shift =Shift w (X);T w =Tokenize(X shift ), Y=f(DWConv((MLP(T W )))), Y shift =Shift H (Y);T H =Tokenize(Y shift ), Y=f(LN(T W +MLP(GELU(T H ))))。 5. The placenta image segmentation algorithm based on the learnable visual center mechanism and multi-layer perceptron according to claim 2 is characterized in that: In the LVC module, the input feature X with shape C×H×W is in Mapped into a set of C-dimensional features X in ={x1,x2,…,x N }, where N = H × W is the total number of input features. Next, X in We will learn an inherent codebook B = {b1, b2, ..., b K } contains two important components: K codewords; a set of smoothing factors S = {s1, s2, ..., s k } is the visual center; the specific implementation steps are: input feature X in First, it is encoded by a combination of a set of convolutional layers (consisting of 1×1 convolution, 3×3 convolution, and 1×1 convolution), where the first 1×1 is used to reduce the channels of the input features, the last 1×1 is used to increase the channels of the output features, and the middle 3×3 has a smaller number of channels for input and output features; The encoded features are then processed through a CBR (Conv+BN+ReLU) block, which consists of a 3×3 convolution with a batch normalization (BatchNormal, BN) layer and a ReLU activation function, and the learned smoothing factor s is input into the codebook. k (i=1,2,…,K) can be used in the codebook to match X i (i=1,2,…,N) and the corresponding b K (i=1,2,…,K), and then generate a learnable weight based on the difference between the two, and multiply the weight by the difference to sum it up, and finally output it as a C-dimensional vector e.

6. The placenta image segmentation algorithm based on the learnable visual center mechanism and multi-layer perceptron according to claim 5 is characterized in that: The information of the entire image relative to the kth codeword can be calculated as: where x i -b k is the difference between N C-dimensional feature vectors and K codeword vectors, at x i -b k Generated learnable weights: Among them, ||·|| represents the norm operation, s k represents the smoothing factor of the Kth codeword, s k ||x i -b k || 2 is the output value of the Kth codeword, then multiply the weight by x i -e k , to obtain information about the position of the pixel relative to the codeword, and then add these N results to get e k , which is the information of the entire image relative to the kth codeword; then, use To integrate all e k , where e k It contains batch normalization layer, ReLU activation function and average layer; finally, the K results are summed to get e, which is all the information about the entire image of K codewords, e k and e are both C-dimensional vectors, and the formula for e is as follows Then predict a set of influencing factors to highlight the categories that need to be highlighted. First, use full connection to map the feature e to a size of C×1×1 as the influencing factor; then, use the input feature X in Channel multiplication is performed between the influence factor coefficient δ(·); the characteristic representation obtained after highlighting is as follows: Where FC is the fully connected layer, δ(·) is the sigmoid function, It is channel-by-channel multiplication; finally, using the input feature X in Channel addition is performed between the local corner area feature Z, and the formula is: in It is a channel-by-channel addition.

7. The placenta image segmentation algorithm based on the learnable visual center mechanism and multi-layer perceptron according to claim 2 is characterized in that: The SE module at the fourth layer of skip connection is a channel attention module, which can enhance the channel features of the input feature map without changing the size of the input feature map. The specific implementation steps are as follows: the dimension of the SE module input feature map is H×W×C, the feature map is first spatially compressed, and then F is used to enhance the channel features of the input feature map. sq To implement global average pooling, C feature maps are compressed into one value, so that the generated channel-level feature map contains context information, alleviating the problem of channel dependency; then, in order to utilize the information gathered in the compression operation, two layers of full connection F ex (·,W) for learning. The first fully connected layer compresses C channels into C / r channels to reduce the amount of calculation, where r is the compression ratio. Then, through a ReLU nonlinear activation layer, the second fully connected layer restores the number of channels to C channels, and then through Sigmoid activation to obtain the weight S. The dimension of S is 1×1×C, which is used to characterize the weights of C feature maps in the input features. Finally, the feature map of channel attention 1×1×C and the feature map of the original input H×W×C are multiplied by the weight coefficient channel by channel to obtain the output of the SE module with channel attention.

Images