Abdominal multi-organ segmentation method based on bridging encoder-decoder

Abstract

This invention discloses a method for abdominal multi-organ segmentation using a bridge encoder-decoder, implemented according to the following steps: Step 1: Preprocess the image; Step 2: Construct a two-stage segmentation framework from coarse to fine, selecting the SegNet network for the coarse segmentation stage and building the BridgeNet network for the fine segmentation stage; Step 3: Train the SegNet network using the preprocessed training data; Step 4: Segment the pre-trained training data using the trained SegNet to obtain the coarse segmentation result, then multiply it by the pre-trained training data to obtain the training data for the fine segmentation stage, and train the BridgeNet network; Step 5: Segment the test set using the trained SegNet and BridgeNet to obtain the preliminary segmentation result, and then apply connected component analysis to the preliminary segmentation result to obtain an accurate abdominal CT multi-organ segmentation map.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology and relates to a method for abdominal multi-organ segmentation based on a bridging encoder-decoder. Background Technology

[0002] Traditional multi-organ segmentation research employs statistical models and multi-atlas label fusion. These methods are easily affected by image registration accuracy and differences in organ morphology among patients, limiting their application in clinical settings. Deep learning-based methods, however, avoid the arduous task of manual feature engineering and have become the current trend in abdominal multi-organ image segmentation. Because multiple organs in abdominal CT images vary significantly in volume, size, and appearance, and their boundaries with surrounding organs and tissues are blurred with low contrast, existing methods suffer from inaccurate identification and segmentation of the organs to be segmented.

[0003] Simultaneous fusion of saliency information from multiple scales and dimensions in an image can reveal the overall relationships between multiple organs and the specific details of each organ, achieving more accurate spatial localization. This is currently an effective method for solving the challenge of simultaneously segmenting multiple organs with varying sizes and heights. However, networks cannot continuously learn deeper information at each scale and dimension. Previous abdominal multi-organ segmentation methods have not efficiently fused information from multiple scales and dimensions simultaneously and in parallel, thus having shortcomings in learning the features of multiple abdominal organs. Furthermore, previous methods did not dynamically mine salient features based on the extracted features, resulting in a lack of specificity and room for improvement in suppressing redundant information. In summary, achieving accurate segmentation of multiple abdominal organs, given their varying shapes, sizes, and contrasts, as well as the effects of spatial occlusion and blurred boundaries between adjacent organs, remains a significant challenge. Summary of the Invention

[0004] The purpose of this invention is to provide a method for abdominal multi-organ segmentation based on a bridging encoder-decoder. This method overcomes the influence of different shapes, sizes and contrasts of multiple organs in the abdomen, as well as spatial occlusion and blurred boundaries between adjacent organs, thereby accurately segmenting multiple organs in abdominal CT images.

[0005] The technical solution adopted in this invention is a method for abdominal multi-organ segmentation based on a bridging encoder-decoder, which specifically includes the following steps:

[0006] Step 1: Preprocess the images in the pre-divided training and test sets of the FLARE2021 dataset to obtain preprocessed training and test set images;

[0007] Step 2: Construct a two-stage abdominal multi-organ segmentation network framework, from coarse to fine, including the coarse segmentation model SegNet and the fine segmentation model BridgeNet.

[0008] Step 3: Train the coarse segmentation stage network SegNet using the training data preprocessed in Step 1 to obtain the trained coarse segmentation stage network SegNet.

[0009] Step 4: Based on the results of Step 3, train the fine segmentation model BridgeNet constructed in Step 2 to obtain the trained BridgeNet.

[0010] Step 5: Use the coarse segmentation model SegNet trained in Step 3 and the BridgeNet trained in Step 4 to segment the preprocessed test set data in Step 1 to obtain the segmented test set data; apply connected component analysis to the segmented test set data to obtain accurate abdominal multi-organ segmentation results.

[0011] The invention is further characterized by:

[0012] The specific process of step 1 is as follows:

[0013] Step 1.1: Input the FLARE2021 dataset and reorient the pre-divided training and test set images in the dataset to the target orientation.

[0014] Step 1.2: Resample all images in the dataset to a fixed size;

[0015] Step 1.3: Clip all images in the dataset to an intensity range of [-325, 325], and finally apply z-score normalization based on the mean and standard deviation of the intensity values.

[0016] In step 2, the BridgeNet fine segmentation model is constructed by embedding a bridging module between the encoder and decoder. This bridging module includes a many-to-one fusion module (MSFM) and a dual-stream adaptive module (DSAM).

[0017] The BridgeNet fine-segmentation model performs the following image segmentation process:

[0018] Step A) Use the encoder of SegNet as the encoder of BridgeNet to extract features and obtain the extracted features I.

[0019] Step B) involves performing convolution-regularization-activation processing on the feature I processed by the encoder to obtain the processed feature I'. Then, I' is fed into the many-to-one fusion module MSFM. The many-to-one fusion module MSFM includes a multi-scale global submodule MSG and a multi-dimensional local submodule MDL. The multi-scale global submodule MSG extracts global information at three scales in parallel. Local information in three dimensions is extracted in parallel using multi-dimensional local submodules (MDL). The specific formula is as follows:

[0020]

[0021]

[0022]

[0023]

[0024]

[0025]

[0026] Where GAP is the global average pooling operation, GMP is the global max pooling operation, and the value of N is the size of feature I';

[0027] Step C) The three parallel branch features extracted by the MSG submodule are further convolved and regularized using ConvIN, then upsampled to the same size and added together. The result of the addition is then further activated using ReLU. ConvINReLU processing is performed to obtain multi-scale features that fuse global features of three scales.

[0028] Step D) The three parallel branch features extracted from the MDL submodule are further convolved and regularized using ConvIN, then upsampled to the same size and added together. The result of the addition is then further activated using ReLU. ConvINReLU processing is performed to obtain multi-dimensional features that integrate local features of the three dimensions.

[0029] Step E) involves concatenating the multi-scale features obtained in step C) and the multi-dimensional features obtained in step D), followed by convolutional-regularized ConvIN to obtain fused multi-scale and multi-dimensional features.

[0030] Step F): Add an input feature I to the multi-scale, multi-dimensional features obtained in step E) to obtain the final feature X;

[0031] Step G) uses the dual-stream adaptive module DSAM to dynamically process feature X, X∈C×D×H×W, where C represents the number of channels of feature X, D represents the depth of feature X, H represents the height of feature X, and W represents the width of feature X, thus obtaining a preliminarily segmented abdominal multi-organ image.

[0032] The specific process of step G) is as follows:

[0033] Step G1) The feature X extracted by the many-to-one fusion module MSFM is split into two parts X1 and X2, and then fed into the spatial attention stream and the channel attention stream respectively to obtain the attention weights of the features extracted by MSFM. The two attention streams Att_S and Att_C are calculated by the following formulas (7) and (8). Then, the two attention streams Att_S and Att_C are merged and the channel random shuffle operation is used to enable cross-group information flow along the channel dimension:

[0034] Att_S=σ{FC[GAP(X1)]}·X1 (7);

[0035] Att_C=σ{FC[GN(X2)]}·X2 (8);

[0036] Where σ represents the sigmoid activation function, and FC represents a fully connected layer;

[0037] Step G2) uses two convolutional layers and a softmax activation function to generate normalized attention weights {π} for k convolutional kernels. k The formula for (x)} is as follows:

[0038]

[0039] Among them, z k This is the output of the second convolutional layer, and τ is the temperature;

[0040] Step G3) involves linearly combining k convolutional kernels using the obtained weights, and employing the dual-stream adaptive module DSAM to mine information from the multi-scale global features and multi-dimensional local features extracted by MSFM. The mined information Y is represented by the following formula:

[0041] Y = (π1W1 + π2W2 + ... + π) k W k )*X (10);

[0042] Among them, W i This indicates that the i-th convolutional kernel... m = 1, 2, ..., k; π i ∈R is the weighted W i Note the scalar, * indicates the convolution operation;

[0043] In step G4), the SegNet decoder is used as the BridgeNet decoder. The mining information obtained in step G3) is fed into the decoder and continuously upsampled until the original input image size is restored, thus obtaining a preliminarily segmented abdominal multi-organ image.

[0044] The beneficial effects of this invention are that it proposes a multi-organ segmentation method for abdominal CT images based on a bridging encoder-decoder. Building upon a two-stage coarse-to-fine segmentation framework, the proposed multi-to-one fusion module and dual-stream adaptive module can simultaneously and in parallel extract sufficient multi-scale global information and multi-dimensional local information, further uncovering salient information. This information is highly helpful in understanding the overall relationships between highly variable abdominal organs, the specific details of each organ, and their spatial location. Validation on publicly available datasets demonstrates that the method of this invention possesses high performance metrics and excellent results. Attached Figure Description

[0045] Figure 1 This is a flowchart illustrating the abdominal multi-organ segmentation method based on a bridging encoder-decoder according to the present invention.

[0046] Figure 2 This is a structural diagram of the complete method framework of the abdominal multi-organ segmentation method based on the bridging encoder-decoder of the present invention;

[0047] Figure 3 This is a structural diagram of the BridgeNet network used in the abdominal multi-organ segmentation method based on the bridge encoder-decoder of this invention.

[0048] Figure 4 This is a block diagram of the many-to-one fusion module (MSFM) in the bridging block of the abdominal multi-organ segmentation method based on bridging encoder-decoder of the present invention.

[0049] Figure 5 This is a block diagram of the dual-stream adaptive module (DSAM) in the bridging block of the abdominal multi-organ segmentation method based on bridging encoder-decoder of the present invention.

[0050] Figure 6 This is a comparison of the segmentation results of the abdominal multi-organ segmentation method based on the bridging encoder-decoder of the present invention. Detailed Implementation

[0051] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0052] A flowchart illustrating the abdominal multi-organ segmentation method based on a bridging encoder-decoder of this invention is shown below. Figure 1As shown, the specific steps include the following:

[0053] Step 1: Preprocess the images in the pre-divided training and test sets of the FLARE2021 dataset to obtain preprocessed training and test set images. Specifically:

[0054] Step 1.1, input the FLARE2021 dataset, which contains 511 contrast-enhanced abdominal CT images from 11 medical centers, of which 361 have publicly available corresponding labels. Each image is... It consists of slices of 512×512 pixels, with a resolution of [missing information]. The images in the pre-divided training and test sets of the dataset are sequentially reoriented to the target orientation;

[0055] Step 1.2: Resample all images to a fixed size to prepare for training the coarse segmentation model and the fine segmentation model. The input size of the coarse segmentation model is [160, 160, 160], and the input size of the fine segmentation model is [192, 192, 192].

[0056] Step 1.3: Clip all images to an intensity range of [-325, 325], and finally apply z-score normalization based on the mean and standard deviation of the intensity values.

[0057] Step 2 involves constructing a two-stage abdominal multi-organ segmentation network framework, from coarse to fine. The coarse segmentation stage uses the SegNet network, while the fine segmentation stage constructs a novel BridgeNet network. Specifically, BridgeNet is built by embedding a bridge block between the traditional encoder and decoder. This bridge block consists of a many-to-one fusion module (MSFM) and a two-stream adaptive module (DSAM).

[0058] The specific process of step 2 is as follows: Figure 2 As shown, the specific process is as follows:

[0059] Step 2.1: Select SegNet as the network for the coarse segmentation stage to perform preliminary segmentation of the image. This can enhance the discrimination information of target organs in the fine segmentation stage, assist the fine segmentation stage to perform more accurate segmentation, and reduce the influence of surrounding non-organ regions on the segmentation of multiple organs in the abdomen.

[0060] Step 2.2: Construct BridgeNet as the network for the fine segmentation stage. The proposed BridgeNet is as follows: Figure 3As shown, it consists of four main parts: a feature encoder module (Encoder), a many-to-one fusion module (MSFM), a two-stream adaptive module (DSAM), and a feature decoder module (Decoder);

[0061] The segmentation process of the BridgeNet network in the fine segmentation stage is as follows:

[0062] Step A) Using the traditional SegNet encoder as the BridgeNet encoder, features can be extracted to obtain the extracted feature I.

[0063] Step B) involves performing Convolution-Regularization-Activation (ConvINReLU) processing on the feature I processed by the encoder to obtain the processed feature I'; then, I' is fed into the many-to-one fusion module MSFM, as shown in step B. Figure 4 As shown, the multi-scale global submodule MSG in this module can extract global information at three scales in parallel, and the multi-dimensional local submodule MDL can extract local information in three dimensions in parallel. The formulas for the branches extracting global information in three dimensions and extracting local information in three dimensions are shown below:

[0064]

[0065]

[0066]

[0067]

[0068]

[0069]

[0070] Wherein, GAP is the global average pooling operation, GMP is the global max pooling operation, and the value of N is the size of feature I' (12 in this invention). The three parallel branch features extracted from the MSG submodule are further convolved and regularized (ConvIN), then upsampled to the same size and added together. The result is then further activated by ReLU and processed by ConvINReLU to obtain a multi-scale feature that integrates global features from three scales. Similarly, the three parallel branch features extracted from the MDL submodule are further convolved and regularized (ConvIN), then upsampled to the same size and added together. The result is then further activated by ReLU and processed by ConvINReLU to obtain a multi-dimensional feature that integrates local features from three dimensions. The integrated multi-scale feature and the integrated multi-dimensional feature are concatenated and then convolved and regularized (ConvIN) to obtain the integrated multi-scale multi-dimensional feature. Here, this invention also adds an input feature I, finally obtaining feature X.

[0071] Step C), using as Figure 5 The dual-stream adaptive module DSAM shown further dynamically processes feature X, X∈C×D×H×W, where C represents the number of channels of feature X, D represents the depth of feature X, H represents the height of feature X, and W represents the width of feature X.

[0072] The specific processing procedure for step C) is as follows:

[0073] Step C1) splits the feature X extracted by MSFM into two parts (X1, X2), and then feeds them into the spatial attention stream and the channel attention stream respectively to obtain the attention weights of the features extracted by MSFM. The two attention streams are calculated by the following formula:

[0074] Att_S=σ{FC[GAP(X1)]}·X1 (7);

[0075] Att_C=σ{FC[GN(X2)]}·X2 (8);

[0076] Here, σ represents the sigmoid activation function, and FC represents a fully connected layer. Then, the two streams are merged and a "channel shuffle" operation is used to enable cross-group information flow along the channel dimension.

[0077] Step C2) uses two convolutional layers (with a ReLU activation function between them) and a softmax activation function to generate normalized attention weights {π} for k convolutional kernels. kThe formula for (x)} is as follows:

[0078]

[0079] Among them, z k This is the output of the second convolutional layer, and τ is the temperature. As τ increases, the sparsity of the output decreases.

[0080] Step C3) involves linearly combining k convolutional kernels using the obtained weights. The combined convolutions are then used to further mine the sufficient multi-scale global features and multi-dimensional local features extracted by MSFM. This operation can be expressed by the following formula:

[0081] Y = (π1W1 + π2W2 + ... + π) k W k )*X (10);

[0082] Among them, W i This indicates that the i-th convolutional kernel... m = 1, 2, ..., k; π i ∈R is the weighted W i Note the scalar, which is calculated by equation (9) with input functionality as a prerequisite; * indicates the convolution operation.

[0083] Step C4) uses the SegNet decoder as the BridgeNet decoder. The saliency information mined by DSAM is fed into the decoder and continuously upsampled until the original input image size is restored. During the restoration process, skip connections are introduced to compensate for the information lost during the process, resulting in a preliminarily segmented abdominal multi-organ image.

[0084] Step 3: Train the coarse segmentation stage network SegNet using the training data preprocessed in Step 1 to obtain the trained coarse segmentation stage network SegNet.

[0085] Specifically, the initial learning rate was set to 0.01, the Adam optimizer was used, and a step decay schedule was employed to decay the learning rate. The Dice loss and focal loss functions were used to optimize the network. The total number of training iterations was set to 300.

[0086] Step 4: Use the SegNet network trained in Step 3 to segment the preprocessed training set data from Step 1 to obtain the coarse segmentation result; perform a dot product operation between the coarse segmentation result and the preprocessed training set data from Step 1 to obtain the training data for the fine segmentation stage; use the processed fine segmentation stage training data to train the fine segmentation model BridgeNet constructed in Step 2 to obtain the trained BridgeNet. Specifically:

[0087] Step 4.1: Use the SegNet network trained in Step 3 to segment the training set data preprocessed in Step 1 to obtain the segmentation results of the coarse stage.

[0088] Step 4.2: Perform a dot product operation between the coarse segmentation results and the preprocessed training set from Step 1 to obtain the training data for the fine segmentation stage.

[0089] Step 4.3: Train the BridgeNet fine-segmentation model constructed in Step 2 using the processed training data from the fine-segmentation stage to obtain the trained BridgeNet. Set the initial learning rate to 0.01, employ the Adam optimizer, and use a step decay schedule to reduce the learning rate. Optimize the network using Dice loss and focal loss functions. Set the total number of training iterations to 300.

[0090] Step 5: Use the trained coarse segmentation models SegNet and BridgeNet to segment the preprocessed test set data from Step 1, obtaining segmented test set data; further apply connected component analysis to the segmented test set data to obtain accurate abdominal multi-organ segmentation results. Specifically:

[0091] Step 5.1: Use the trained coarse segmentation models SegNet and BridgeNet to segment the preprocessed test set data from Step 1 to obtain the segmented test set data; the segmentation process of BridgeNet here is the same as the segmentation process of Steps A) to C) above.

[0092] Step 5.2: Connected component analysis is further applied to the segmented test set data to obtain accurate abdominal multi-organ segmentation results. Figure 6 The segmentation results are shown. Row (a) shows the actual organ labeling, row (b) shows the abdominal multi-organ segmentation results obtained from the baseline coarse-to-fine frame, and row (c) shows the abdominal multi-organ segmentation results obtained by the method of this patent.

Claims

1. A method for abdominal multi-organ segmentation based on bridged encoder-decoder, characterized in that: Specifically comprising the following steps: Step 1, preprocessing the images in the training set and the test set divided in the FLARE2021 data set to obtain the preprocessed training set and test set images; The specific process of step 1 is: Step 1.1, input the FLARE2021 data set, and re-orient the training set and test set images divided in the data set to the target direction in sequence; Step 1.2, resample all images in the data set to a fixed size; Step 1.3, cut all images in the data set to an intensity of [-325, 325], and finally apply z-score normalization based on the average and standard deviation of the intensity value; Step 2, build a two-stage abdominal multi-organ segmentation network framework from coarse to fine, including a coarse segmentation model SegNet and a fine segmentation model BridgeNet; In step 2, the BridgeNet fine segmentation model is built in the following manner: a bridge module is embedded between the encoder and the decoder, and the bridge module includes a many-to-one fusion module MSFM and a double-stream adaptive module DSAM; The segmentation process of the BridgeNet fine segmentation model on the image is as follows: Step A), feature extraction is performed using the encoder of SegNet as the encoder of BridgeNet to obtain extracted features ; Step B), the features processed by the encoder Convolution-regularization-activation processing is performed to obtain processed features Then the processed features are sent to a many-to-one fusion module MSFM, which includes a multi-scale global sub-module MSG and a multi-dimensional local sub-module MDL. 、 、 Through the multi-scale global sub-module MSG, global information of three scales is extracted in parallel 、 、 The specific formula is as follows: （1） （2） （3） （4） （5） （6） wherein GAP is a global average pooling operation, and GMP is a global max pooling operation, ; Step C), further convolution-regularization ConvIN of the three parallel branch features extracted by the MSG submodule, then up-sampling Upsample to the same size and adding, then further ReLU activation, convolution-regularization-activation ConvINReLU processing of the added results, to obtain multi-scale features fused with three scale global features; Step D), further convolution-regularization ConvIN of the three parallel branch features extracted by the MDL submodule, then up-sampling Upsample to the same size and adding, then further ReLU activation, convolution-regularization-activation ConvINReLU processing of the added results, to obtain multi-dimensional features fused with three-dimensional local features; Step E), concatenate the multi-scale features obtained in step C) and the multi-dimensional features obtained in step D), then convolution-regularization ConvIN, to obtain fused multi-scale multi-dimensional features, Step F), adding an input feature to the multi-scale multi-dimensional features obtained in Step E) , to obtain final features ; Step G), dynamic processing features with double stream adaptive module (DSAM) , C represents the number of channels of the feature , D represents the depth of the feature , H represents the height of the feature , and W represents the width of the feature , to obtain the preliminary segmented abdominal multi-organ image; Step 3, training the coarse segmentation stage network SegNet using the preprocessed training data in step 1 to obtain the trained coarse segmentation stage network SegNet; Step 4, training the fine segmentation model BridgeNet constructed in step 2 based on the results of step 3 to obtain the trained BridgeNet; Step 5, using the trained coarse segmentation model SegNet in step 3 and the trained BridgeNet in step 4 to segment the test set data preprocessed in step 1 to obtain segmented test set data; applying connected component analysis to the segmented test set data to obtain accurate abdominal multi-organ segmentation results; The specific process of step G) is: Step G1), extracting features from a multi-to-one fusion module MSFM Split into two parts and Then fed into spatial attention flow and channel attention flow respectively to obtain attention weights of the features extracted by MSFM, two attention flows and Calculated by formula (7) and formula (8) as follows, and then, two attention flows and Merged and Channel Shuffle operation is adopted to enable cross-group information flow along the channel dimension: （7） （8） Wherein, σ represents a sigmoid activation function, and FC represents a fully connected layer; Step G2), two convolutional layers and softmax activation function are used to generate normalized attention weights for k convolutional kernels The formula is as follows: （9） wherein, is the output of the second convolutional layer, and τ is the temperature; Step G3), the linear combination of the k convolution kernels is performed by the obtained weight, and the double-stream adaptive module DSAM is used to mine the multi-scale global features and multi-dimensional local features extracted by the MSFM. （10） wherein, denotes the i-th convolution kernel , ; is a weighted attention scalar, denotes a convolution operation; Step G4), the decoder of the SegNet is used as the decoder Decoder of the BridgeNet, and the mined information obtained in step G3) is input into the Decoder for continuous upsampling until the original image size is restored, i.e. the abdominal multi-organ image preliminarily segmented is obtained.

Images