A multi-organ medical image segmentation method and system based on improved TransUNet

Abstract

The application discloses a kind of based on the multi-organ medical image segmentation method and system of improved TransUNet, it uses the image segmentation model based on TransUNet construction and carries out segmentation to multi-organ medical image, comprising: S100: encoder extracts shallow feature map of different scales from input image, shallow feature map of different scales is screened to space gate, generate first fusion feature map;S200: bottleneck layer first divides Patch block into first fusion feature map and then is converted into one-dimensional data, then second fusion data is generated by gate learning;S300: decoder restores second fusion data into two-dimensional feature map, then the two-dimensional feature map and shallow feature map sequentially carry out multi-level interactive fusion, and generate class prediction image consistent with the size of the input image according to last level interactive fusion feature.The application can more accurately process complex medical targets with variable morphology and fuzzy boundaries, thereby improving the accuracy of multi-organ medical image segmentation.

Description

Technical Field

[0001] This application belongs to the field of medical image segmentation, specifically involving a multi-organ medical image segmentation method and system based on an improved TransUNet. Background Technology

[0002] At present, TransUNet and its derived network architectures are widely used in the field of image segmentation, but there are still many limitations in the application of multi-organ medical image segmentation: (1) Due to the high complexity of human anatomical structure, different organs and even the same organ in different individuals have huge differences in shape, size and texture, resulting in uncertainty of segmentation targets; (2) High-quality labeled data for multi-organ medical image segmentation heavily rely on manual delineation by professional physicians, which is costly and prone to subjective differences, resulting in a small amount of related image data; (3) TransUNet uses CNN to extract local features, but CNN cannot capture global context information and it is difficult to establish long-range dependencies between distant pixels in the image; although Transformer can capture long-range dependencies to make up for the defects of CNN, it will lose local details due to the attention mechanism.

[0003] In summary, although the current TransUNet network architecture performs well in multi-organ medical image segmentation applications, it has limitations in balancing long-range modeling, local perception, and computational efficiency, which restricts the improvement of the accuracy of multi-organ medical image segmentation. Summary of the Invention

[0004] To address or partially address the problems existing in related technologies, this application provides a multi-organ medical image segmentation method and system based on an improved TransUNet.

[0005] This application provides a multi-organ medical image segmentation method based on an improved TransUNet, which uses an image segmentation model built on TransUNet to segment multi-organ medical images, including:

[0006] S100: The encoder extracts shallow feature maps of different scales from the input image, performs spatial gating on the shallow feature maps of different scales, and generates the first fused feature map; wherein, the shallow feature maps of different scales are also input into the corresponding skip connection layers respectively.

[0007] S200: The bottleneck layer first divides the first fused feature map into patch blocks and then converts it into one-dimensional data, and then executes:

[0008] S210: Calculate the attention probability matrix between the preset category and the Patch block based on the one-dimensional data, and obtain the foreground probability and background probability;

[0009] S220: Divide the one-dimensional data into a first equal-divided data and a second equal-divided data. Perform attention learning on the first equal-divided data to obtain a first key-value pair. Perform gated memory learning on the first key-value pair to generate memory data. After residual processing, the memory data is fused with the activated second equal-divided data to generate the first fused data.

[0010] S230: Generate second fused data by multiplying the first fused data with the background probability by a dot product;

[0011] S300: The decoder restores the second fused data into a two-dimensional feature map, and then performs multi-level interactive fusion sequentially. Based on the last level of interactive fusion features, it generates a category prediction image with the same size as the input image.

[0012] The sequential multi-level interactive fusion is completed by a multi-level skip connection layer, including: in each level of interactive fusion, the decoder first upsamples the input feature map to obtain a deep feature map, and then the skip connection layer performs interactive fusion on the deep feature map and the corresponding scale shallow feature map to generate interactive fused features; wherein, in the first level of interactive fusion, the input feature map is the two-dimensional feature map; in other levels of interactive fusion, the input feature is the interactive fused feature of the previous level.

[0013] In some specific implementations, shallow feature maps of different scales are designated as the first feature map, second feature map, and third feature map in descending order of scale; spatial gating is performed on the shallow feature maps of different scales, including:

[0014] S110: Downsample the first feature map and the second feature map to have the same spatial size as the third feature map;

[0015] S120: The spliced ​​feature is obtained by concatenating the downsampling results of the first feature map and the second feature map;

[0016] S130: Map the feature values ​​of the spliced ​​features to the interval [0, 1] to generate spatial gating weights;

[0017] S140: Multiply the spatial gating weights element-wise with the third feature map, and then superimpose the third feature map to form a residual connection to obtain the first fused feature map.

[0018] In some specific embodiments, sub-step S210 further includes:

[0019] S211: Perform mapping learning on each patch block to obtain the index vector K corresponding to each patch block; at the same time, perform concatenated learning of feature mapping and category mapping on each patch block to obtain the corresponding category query vector Q; Q and K constitute the second key-value pair.

[0020] S212: Calculate the attention probability matrix between the preset category and the Patch block based on the second key-value pair;

[0021] S213: Perform a segmentation operation on the attention probability matrix to separate background features and foreground features, thereby constructing foreground probabilities and background probabilities respectively.

[0022] In some specific implementations, step S200 further includes: repeating sub-step S220;

[0023] During the first execution of sub-step S220, the one-dimensional data is processed;

[0024] In each subsequent execution of sub-step S220, the first fused data generated in the previous execution is processed;

[0025] And, in sub-step S230, the first fusion data generated by the last execution of sub-step S220 is multiplied by the background probability to generate second fusion data.

[0026] In some specific embodiments, sub-step S220, which involves gating memory learning of the first key-value pair and generating memory data, includes:

[0027] The first key-value pair is learned using the mLSTM module to construct a gating system, thereby obtaining the hidden state. The hidden state is then used as memory data and output.

[0028] And when constructing the gating system, the foreground probability is used as the m-update input gate;

[0029] The foreground probability is used as the m-update input gate. The formula is as follows:

[0030] ;

[0031] in, A protection constant to prevent numerical overflow; is the input gate at time t.

[0032] In some specific implementations, a skip connection layer interactively fuses the deep feature map and the corresponding shallow feature map, including:

[0033] S310: Obtain reference key features and global reference key features from deep feature maps, and obtain query features and global query features from shallow feature maps;

[0034] S320: Calculate the cross-correlation feature map based on the global reference key feature and the global query feature;

[0035] S330: Extract local and global features from the cross-correlation feature map, concatenate the local and global features, and obtain the gating weights by learning the concatenated features;

[0036] S340: The reference key features are weighted and corrected using gating weights, then concatenated with the query features, and finally reconstructed and fused features are generated through feature reconstruction.

[0037] S350: Obtain basic residual features from the deep feature map, and generate interactive fusion features by weighted fusion of the basic residual features and the reconstructed fusion features.

[0038] This application also provides a multi-organ medical image segmentation system based on an improved TransUNet, including:

[0039] The first module is configured to extract shallow feature maps of different scales from the input image using an encoder, perform spatial gating on the shallow feature maps of different scales, and generate a first fused feature map; wherein the shallow feature maps of different scales are also input into the corresponding skip connection layers respectively.

[0040] The second module is configured to use a bottleneck layer to first divide the first fused feature map into patch blocks and then convert it into one-dimensional data. Then, it performs the following: calculating the attention probability matrix between a preset category and the patch blocks based on the one-dimensional data, and obtaining foreground and background probabilities; dividing the one-dimensional data into first equal-division data and second equal-division data; performing attention learning on the first equal-division data to obtain first key-value pairs; performing gated memory learning on the first key-value pairs to generate memory data; fusing the memory data with the activated second equal-division data after residual processing to generate first fused data; and multiplying the first fused data with the background probabilities to generate second fused data.

[0041] The third module is configured to use a decoder to restore the second fused data into a two-dimensional feature map, and then perform multi-level interactive fusion sequentially, and generate a category prediction image with the same size as the input image based on the last level of interactive fusion features.

[0042] The sequential multi-level interactive fusion is completed by a multi-level skip connection layer, including: in each level of interactive fusion, the decoder first upsamples the input feature map to obtain a deep feature map, and then the skip connection layer performs interactive fusion on the deep feature map and the corresponding scale shallow feature map to generate interactive fused features; wherein, in the first level of interactive fusion, the input feature map is the two-dimensional feature map; in other levels of interactive fusion, the input feature is the interactive fused feature of the previous level.

[0043] In some specific embodiments, the first module further includes:

[0044] The ResNetV2 module consists of three progressively layered convolutional neural networks, which are configured to extract three shallow feature maps of decreasing scale sequentially from the input image.

[0045] The SpatialGate module is configured to perform spatial gating filtering on the three shallow feature maps.

[0046] In some specific embodiments, the second module further includes:

[0047] The Embedding layer is configured to divide the first fused feature map into patch blocks and then convert it into one-dimensional data.

[0048] The Positional layer is configured to add positional information to the one-dimensional data;

[0049] The Tra-VIL module includes a Tra module and multiple VIL modules. The Tra module forms the first branch, and the multiple VIL modules are connected sequentially to form the second branch. The output of the Positional layer is connected to the input of the first branch and the second branch. The outputs of the first branch and the second branch are multiplied together to generate the second fused data.

[0050] The Tra module is configured to calculate the attention probability matrix between a preset category and a Patch block based on the input one-dimensional data, and to obtain the foreground probability and background probability; wherein the background probability is the output of the first branch, and the foreground probability is input to each VIL module;

[0051] The VIL module is configured to divide the input one-dimensional data into a first equal-division data and a second equal-division data, perform attention learning on the first equal-division data to obtain a first key-value pair, use the input foreground probability to perform gated memory learning on the first key-value pair to generate memory data, and after residual processing, fuse the memory data with the activated second equal-division data to generate the first fused data.

[0052] In some specific implementations, the skip connection layer further includes:

[0053] The reference key feature extraction unit is configured to sequentially perform convolution, grouping normalization, and SiLU activation on the deep feature map to obtain reference key features.

[0054] The query feature extraction unit is configured to sequentially perform convolution, grouping normalization, and SiLU activation on the shallow feature map to obtain the query features.

[0055] The global reference key feature extraction unit is configured to sequentially perform convolution, grouping normalization, and ReLU activation on the deep feature map to obtain global reference key features.

[0056] The global query feature extraction unit is configured to sequentially perform convolution, grouping normalization, and ReLU activation on the shallow feature map to obtain the global query features.

[0057] The local and global feature extraction units are configured to calculate a cross-correlation feature map based on global reference key features and global query features, and then extract local and global features from the cross-correlation feature map using the CTX module and DTX module, respectively.

[0058] The gate weight learning unit is configured to learn the concatenated features of local and global features using the Gate_head module to obtain gate weights.

[0059] The fusion feature reconstruction unit is configured to perform weighted correction on the reference key features using gating weights, then concatenate them with the query features, and then use Reconstruct to reconstruct the features to generate reconstructed fusion features.

[0060] The interactive fusion unit is configured to perform convolution operations on the deep feature map to obtain basic residual features, and then weightedly fuse the basic residual features with the reconstructed fusion features to generate interactive fusion features.

[0061] Compared with the prior art, this application may include the following beneficial effects:

[0062] To address the shortcomings of the current TransUNet framework in simultaneously achieving long-range dependency modeling, local detail awareness, and computational efficiency, this application improves the TransUNet framework. The improved TransUNet can compensate for the deficiencies of NLP and CNN. Applying the improved TransUNet to multi-organ medical image segmentation can more accurately handle complex medical targets with varied shapes and blurred boundaries, thereby improving the accuracy of multi-organ medical image segmentation. Attached Figure Description

[0063] The above and other objects, features and advantages of this application will become more apparent from the following description of exemplary embodiments of this application in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of this application.

[0064] Figure 1 This is a flowchart illustrating the multi-organ medical image segmentation method of this application;

[0065] Figure 2 This is a schematic diagram of the framework of the image segmentation model in the embodiment;

[0066] Figure 3 This is a schematic diagram of the Tra-VIL module in the bottleneck layer of an embodiment.

[0067] Figure 4 This is a schematic diagram of the AIFI module in the encoder of the embodiment;

[0068] Figure 5 This is to simulate the visual effects of segmentation results from different models in the experiment. Detailed Implementation

[0069] The specific embodiments of this application will now be described in more detail with reference to the accompanying drawings. While specific embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be limited to the specific embodiments set forth herein. Rather, these specific embodiments are provided to make this application more thorough and complete, and to fully convey the scope of this application to those skilled in the art.

[0070] The multi-organ medical image segmentation method based on the improved TransUNet provided in this application includes: automatically segmenting multi-organ medical images using a trained image segmentation model; the image segmentation model is built based on TransUNet and includes an encoder, a bottleneck layer, a decoder, and a multi-level skip connection layer.

[0071] Please see Figure 1 The diagram shown is a flowchart of the multi-organ medical image segmentation method of this application. The steps of the multi-organ medical image segmentation method are as follows:

[0072] A TransUNet-based image segmentation model was used to segment multi-organ medical images, including:

[0073] S100: The encoder extracts shallow feature maps of different scales from the input image, performs spatial gating on the shallow feature maps of different scales, and generates the first fused feature map; wherein, the shallow feature maps of different scales are also input into the corresponding skip connection layers respectively.

[0074] S200: The bottleneck layer first divides the first fused feature map into patch blocks and then converts it into one-dimensional data, and then executes:

[0075] S210: Calculate the attention probability matrix between the preset category and the Patch block based on the one-dimensional data, and obtain the foreground probability and background probability;

[0076] S220: Divide the one-dimensional data into a first equal-divided data and a second equal-divided data. Perform attention learning on the first equal-divided data to obtain a first key-value pair. Perform gated memory learning on the first key-value pair to generate memory data. After residual processing, the memory data is fused with the activated second equal-divided data to generate the first fused data.

[0077] S230: Generate second fused data by multiplying the first fused data with the background probability by a dot product;

[0078] S300: The decoder restores the second fused data into a two-dimensional feature map, and then performs multi-level interactive fusion sequentially. Based on the last level of interactive fusion features, it generates a category prediction image with the same size as the input image.

[0079] The sequential multi-level interactive fusion is completed by a multi-level skip connection layer, including: in each level of interactive fusion, the decoder first upsamples the input feature map to obtain a deep feature map, and then the skip connection layer performs interactive fusion on the deep feature map and the corresponding scale shallow feature map to generate interactive fused features; wherein, in the first level of interactive fusion, the input feature map is the two-dimensional feature map; in other levels of interactive fusion, the input feature is the interactive fused feature of the previous level.

[0080] This embodiment provides a specific image segmentation model based on an improved TransUNet, which can be used for the aforementioned multi-organ medical image segmentation. The following will combine... Figures 2-4 The image segmentation model provided in this embodiment will be described in detail.

[0081] In this embodiment, the encoder includes a ResNetV2 module and a SpatialGate module; wherein, the ResNetV2 module includes a Root module and three progressive convolutional neural network layers Block1, Block2, and Block3, and the Root module, Block1, Block2, and Block3 are connected sequentially.

[0082] The Root module is a preprocessing module for ResNetV2, used to perform two-dimensional convolutions on the input image sequentially (e.g., ...). Group normalization (GroupNorm) and ReLU activation are used to expand the image channels and perform preliminary shallow feature extraction.

[0083] In the text This represents the convolution operation. Indicates the kernel size. Indicates step size, This indicates filling. Therefore, Indicates the kernel size as A convolution operation with a stride of 2 and padding of 3.

[0084] Convolution operations can change the dimensions of data; the relevant formula is as follows:

[0085] (1)

[0086] In formula (1): Indicates the dimension of the input data. Indicates the dimension of the output data.

[0087] The data dimension is reduced by convolution operation. Become This allows for changes in the size of the data dimension; and by designing the number of convolution kernels in the convolution operation, the number of output channels can be adjusted to achieve dimensionality increase and decrease operations of the channel dimension.

[0088] The output of the Root module is connected to three progressively stacked convolutional neural network layers, consisting of Block1, Block2, and Block3. Specifically, the output of the Root module is connected to Block1, and Block1, Block2, and Block3 are connected sequentially. Block1, Block2, and Block3 output feature maps I1, I2, and I3 with decreasing scales, respectively. Feature maps I1, I2, and I3 are shallow feature maps of different scales. Feature maps I1, I2, and I3 are all input to the SpatialGate module, which performs spatially gated feature fusion. Feature maps I1, I2, and I3 also serve as inputs to three skip connection layers.

[0089] Block 1, Block 2, and Block 3 are each composed of stacked units 1 and 2 of varying numbers. In this embodiment, Block 1 includes 1 unit 1 and 2 units 2, Block 2 includes 1 unit 1 and 3 units 2, and Block 3 includes 1 unit 1 and 8 units 2.

[0090] In the encoder, both Unit 1 and Unit 2 employ a residual bottleneck architecture, but they perform different network functions. Unit 1, as the Projection Block, is responsible for performing spatial downsampling or channel dimension alignment to ensure stable feature transfer and residual aggregation across stages. Unit 2, as the Identity Block, is primarily responsible for extracting, processing, and summarizing deep semantic information at the same scale. The structural frameworks of Unit 1 and Unit 2 are shown below. Figure 2Here, Mish represents the activation function, which is used to endow neural networks with the necessary nonlinear characteristics and expand their expressive power.

[0091] The feature maps I1, I2, and I3 output from convolutional neural network layers Block1, Block2, and Block3 are input into the SpatialGate module. The SpatialGate module aims to integrate and optimize feature maps I1, I2, and I3. The detailed process of integration and optimization is as follows:

[0092] S110: Downsample feature maps I1 and I2 to the same spatial size as Block3; specifically, downsample feature map I1 through convolution (Conv) and downsample feature map I2 through pooling (Pool).

[0093] S120: The two downsampling results are concatenated (Concat) to achieve preliminary fusion of shallow detail features. To further extract effective features and adjust the number of channels to match Block 3, convolution processing is also performed on the concatenated features.

[0094] S130: The sigmoid activation function is used to map the feature values ​​of the concatenated features to the interval [0, 1] to generate spatial gating weights;

[0095] S140: Multiply the spatial gating weights element-wise with the feature map I3, and then superimpose the original feature map I3 to form a residual connection, thereby outputting the spatially gating fused feature map Y, which is then fed into the bottleneck layer; whereby the fused feature map Y is also called the first fused feature map.

[0096] In this embodiment, the bottleneck layer further includes an Embedding layer, a Positional layer, and a Tra-VIL module; the Tra-VIL module is the core of the bottleneck layer, and its framework diagram is shown below. Figure 3 The Tra-VIL module is an integration of the Tra module and the VIL module, where Tra is the self-attention filtering module.

[0097] The aforementioned fused feature map Y, denoted as x, is the input to the bottleneck layer. First, the fused feature map Y is divided into patch blocks of uniform size and non-overlapping. Then, the embedding layer is used to reorganize the data of x, flattening the two-dimensional data x into one-dimensional data x. emb Where the size of the two-dimensional data x is denoted as (C, H, W), then the size of the one-dimensional data x... emb The size is (C, H*W).

[0098] The positional layer, or positional encoding embedding layer, is used to encode one-dimensional data x. embAdd shallow location information, specifically, to generate a unique location code for each pos position. The relevant mathematical formula is as follows:

[0099] (2)

[0100] In formula (2): This represents absolute position encoding, which is also sine / cosine position encoding; x represents adding shallow location information emb Its size is (C, H*W).

[0101] Absolute position encoding The calculation formula is as follows:

[0102] (3)

[0103] In equation (3): pos represents the position index. , represents the position encoding values ​​of the 2i-th and 2i+1-th dimensions at the pos-th position, respectively; 2i and 2i+1 represent the dimension grouping indices of the feature vector, where 2i represents an even dimension and 2i+1 represents an odd dimension; This represents the one-dimensional data x output by the Embedding layer. emb The length.

[0104] The Tra module is a trajectory-constrained self-attention filtering layer that abandons the traditional self-attention mechanism and instead introduces learnable organ semantic cues. The Tra module is used to process the input... The specific process for processing is as follows:

[0105] S211: Use Linear to perform mapping learning on each Patch block to obtain the index vector K corresponding to each Patch block; at the same time, perform cascade learning of feature mapping and category mapping on each Patch block to obtain the corresponding category query vector Q, and Q and K constitute the second key-value pair.

[0106] Further explanation is needed regarding the linear layer. K Mapping learning is performed on each patch block so that the index vector K has a size of (S, D), where S represents the sequence length and also the number of patches; D represents the mapping dimension. Simultaneously, a linear layer... Q The input patch is feature-mapped, and then passed through a class mapping layer and a linear layer again. S Generate the corresponding category query vector Q, with a dimension of (9, D), where 9 is the total number of categories, including one background category and eight target categories.

[0107] S212: Based on the query vector Q and index vector K, and using MatMul to calculate the raw attention scores between the category and the patch block, the attention probability matrix of the category is obtained by normalization using Softmax. Due to the setting of Q key-value pairs, The data has a dimension of (9, S) to represent the attention probability matrix for each category.

[0108] S213: The attention probability matrix is ​​segmented using Split. Indices 0 and 1-8 are taken from the attention probability matrix for segmentation. Since index 0 corresponds to the background category and 1-8 correspond to the eight target categories, the background features and foreground features are separated, and the foreground probability (fp) and background probability (bp) are constructed respectively. The tensor space dimensions of the background features and foreground features are (1,S) and (8,S) respectively. The foreground probability is obtained by summing the foreground features along the channel dimension.

[0109] The foreground probability represents the sum of thermal values ​​of all organ target features, while the background probability represents the sum of thermal values ​​of the background region. The output dimensions of both are unified as (1, S).

[0110] In the Tra module, abandoning the learning of V in key-value pairs can save computational overhead and facilitate the construction of lightweight semantic navigation.

[0111] See Figure 3 As shown, in the Tra module, Linear represents a linear layer, used to linearly weight and bias the input data, where the weight matrix and bias are learnable parameters; MatMul represents a matrix multiplication operation, used to calculate the original attention score through dot product; Split represents a splitting operation, used to split the input data.

[0112] The foreground probability fp output by the Tra module is used as the input gate i directly applied to the VIL module by m. t Its takeover formula is as follows:

[0113] (4)

[0114] In equation (4), m is the foreground probability fp output by the Tra module, which is used to introduce organ multi-scale spatial prior and inter-organ association information based on the global receptive field to dynamically modulate the activation degree of the input gate. A protection constant to prevent numerical overflow; is the input gate at time t.

[0115] data It also inputs the VIL module, which is used to learn the long-range dependencies of the data and the correlation between the segmentation targets, and finally outputs... ;Will The boundary is further sharpened by multiplying the background probability bp output by the Tra module. The result of the multiplication is expressed as follows: , For learnable parameters, This is the final output of the Tra-VIL module, which is also the output of the bottleneck layer.

[0116] In this embodiment, the VIL module is mainly composed of the Xlstm module, which is configured to perform five steps: data dimensionality enhancement, attention learning, gating memory, residual learning, and data dimensionality reduction.

[0117] The specific process of the VIL module is as follows:

[0118] S221: Using Up Projection The data is upgraded to higher dimensionality to obtain data B.

[0119] S222: Split the data B into two equal parts, B1 and B2. The two equal parts are used to retain the original data information and to perform gating learning, respectively. Data B1 and B2 are also called the first equal part data and the second equal part data. Data B1 and B2 are processed by the two branches respectively.

[0120] S223: In the first branch, attention learning is first performed on data B1 to obtain key-value pairs Q, K, and V, which are also called the first key-value pairs. The obtained key-value pairs and the foreground probability fp output by the Tra module are sent to the gated memory part (MatrixLSTM Cell, mLSTM) to obtain data B3, which is also called memory data.

[0121] S224: In the second branch, SiLU activation is performed on data B2; after residual processing, data B3 is multiplied element-wise with the activated data B2 to obtain data B4.

[0122] S225: Use Down Projection to reduce the dimensionality of data B4 to obtain the output data B5 of the VIL module. Data B5 is also called the first fused data.

[0123] In this embodiment, multiple VIL modules are integrated into the Tra-VIL module. These VIL modules are connected sequentially, with the output of one VIL module serving as the input of the next. Therefore, data B5 is also the input of the next VIL module. The output of the last VIL module is denoted as... , Generate by multiplying the background probability bp , This refers to the output of the bottleneck layer, also known as the second fused data.

[0124] It should be noted that the number of loops, i.e. the number of VIL modules, is determined by the user setting hyperparameters. In this embodiment, the number of loops is set to 10.

[0125] See Figure 3 As shown, LayNorm in the VIL module is a layer normalization operation, while Up Projection and Down Projection are the up projection operation and the down projection operation, respectively. Both of them modify the number of channels in the data. Up Projection expands the number of channels to enrich the data features and completes the data dimensionality increase operation, while Down Projection compresses the number of channels to reduce data redundancy and completes the data dimensionality decrease operation.

[0126] The VIL module employs an mLSTM module to implement gated memory. The mLSTM module is an improvement on the Long Short-Term Neural Network (LSTM), overcoming the vanishing gradient, exploding gradient, and single-threading problems inherent in LSTM. The mLSTM module receives key-value pairs Q, K, and V and learns their data features to construct the gating system, including the forget gate. Input gate Output gate Normalized state To obtain the hidden state And output it.

[0127] Key-value pairs Q, K, V, and input data at time t Due to the influence of gating memory, data is filtered by gating memory. Specifically, the input data at time t... This refers to the data at position t in the input data of the VIL module. The gating system controls the key-value pairs and inputs into the gating through random initialization and autonomous learning. Weighted processing is performed, where the input gate in the gating memory is used. Output gate Forgotten Gate The calculation formula is as follows:

[0128] (5)

[0129] In equation (5), , , These are the weight coefficients for the input gate, output gate, and forget gate, respectively. , , These are the bias coefficients for the input gate, output gate, and forget gate, respectively.

[0130] While gated memory can finely divide data and clarify causal relationships within the overall data, it lacks multidimensional parallel capabilities. Therefore, the mLSTM module, combined with parallel processing, handles cell states... Configure the matrix and modify the normalization state. and hidden state This allows for parallel global learning of the input data, while simultaneously learning stabilizing features based on the obtained data characteristics, ensuring that the results stabilize and do not explode or disappear.

[0131] The relevant mathematical formulas are as follows:

[0132] (6)

[0133] In equation (6), , , Let Q, K, and V represent the values ​​at time t, respectively. This indicates the candidate is in a hidden state.

[0134] This application also utilizes the macroscopic perspective of the Tra module to influence the input gate. Specifically, the foreground probability fp output by the Tra module is used as the input gate for m directly applied to the internal VIL module. The relevant mathematical formula is shown in equation (4), which is used to enhance the VIL module's ability to learn key information.

[0135] Specifically, firstly, the input gate is calculated using equation (5). Output gate Forgotten Gate Then, equation (4) is used to calculate the input gate. The updated input gate is then used in equation (6). Perform data feature learning.

[0136] In this embodiment, the decoder includes a progressive upsampling backbone and a segmentation head. The progressive upsampling backbone comprises three progressive upsampling modules, each of which includes an upsampling layer (BilinearUpsampling) and a convolutional layer. The system consists of a ReLU activation layer and three progressive upsampling modules that output feature maps Up3, Up2, and Up1, respectively. The Bilinear Upsampling layer represents bilinear upsampling, a common image scaling interpolation method. The Segmentation Head receives the high-dimensional, dense feature map reconstructed from the progressive upsampling backbone, and then translates it into a category prediction image with the same size as the input image through Conv and Upsampling before outputting it.

[0137] Due to the output of the bottleneck layer Since the data is one-dimensional, the decoder first uses the Reshape module to... The feature map is restored to a two-dimensional feature map, and then input into the progressive upsampling backbone for deep feature extraction.

[0138] In this embodiment, the skip connection layer uses the AIFI module to fuse the feature maps (I1, I2, I3) output by the three progressively layered convolutional neural network layers with the feature maps (Up3, Up2, Up1) output by the three progressively layered upsampling module. The feature maps (I1, I2, I3), as feature data from the shallow layers, are denoted as Skip. i For ease of description, Skip i This is denoted as the shallow feature map; the feature map (Up3, Up2, Up1) is denoted as Up. i For ease of description, Up i This is denoted as a deep feature map. i Its characteristics are strong semantics but blurred edges, Skip i The characteristic is that it has clear edges but weak semantics. The AIFI module is used to skip feature data. i Up i By interacting and integrating, deep semantic information and shallow detailed features are effectively combined, thereby enhancing the features.

[0139] This application includes a multi-level skip connection layer, which is used to sequentially interact and fuse deep feature maps and shallow feature maps of corresponding scales. The shallow feature maps are interacted and fused sequentially in ascending order of size. Specifically, in this embodiment, I3, I2, and I1 are interacted and fused with the deep feature maps in the order of I3, I2, and I1.

[0140] Please see Figure 4 The image shows the AIFI module, whose output serves as the input to the next upsampling module in the progressive upsampling backbone.

[0141] The specific workflow of the AIFI module will be described in detail below:

[0142] S310: From deep feature map Up iObtain the reference key feature K Skip and global reference key feature K g_Skip And, from shallow feature maps Skip i Obtain query feature Q up and global query feature Q g_up .

[0143] In this embodiment, Skip i Up i The reference key features K are obtained by performing convolution, grouping normalization, and SiLU activation sequentially. Skip Query feature Q up ; for Skip i Up i The global reference key feature K is obtained by performing convolution, group normalization, and ReLU activation sequentially. g_Skip Global query feature Q g_up .

[0144] S320: Based on global reference key feature K g_Skip and global query feature Q g_up Calculate the cross-correlation feature map inter. The formula for calculating inter is inter = Q. g_up* K g_Skip .

[0145] S330: Extract local features from the cross-correlation feature map inter. and global features splicing local features and global features Then, the gating weights are obtained by learning the splicing features. .

[0146] To process cross-correlation feature maps with different receptive fields, local and global features are extracted separately in this step. In this embodiment, the CTX module and DTX module are used to extract local features respectively. Global features ; local features Global features The splicing results in the splicing feature gate. conf Learn the splicing feature gate using the Gate_head module. conf Obtain the gating weight .

[0147] The CTX module is used for learning local information, while the DTX module is used for learning information over a larger scope. Together, they form a multi-scale learning field of view. The CTX module mainly consists of depthwise separable convolutions and activation functions. The depthwise convolutions are obtained by combining depthwise and pointwise methods. Depthwise convolutions are used to learn spatial features within and between channels; pointwise convolutions are used for cross-channel information interaction. The DTX module mainly consists of dilated depthwise separable convolutions and activation functions, introducing dilated convolutions on top of the CTX module. Dilated convolutions inject "holes" between convolution kernel elements, significantly expanding the receptive field without increasing additional parameters or computation. The Gate_head module is the gate weight head generation module, responsible for generating spatially adaptive gate weights and performing nonlinear transformations on the contextual features fused from CTX and DTX.

[0148] S340: Utilizing Gated Weights For reference key feature K Skip Perform weighted correction, then compare with query feature Q up The features are concatenated and then reconstructed to generate fused features; among them, gating weights are used. For reference key feature K Skip Weighted correction can suppress useless background features and enhance effective foreground features.

[0149] In this embodiment, the feature reconstruction module Reconstruct is used to reconstruct features. Reconstruct is used to perform convolution, grouping normalization and SiLu activation on the spliced ​​features in sequence, thereby smoothing the spliced ​​features and generating high-quality reconstructed fused features.

[0150] The weighted correction mathematical formula for the reference key features is as follows:

[0151] (7)

[0152] In equation (7), K ref The reference key features are after weighted correction; The lower limit of the gating weight is a fixed parameter used to control the gating weight, which is fixed at 0.2 in this embodiment.

[0153] S350: From deep feature maps Up i Obtain the base residual features, and then perform weighted fusion of the base residual features and the reconstructed fusion features to generate the interactive fusion feature F. out .

[0154] In this embodiment, by analyzing the deep feature map Up iSpatial upsampling followed by convolution operations yields the base residual features.

[0155] The formula for the weighted fusion above is as follows:

[0156] (8)

[0157] In equation (8), t represents the weight coefficient, which is obtained by generating a learnable parameter ft from the AIFI module and then mapping it to the [0,1] interval through Sigmoid activation.

[0158] In this embodiment, the input Up of the AIFI module i Skip i and output All dimensions are C*H*W.

[0159] In this application, each AIFI module generates and outputs interactive fusion features, and its output is denoted as Z. i Z i As input to the next-layer upsampling module. In this embodiment, the output of the last-stage AIFI module. The multi-organ segmentation probability map is predicted pixel by pixel by the segmentation head, thus obtaining the final category prediction image.

[0160] In the AIFI module, Skip i Up i Projecting onto the intermediate dimension yields the reference key feature K. Skip Query feature Q up and the corresponding gating K g_Skip Q g_up Generate cross-correlation feature maps based on gating to achieve cross-dimensional interactive attention; process cross-correlation feature maps with different receptive fields, including processing cross-correlation maps with different receptive fields through standard convolution and dilated convolution, and obtain gating weights; use gating weights to suppress useless background features and enhance effective foreground features.

[0161] Furthermore, to address the issue of small targets being easily lost in medical image segmentation, the AIFI module introduces a lower bound protection to ensure that some original details are preserved even when the gating is closed, preventing excessive blind spots. Finally, an asymmetric residual fusion logic is employed, which, by setting learnable parameters, achieves an adaptive balance between the original semantics of the file and the refined fusion features.

[0162] In application, the original medical image is input into the image segmentation model that has been trained and has its optimal weights saved. The model extracts multi-scale features through the encoder, performs global context modeling through the bottleneck layer, and fuses features step by step with the decoder and skip connections. Finally, it generates a multi-organ segmentation mask with the same spatial resolution as the input image at the output end, thus completing the accurate and automatic segmentation of multiple organs.

[0163] Obviously, the image segmentation model mentioned above needs to be trained in advance.

[0164] In the training of this embodiment, the loss function used is By Dice loss function Cross-entropy loss function and boundary loss function The loss function is composed of: Dice loss function, which is responsible for the macro-level segmentation quality; cross-entropy loss function, which is responsible for the micro-level classification accuracy; and boundary loss function, which uses the Laplacian Kernel to extract the boundary between the predicted probability map and the true label.

[0165] The relevant formulas for the loss function are as follows:

[0166] (9)

[0167] in, , These are the learnable parameters during model training. , , .

[0168] To further illustrate the advantages of this application, the following simulation experiments will be conducted based on the Synapse dataset and the ACDC dataset. The Synapse dataset contains images of eight abdominal organs from 30 scans, covering the left kidney, right kidney, aorta, spleen, gallbladder, liver, stomach, and pancreas, totaling 3779 axially enhanced abdominal clinical CT images. Each CT dataset consists of 85–198 512×512 pixel slices. The ACDC dataset is primarily used to evaluate and compare the performance of automated cardiac image analysis algorithms, including cardiac magnetic resonance imaging from 100 different patients. Each set of images includes three tissues of the heart: the right ventricle, myocardium, and left ventricle.

[0169] In this simulation experiment, "TransUNet: Transformers Make Strong Encoder for Medical Image Segmentation" (Catalogue: Chen J, Lu Y, Yu Q, et al. TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation[J]. 2021. DOI: 10.48550 / arXiv.2102.04306) was used as the benchmark comparison. The model of this application was compared with the benchmark comparison and several derivative models based on the TransUNet model. The Disse similarity coefficient (DSC), 95% Hausdorff distance (HD95), and mean intersection-union ratio (mIOU) were used as evaluation indicators.

[0170] The experimental results are shown in Tables 1 and 2. Table 1 shows the comparison results on the Synapse dataset. In the table, Aorta, GB, KL, KR, Liver, PC, SP, and SM represent eight types of organs, namely the aorta, gallbladder, left kidney, right kidney, liver, pancreas, spleen, and small intestine.

[0171] As shown in Table 1, compared with other baseline models, the evaluation metrics of the model in this application show a significant advantage, indicating that the model in this application has excellent overall segmentation accuracy; it also has excellent localization similarity and boundary fitting ability, and performs better for the complex and delicate task of contour prediction; and it has an extremely outstanding performance in the spatial overlap between the predicted region and the real label.

[0172] Table 1 Evaluation indicators for different models in the simulation experiment

[0173]

[0174] To further verify the generalization ability and robustness of the proposed model, cross-task evaluation was conducted on the ACDC dataset, using the Dice coefficient as the evaluation metric. The Dice coefficient values ​​for different models are listed in Table 2 below. In the table, RV, Myo, and LV represent the right ventricular cavity, left ventricular myocardium, and left ventricular cavity, respectively, and Avg Dice represents the average Dice coefficient for RV, Myo, and LV, also denoted as the average Dice coefficient. As shown in Table 2, for different organ morphological characteristics and data distributions, the average Dice coefficient of the proposed model is as high as 91.68%, demonstrating strong adaptability.

[0175] Table 2. Dice coefficient values ​​of different models in the simulation experiment.

[0176]

[0177] The model does not overfit to a single dataset and maintains high efficiency and excellent robustness across a variety of medical image segmentation tasks.

[0178] Furthermore, a visual comparison of the model in this application and the baseline model TransUNet was performed on the Synapse dataset, such as... Figure 5 As shown, Figures (a-1), (b-1), and (c-1) are three original multi-organ medical images; Figures (a-2), (b-2), and (c-2) are manually annotated ground truth segmentation maps; Figures (a-3), (b-3), and (c-3) are the segmentation results of the baseline model TransUNet; and Figures (a-4), (b-4), and (c-4) are the segmentation results of the model in this application.

[0179] The simulation experiments on the Synapse and ACDC datasets have demonstrated the superiority and strong generalization ability of the model in this application for complex medical image analysis tasks.

[0180] The main improvements made to the TransUNet network architecture in relation to the above embodiments include:

[0181] (1) For the encoder, in order to ensure the stable transition and output of focused information and enhance the integration capability of multi-scale information, a spatial gate attention module is introduced into the encoder to learn the influence between the key attention areas;

[0182] (2) For the bottleneck layer, the Tra-VIL module is used to replace the traditional Transformer module to construct the overall long-range information of the data. The aim is to retain the global information learning ability while strengthening the local detail perception and boundary characterization ability. The Tra-VIL module integrates the self-attention filtering module Tra and the long-range information capture module VIL. Through the macro-control of semantic mask and the memory gating learning in VIL, it achieves better long-range dependency and target organ correlation learning. During the research process, the inventors found that the single VIL module has a strong ability to learn the correlation of the whole organ, but there is a learning drift problem. Therefore, the parallel Tra module is further constructed to achieve macro-control, thereby strengthening the VIL's ability to characterize the boundary.

[0183] (3) For the skip connection layer, the skip connection layer adopts the AIFI module. The AIFI module achieves efficient coordination of deep abstract semantics and shallow spatial details through gating interaction and adaptive reconstruction mechanism between features, thereby accurately depicting the edge contour of complex medical targets and reducing positioning errors.

[0184] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A multi-organ medical image segmentation method based on an improved TransUNet, characterized in that: A TransUNet-based image segmentation model was used to segment multi-organ medical images, including: S100: The encoder extracts shallow feature maps of different scales from the input image, performs spatial gating on the shallow feature maps of different scales, and generates the first fused feature map; wherein, the shallow feature maps of different scales are also input into the corresponding skip connection layers respectively. S200: The bottleneck layer first divides the first fused feature map into patch blocks and then converts it into one-dimensional data, and then executes: S210: Calculate the attention probability matrix between the preset category and the Patch block based on the one-dimensional data, and obtain the foreground probability and background probability; S220: Divide the one-dimensional data into a first equal-divided data and a second equal-divided data. Perform attention learning on the first equal-divided data to obtain a first key-value pair. Perform gated memory learning on the first key-value pair to generate memory data. After residual processing, the memory data is fused with the activated second equal-divided data to generate the first fused data. S230: Generate second fused data by multiplying the first fused data with the background probability by a dot product; S300: The decoder restores the second fused data into a two-dimensional feature map, and then performs multi-level interactive fusion sequentially. Based on the last level of interactive fusion features, it generates a category prediction image with the same size as the input image. The sequential multi-level interactive fusion is completed by a multi-level skip connection layer, including: in each level of interactive fusion, the decoder first upsamples the input feature map to obtain a deep feature map, and then the skip connection layer performs interactive fusion on the deep feature map and the corresponding scale shallow feature map to generate interactive fused features; wherein, in the first level of interactive fusion, the input feature map is the two-dimensional feature map; in other levels of interactive fusion, the input feature is the interactive fused feature of the previous level.

2. The multi-organ medical image segmentation method as described in claim 1, characterized in that: The shallow feature maps of different scales are denoted as the first feature map, the second feature map, and the third feature map in descending order of scale. The spatial gating filtering of shallow feature maps at different scales includes: S110: Downsample the first feature map and the second feature map to have the same spatial size as the third feature map; S120: The spliced ​​feature is obtained by concatenating the downsampling results of the first feature map and the second feature map; S130: Map the feature values ​​of the spliced ​​features to the interval [0, 1] to generate spatial gating weights; S140: Multiply the spatial gating weights element-wise with the third feature map, and then superimpose the third feature map to form a residual connection to obtain the first fused feature map.

3. The multi-organ medical image segmentation method as described in claim 1, characterized in that: Sub-step S210 further includes: S211: Perform mapping learning on each patch block to obtain the index vector K corresponding to each patch block; at the same time, perform concatenated learning of feature mapping and category mapping on each patch block to obtain the corresponding category query vector Q; Q and K constitute the second key-value pair. S212: Calculate the attention probability matrix between the preset category and the Patch block based on the second key-value pair; S213: Perform a segmentation operation on the attention probability matrix to separate background features and foreground features, thereby constructing foreground probabilities and background probabilities respectively.

4. The multi-organ medical image segmentation method as described in claim 1, characterized in that: Step S200 also includes: repeating sub-step S220; During the first execution of sub-step S220, the one-dimensional data is processed; In each subsequent execution of sub-step S220, the first fused data generated in the previous execution is processed; And, in sub-step S230, the first fusion data generated by the last execution of sub-step S220 is multiplied by the background probability to generate second fusion data.

5. The multi-organ medical image segmentation method as described in claim 1, characterized in that: In sub-step S220, the gated memory learning of the first key-value pair and the generation of memory data includes: The first key-value pair is learned using the mLSTM module to construct a gating system, thereby obtaining the hidden state. The hidden state is then used as memory data and output. Furthermore, when constructing the gating system, the foreground probability is used as the m-update input gate. ; The foreground probability is used as the m-update input gate. The formula is as follows: ； in, A protection constant to prevent numerical overflow; is the input gate at time t.

6. The multi-organ medical image segmentation method as described in claim 1, characterized in that: The interactive fusion of the deep feature map and the corresponding shallow feature map at different scales by the skip connection layer includes: S310: Obtain reference key features and global reference key features from deep feature maps, and obtain query features and global query features from shallow feature maps; S320: Calculate the cross-correlation feature map based on the global reference key feature and the global query feature; S330: Extract local and global features from the cross-correlation feature map, concatenate the local and global features, and obtain the gating weights by learning the concatenated features; S340: The reference key features are weighted and corrected using gating weights, then concatenated with the query features, and finally reconstructed and fused features are generated through feature reconstruction. S350: Obtain basic residual features from the deep feature map, and generate interactive fusion features by weighted fusion of the basic residual features and the reconstructed fusion features.

7. A multi-organ medical image segmentation system based on an improved TransUNet, characterized in that, include: The first module is configured to extract shallow feature maps of different scales from the input image using an encoder, perform spatial gating on the shallow feature maps of different scales, and generate a first fused feature map; wherein the shallow feature maps of different scales are also input into the corresponding skip connection layers respectively. The second module is configured to use a bottleneck layer to first divide the first fused feature map into patch blocks and then convert it into one-dimensional data. Then, it performs the following: calculating the attention probability matrix between a preset category and the patch blocks based on the one-dimensional data, and obtaining foreground and background probabilities; dividing the one-dimensional data into first equal-division data and second equal-division data; performing attention learning on the first equal-division data to obtain first key-value pairs; performing gated memory learning on the first key-value pairs to generate memory data; fusing the memory data with the activated second equal-division data after residual processing to generate first fused data; and multiplying the first fused data with the background probabilities to generate second fused data. The third module is configured to use a decoder to restore the second fused data into a two-dimensional feature map, and then perform multi-level interactive fusion sequentially, and generate a category prediction image with the same size as the input image based on the last level of interactive fusion features. The sequential multi-level interactive fusion is completed by a multi-level skip connection layer, including: in each level of interactive fusion, the decoder first upsamples the input feature map to obtain a deep feature map, and then the skip connection layer performs interactive fusion on the deep feature map and the corresponding scale shallow feature map to generate interactive fused features; wherein, in the first level of interactive fusion, the input feature map is the two-dimensional feature map; in other levels of interactive fusion, the input feature is the interactive fused feature of the previous level.

8. The multi-organ medical image segmentation system as described in claim 7, characterized in that: The first module further includes: The ResNetV2 module consists of three progressively layered convolutional neural networks, which are configured to extract three shallow feature maps of decreasing scale sequentially from the input image. The SpatialGate module is configured to perform spatial gating filtering on the three shallow feature maps.

9. The multi-organ medical image segmentation system as described in claim 7, characterized in that: The second module further includes: The Embedding layer is configured to divide the first fused feature map into patch blocks and then convert it into one-dimensional data. The Positional layer is configured to add positional information to the one-dimensional data; The Tra-VIL module includes a Tra module and multiple VIL modules. The Tra module forms the first branch, and the multiple VIL modules are connected sequentially to form the second branch. The output of the Positional layer is connected to the input of the first branch and the second branch. The outputs of the first branch and the second branch are multiplied together to generate the second fused data. The Tra module is configured to calculate the attention probability matrix between a preset category and a Patch block based on the input one-dimensional data, and to obtain the foreground probability and background probability; wherein the background probability is the output of the first branch, and the foreground probability is input to each VIL module; The VIL module is configured to divide the input one-dimensional data into a first equal-division data and a second equal-division data, perform attention learning on the first equal-division data to obtain a first key-value pair, use the input foreground probability to assist the first key-value pair in gating memory learning and generate memory data, and after residual processing, the memory data is fused with the activated second equal-division data to generate the first fused data.

10. The multi-organ medical image segmentation system as described in claim 7, characterized in that: The skip connection layer further includes: The reference key feature extraction unit is configured to sequentially perform convolution, grouping normalization, and SiLU activation on the deep feature map to obtain reference key features. The query feature extraction unit is configured to sequentially perform convolution, grouping normalization, and SiLU activation on the shallow feature map to obtain the query features. The global reference key feature extraction unit is configured to sequentially perform convolution, grouping normalization, and ReLU activation on the deep feature map to obtain global reference key features. The global query feature extraction unit is configured to sequentially perform convolution, grouping normalization, and ReLU activation on the shallow feature map to obtain the global query features. The local and global feature extraction units are configured to calculate a cross-correlation feature map based on global reference key features and global query features, and then extract local and global features from the cross-correlation feature map using the CTX module and DTX module, respectively. The gate weight learning unit is configured to learn the concatenated features of local and global features using the Gate_head module to obtain gate weights. The fusion feature reconstruction unit is configured to perform weighted correction on the reference key features using gating weights, then concatenate them with the query features, and then use Reconstruct to reconstruct the features to generate reconstructed fusion features. The interactive fusion unit is configured to perform convolution operations on the deep feature map to obtain basic residual features, and then weightedly fuse the basic residual features with the reconstructed fusion features to generate interactive fusion features.

Images