A medical image segmentation method and device
By combining convolution and Transformer's hybrid backbone network, using the multi-path self-attention mechanism and the detail feedforward layer, the problems of large-objective misdetection and insufficient precision of small-objectives in medical image segmentation are solved, and the problem of traditional convolutional networks is achieved. The precise segmentation of large-scale and small-scale targets improves the overall performance of medical image segmentation.
Patent Information
- Application Number
- CN202411005271.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-25
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2044-07-25
AI Technical Summary
Traditional convolutional neural networks have false detection problems in large-objective segmentation and insufficient accuracy in small-objective segmentation in multi-objective segmentation tasks in medical images, especially because the limitations of the convolutional layer limit their learning of remote space dependencies. , the ability to segment large and small goals is poor.
Using a hybrid backbone network, combining convolution and Transformer, the global and local features of the image are captured through the multi-path self-attention mechanism and the detail feedforward layer, improving the generalization ability and robustness of large-target representations, and ensuring large-scale The characteristics of small and medium-sized targets are accurately extracted under the premise of target representation ability.
It effectively alleviates the spatial positioning information ability of long-term dependencies, improves the modeling ability of objects of various scales, achieves accurate segmentation of large-scale and small-scale targets, and improves the overall performance of medical image segmentation.
Smart Images

Figure CN118552736B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to a medical image segmentation method and device. Background Art
[0002] In the field of medical image segmentation, traditional convolutional neural networks (such as U-Net and its derivative network nnUNet) are widely used in the segmentation and analysis tasks of various structured medical images (such as MRI, CT, etc.). nnUnet is the most popular convolutional neural network architecture for medical image segmentation. It is a pure convolutional network. The convolution operation usually contains precise spatial information and provides hierarchical concepts, but the inherent limitations of the convolutional layer limit its ability to learn long-range spatial dependencies, which may eventually lead to FP in the segmentation results of large targets such as skin, bones, or lungs in the multi-target segmentation task of medical images, mainly body surface positioning belts or mattresses on bed boards.
[0003] To address the above issues, the nnFormer network actively explored the combination of conv and transformer to give full play to their respective advantages. The use of continuous Transformer blocks enables the global dependencies captured by them to have two effects on multi-classification segmentation: for large targets, FP is reduced; but for small targets, the precise spatial information provided by the convolution is weakened, resulting in a weakened segmentation effect of the network structure for small targets, such as bronchi and blood vessels.
[0004] Therefore, the above networks have some limitations when dealing with complex scenes containing large and small objects, such as the problem of false detection in the segmentation of large objects and insufficient accuracy in the segmentation of small objects. Summary of the invention
[0005] In view of this, the present invention provides a medical image segmentation method, comprising:
[0006] A medical image is segmented using a neural network model, the neural network model comprising an encoding module, a decoding module, a skip connection layer, an expansion layer and a linear projection layer, the encoding module is used to perform multiple downsampling and feature extraction on the medical image to obtain multiple image features of different scales and dimensions, the skip connection layer is used to add the multiple image features to the decoding module respectively, the decoding module is used to perform multiple upsampling operations on the last image feature, and process it with the multiple image features input by the skip connection respectively, the expansion layer is used to perform an upsampling operation on the feature map generated by the decoding module to obtain a feature map with the same resolution as the medical image, and the linear projection layer is used to perform convolution on the feature map generated by the expansion layer to change the dimension of the feature map to obtain a segmentation result of the medical image;
[0007] The encoding module includes a multi-order encoding module, the first-order encoding module is used to extract features from medical images, and the remaining encoding modules of each order extract features from the image features extracted by the encoding module of the previous order, so as to obtain image features of each order with decreasing scales and increasing dimensions, and the remaining encoding modules of each order include a first multi-path conversion block layer, the first multi-path conversion block layer includes a first multi-path self-attention mechanism layer and a first detail feedforward layer, the first multi-path self-attention mechanism layer is used to capture features of different granularities in the downsampled image features, and the first detail feedforward layer is used to supplement specific detail features in the multi-granularity image features to the local features;
[0008] The decoding module includes multi-order decoding modules, and the last-order encoding module is used to upsample the image features extracted by the previous order. The remaining decoding modules all include a second multi-path conversion block layer, and the second multi-path conversion block layer includes a second multi-path self-attention mechanism layer and a second detail feedforward layer. The second multi-path self-attention mechanism layer is used to capture features of different granularities in the upsampled image features, and the second detail feedforward layer is used to supplement specific detail features in the multi-granularity image features into local features.
[0009] Optionally, before using the neural network model to segment the medical image, the method further includes:
[0010] Obtain statistical information and network parameter configuration information of medical images, wherein the statistical information includes pixel spacing in medical images, statistical means of all medical image foregrounds, statistical standard deviations of all medical image foregrounds, and average sizes of all medical image foregrounds, and the network parameter configuration information includes the number of medical images, block sizes of segmented images, step sizes of convolutions of various orders, sizes of convolution kernels of various orders, and maximum dimensions of feature representation.
[0011] Optionally, the first-order encoding module includes a coding convolution layer, which includes an input layer, a first convolution layer, an activation function and normalization layer, and a second convolution layer. The first convolution layer is used to perform a convolution operation on the medical image input by the input layer according to a convolution kernel size of a corresponding order and a convolution step size of a corresponding order. The activation function and normalization layer are used to perform a nonlinear transformation and normalization on the convolved image features. The second convolution layer performs a convolution operation on the normalized image features again using a convolution kernel of size 3 and a convolution step size of size 1.
[0012] Optionally, the remaining encoding modules of each order also include a downsampling layer, which includes an activation function layer, a normalization layer, and a downsampling convolution layer. The activation function layer is used to perform nonlinear transformation on the image features extracted by the encoding module of the previous order, the normalization layer is used to normalize the image features after the nonlinear transformation, and the downsampling convolution layer is used to perform a convolution operation on the normalized image features according to the convolution kernel size of the corresponding order and the convolution step size of the corresponding order.
[0013] Optionally, each path self-attention mechanism layer in the first multi-path self-attention mechanism layer and the second multi-path self-attention mechanism layer includes multiple attention heads, each attention head is responsible for different queries, keys, and values, and the attention components are calculated according to the linear transformation parameters of the queries, keys, and values. The first multi-path self-attention mechanism layer and the second multi-path self-attention mechanism layer perform attention calculations on the attention components, and concatenate and fuse the results of each attention calculation with linear transformation to obtain multi-granularity image features.
[0014] Optionally, the first detail feed-forward layer and the second detail feed-forward layer both include a fully connected layer and a detail specific layer, the fully connected layer is used to process global features, and then the local features are processed by the detail specific layer, and the results of the fully connected layer and the detail specific layer are added and then linearly processed to supplement the specific detail features in the multi-granularity image features into the local features.
[0015] Optionally, the last-order decoding module includes a first upsampling layer and a decoding convolution layer, the first upsampling layer is used to perform an upsampling operation on the image features extracted in the previous order, and the decoding convolution layer is used to perform a convolution operation on the upsampled image features. The remaining decoding modules also include a second upsampling layer, and the second upsampling layer is used to perform an upsampling operation on the image features extracted in the previous order.
[0016] Optionally, before concatenating and fusing the results of each attention calculation with a linear transformation, the second multi-path conversion block layer further includes:
[0017] The image features extracted at the same order as the decoding module in the encoding module and the upsampled image features are respectively calculated with attention components through linear transformation parameters through a skip connection layer, and then SSA calculation is performed using each component. The calculation results of each SSA are concatenated and fused with linear transformation to integrate the image features of the same order in the encoder and decoder.
[0018] Optionally, before performing a convolution operation on the upsampled image features, the decoding convolution layer in the last-order encoding module further includes:
[0019] The image features extracted at the same level as the decoding module in the encoding module are spliced with the upsampled image features through a skip connection layer, and the spliced image features are convolved to adjust the dimension of the image features.
[0020] Optionally, upsampling multiples of the first upsampling layer and the second upsampling layer are the same as downsampling multiples of a downsampling layer of the same order in the encoding module.
[0021] On the other hand, the present invention also provides a medical image segmentation device, comprising: a processor and a memory connected to the processor; wherein the memory stores instructions executable by the processor, and the instructions are executed by the processor so that the processor executes the above-mentioned medical image segmentation method.
[0022] The present invention uses a hybrid backbone network to perform organ segmentation on medical images, that is, convolution and transformer are used alternately, long-term dependencies are completely interwoven with features of different scales, and the generalization and robustness of large target representation are improved. In addition, the Shunted transformer block is used to replace the original Transformer block, and the multi-path self-attention mechanism SSA introduced therein has different attention granularities. When processing image features, the model can understand and express image content from global to local, from coarse-grained to fine-grained, and alleviate the ability of spatial positioning information weakened by long-term dependencies. Under the premise of ensuring the ability to represent large targets, it can also accurately extract the features of small and medium targets, and can simultaneously model objects of various scales. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0024] Figure 1 is a flow chart of a medical image segmentation method in an embodiment of the present invention;
[0025] Figure 2 It is a structural diagram of the first multi-path self-attention mechanism layer and the second multi-path self-attention mechanism layer in an embodiment of the present invention;
[0026] Figure 3 is a flowchart of operations of the first detail feed-forward layer and the second detail feed-forward layer in an embodiment of the present invention;
[0027] Figure 4 The flowchart of the jump layer connection layer performing the S operation in the embodiment of the present invention. DETAILED DESCRIPTION
[0028] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0029] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.
[0030] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, it can also be the internal connection of two components, it can be a wireless connection, or it can be a wired connection. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0031] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0032] The image segmentation method provided by the present invention has a wide range of applications in medical image processing. For example, in medical images such as CT images and MRI or natural images, the present invention can accurately identify and segment various organ structures, such as large-scale targets such as the heart, lungs, and bones. For small-scale structures, such as blood vessels and bronchi, the method can also achieve accurate positioning and segmentation. In addition, the present invention is also applicable to image segmentation tasks in various scenarios, such as segmentation of body surface positioning spheres or other small parts.
[0033] Therefore, if Figure 1As shown, an embodiment of the present invention provides a medical image segmentation method to segment organs in medical images using a nn-ShuntedFormer network structure. The method is executed by an electronic device such as a computer or a server, and specifically includes:
[0034] A neural network model is used to segment a medical image. The neural network model includes an encoding module 1, a decoding module 2, a skip connection layer 3, an expansion layer 4 and a linear projection layer 5. The encoding module 1 is used to perform multiple downsampling and feature extraction on the medical image to obtain multiple image features of different scales and dimensions. The skip connection layer 3 is used to add the multiple image features to the decoding module respectively. The decoding module 2 is used to perform multiple upsampling operations on the last image feature and process it with the multiple image features input by the skip connection respectively. The expansion layer 4 is used to perform an upsampling operation on the feature map generated by the decoding module to obtain a feature map with the same resolution as the medical image. The linear projection layer 5 is used to convolve the feature map generated by the expansion layer to change the dimension of the feature map to obtain the segmentation result of the medical image.
[0035] Among them, the multi-stage encoding modules of the encoding module 1 are stage0 module 6, stage1 module 7, stage2 module 8, stage3 module 9 and stage4 module 10. The stage0 (first-stage encoding module) module 6 is used to extract features of medical images and obtain image features. , the remaining encoding modules of each order perform feature extraction on the image features extracted by the encoding module of the previous order, and obtain image features of each order with decreasing scale and increasing dimension, which are respectively , , , . , , , , The sizes are ,in, , , They are The downsampling rate of the resolution in the H, W, and D directions, is the feature dimension. Generally, as the resolution decreases, the feature dimension doubles.
[0036] The remaining encoding modules of each stage (stage1 module 7, stage2 module 8, stage3 module 9 and stage4 module 10) all include a first multi-path transformation block layer Shunted Transformer Block1, and the first multi-path transformation block layer includes a first multi-path self-attention mechanism layer and a first detail feed-forward layer. The first multi-path self-attention mechanism layer is used to capture features of different granularities in the down-sampled image features, and the first detail feed-forward layer is used to supplement specific detail features in the multi-granularity image features into the local features.
[0037] The decoding module 2 includes multi-order decoding modules, and the last-order encoding module is used to upsample the image features extracted in the previous order. The remaining decoding modules all include a second multi-path transformation block layer Shunted TransformerBlock2, and the second multi-path transformation block layer includes a second multi-path self-attention mechanism layer and a second detail feedforward layer. The second multi-path self-attention mechanism layer is used to capture features of different granularities in the upsampled image features, and the second detail feedforward layer is used to supplement specific detail features in the multi-granularity image features into the local features.
[0038] like Figure 2 As shown in the figure, the structure of the first multi-path self-attention mechanism layer and the second multi-path self-attention mechanism layer is the same, both of which include a multi-path self-attention mechanism layer and a detail feed-forward layer. Taking stage 1 module 7 as an example, the image feature The input is sent to stage 1 module 7, where it is first normalized and processed through a multi-path self-attention mechanism layer to obtain multi-granularity image features, which are then combined with the image features at the beginning of the input. Perform normalization operation together, add specific detail features in the normalized multi-granularity image features to the local features, and mix them with the image features before the second normalization operation to finally output the image features. .
[0039] Specifically, the encoding module 1 extracts features of medical images such as CT images through the stage0 module 6 to obtain image features , use the remaining encoding modules of each stage (stage1 module 7, stage2 module 8, stage3 module 9 and stage4 module 10) to extract the image features extracted by the encoding module of the previous stage, and obtain the image features of each stage with decreasing scale and increasing dimension. , , , , Will enter the decoding module 2 for processing, and , , , The skip-layer connection features as part of the encoding module 1 are input into the corresponding decoding modules of each order in the decoding module 2, and are combined with the features that have been upsampled multiple times in the decoding module. After processing, the final predicted segmentation result is output through the expansion layer 4 and the linear projection layer 5. Since the first multi-path conversion block layer and the second multi-path self-attention mechanism layer both include a multi-path self-attention mechanism layer and a detail feed-forward layer, the multi-path self-attention mechanism layer includes multiple Self-Attentions, so it is used to capture features of different granularities in the downsampled image features, and the detail feed-forward layer is used to supplement specific detail features in the multi-granularity image features to the local features, so as to better capture and utilize subtle image details.
[0040] This embodiment uses a hybrid backbone network to perform organ segmentation on medical images, that is, convolution and transformer are used alternately, long-term dependencies are completely interwoven with features of different scales, and the generalization and robustness of large target representation are improved. In addition, the Shunted transformer block is used to replace the original Transformer block, and the multi-path self-attention mechanism SSA introduced in it has different attention granularities. When processing image features, the model can understand and express image content from global to local, from coarse-grained to fine-grained, and alleviate the ability of spatial positioning information weakened by long-term dependencies. Under the premise of ensuring the ability to represent large targets, it can also accurately extract the features of small and medium targets, and can simultaneously model objects of various scales.
[0041] In one embodiment, before segmenting the medical image using the neural network model, the method further includes:
[0042] Obtain statistical information and network parameter configuration information of medical images, wherein the statistical information includes pixel spacing in medical images, statistical means of all medical image foregrounds, statistical standard deviations of all medical image foregrounds, and average sizes of all medical image foregrounds, and the network parameter configuration information includes the number of medical images, block sizes of segmented images, sizes of convolution kernels of various orders, sizes of convolution steps of various orders, and maximum dimensions of feature representation.
[0043] Specifically, before segmenting the medical image, the medical image needs to be preprocessed in the nnUnet manner to obtain statistical information FingerPrint about the medical image and configuration information ExperimentPlans related to the network parameters, where the statistical information of the training data mainly includes: pixel spacing in the medical image, statistical mean mean of all medical image foregrounds, statistical standard deviation std of all medical image foregrounds, average size median_size of all medical image foregrounds, etc.; the configuration information related to the network parameters mainly includes: batch_size, the number of medical images, patch_size, the block size of the segmented image, conv_kerner_size, the stride_size of each order of convolution kernels, the maximum dimension max_num_features of the feature representation, etc.
[0044] In one embodiment, the first-order encoding module (stage0 module 6) includes an encoding convolution layer, which includes an input layer, a first convolution layer, an activation function and normalization layer, and a second convolution layer. The first convolution layer is used to perform a convolution operation on the medical image input by the input layer according to the convolution kernel size of the corresponding order and the convolution step size of the corresponding order. The activation function and normalization layer are used to perform nonlinear transformation and normalization on the convolved image features. The second convolution layer uses a convolution kernel of size 3 and a convolution step of size 1 to perform a convolution operation on the normalized image features again.
[0045] Specifically, the first-order encoding module (stage0 module 6) includes an encoding convolution layer, which includes an input layer, a first convolution layer, an activation function and normalization layer (GELU+Norm), and a second convolution layer. The convolution kernel size k and stride s of the first convolution layer are the values of the corresponding layers in the convolution kernel size conv_kerner_size and convolution step size stride_size of the segmented image obtained in the preprocessing, and the convolution kernel size and stride s of the second convolution layer are fixed values, which are 3 and 1, respectively.
[0046] In one embodiment, the remaining encoding modules of each order also include a downsampling layer, which includes an activation function layer, a normalization layer, and a downsampling convolution layer. The activation function layer is used to perform nonlinear transformation on the image features extracted by the encoding module of the previous order, the normalization layer is used to normalize the image features after the nonlinear transformation, and the downsampling convolution layer is used to perform a convolution operation on the normalized image features according to the convolution kernel size of the corresponding order and the convolution step size of the corresponding order.
[0047] like Figure 1As shown, stage1 module 7, stage2 module 8, stage3 module 9 and stage4 module 10 also include a downsampling layer, which includes an activation function layer, a normalization layer and a downsampling convolution layer, wherein the activation function used in the activation function layer Act is GELU, the normalization layer is LayerNorm, the convolution kernel size k and the stride s are the values of the corresponding layers in the convolution kernel size conv_kerner_size and the convolution stride_size of the segmented image obtained in the preprocessing, such as stride=1 / 2; taking stage1 module 7 as an example, the activation function layer converts the image features A nonlinear transformation is performed, and then the normalization layer is used to normalize the image features after the nonlinear transformation. Finally, the downsampling convolution layer is used to perform a convolution operation on the normalized image features according to the convolution kernel size of the corresponding order and the convolution step size of the corresponding order. The obtained image features will enter the first multipath conversion block layer for processing.
[0048] In one embodiment, each path self-attention mechanism layer in the first multi-path self-attention mechanism layer and the second multi-path self-attention mechanism layer (Multi Shunted Self-Attention) includes multiple attention heads, each attention head is responsible for different queries, keys, and values, and the attention components are calculated according to the linear transformation parameters of the queries, keys, and values. The first multi-path self-attention mechanism layer and the second multi-path self-attention mechanism layer calculate the attention components, and concatenate and fuse the calculation results of each attention with linear transformation to obtain multi-granularity image features.
[0049] Specifically, taking the first multi-path self-attention mechanism layer in the encoding module as an example, according to different tasks, the number of multi-path transformation block layers (Shunted Transformer Block) in each Stage and the number of Self-Attention heads in each Block can be adjusted. Optionally, the number of Self-Attention heads in each Block can be set to: 2, 4, 8, 16. In order to reduce the amount of calculation while capturing information of different granularities, in different attention heads of the same Self-Attention, K and V are downsampled to different sizes, and the calculation formulas for the three components of Q, K and V are as follows:
[0050]
[0051]
[0052] ,
[0053] in, Represents the input of the Shunted Transformer Block layer of the multipath transformation block. It is the Multi-scale Token Aggregation layer of the i-th head. is the downsampling rate, yes The local enhancement component of , and There are three components in the self-attention mechanism, representing Query, Key and Value respectively. , and It is the linear transformation parameter for calculating the three components in the i-th attention head.
[0054] Specifically, It is through the convolution kernel size conv_kernel_size = stride (the number of pixels moved each time) = Downsampling achieved by convolution; in different attention heads of the same Self-Attention are different, so K and V can capture multi-scale information in Self-Attention.
[0055] The first multi-path self-attention mechanism layer calculates the attention component SSA (Shunted Self-Attention) normalized calculation formula:
[0056] ,
[0057] in, represents the i-th attention head, Represents the dimension, is a positional encoding.
[0058] The calculation result of each attention SSA The calculation formula of MSSA (Multi-head Shunted Self-Attention) is obtained by splicing and fusing linear transformations to obtain multi-granularity image features:
[0059] ,
[0060] in, It is a splicing operation. Indicates the number of attention heads, is the fused linear transformation parameter.
[0061] Furthermore, the computational cost of the above MSSA (Multi-head Shunted Self-Attention) and MSA (Multi-head Self-Attention) (ignoring the calculation of softmax):
[0062] ,
[0063] ,
[0064] in: , , Indicates the size of the volume. Indicates the dimension of Token. Indicates the number of attention heads of the current MSSA; and express Downsampling rate (multi-scale), generally, assuming that When the downsampling rate is ,when When the downsampling rate is ; Of course, you can also set the downsampling rate of different heads according to other rules;
[0065] It can be seen that the computational cost of MSSA depends on the downsampling rate and ,when and As it gets bigger, K and V become smaller ( or ), more Tokens are aggregated, which can improve the ability to capture large targets and reduce the computational cost. On the contrary, when and When it becomes smaller, K and V are larger, fewer tokens are aggregated, more details are retained, and the computational cost is higher.
[0066] Therefore, the first multi-path self-attention mechanism layer and the second multi-path self-attention mechanism layer can capture multi-granularity (multi-scale) information of image features according to the downsampling rate. For the fine-grained group, a small number of tokens can be aggregated and more local information can be retained, while the coarse-grained group can aggregate a large number of tokens, which can capture large targets while reducing the amount of computation. The multi-granularity group jointly learns multi-granularity information, so that the model can effectively model multi-scale targets and better complete the multi-target segmentation task in clinical practical applications.
[0067] In one embodiment, the first detail feedforward layer and the second detail feedforward layer (Detail Specific Feed Forward layer) both include a fully connected layer and a detail specific layer. The fully connected layer is used to process global features, and then the local features are processed by the detail specific layer. The results of the fully connected layer and the detail specific layer are added and then linearly processed to supplement the specific detail features in the multi-granularity image features into the local features.
[0068] like Figure 3 As shown, this embodiment uses the Detail Specific FeedForward layer to replace the traditional Feed Forward Network implemented by the fully connected layer, the purpose of which is to supplement the local information by specifying the details. The specific operation process of the first detail feedforward layer and the second detail feedforward layer (Detail Specific FeedForward layer) is as follows:
[0069] ,
[0070] ,
[0071] in, The parameter is The fully connected layer, The parameter is Specific layers of detail, specific It can be achieved by Depth-wise convolution. is the activation function (i.e. Figure 3 ), represents the input of the first detail feed-forward layer and the second detail feed-forward layer, yes The output, are the outputs of the first detail feed-forward layer and the second detail feed-forward layer.
[0072] like Figure 1 As shown, the last-order decoding code module includes a first upsampling layer and a decoding convolution layer. The first upsampling layer is used to upsample the image features extracted in the previous order, and the decoding convolution layer is used to perform a convolution operation on the upsampled image features. The remaining decoding modules also include a second upsampling layer, and the second upsampling layer is used to upsample the image features extracted in the previous order.
[0073] Specifically, the first upsampling layer, the second upsampling layer and the extended layer include a normalization layer Norm and an upsampling operation layer Upsampling / Transposed conv to complete the upsampling operation. The upsampling multiple is consistent with the downsampling multiple of the corresponding layer in the encoding module. For example, the upsampling feature size of the extended layer is Both the decoding convolution layer and the linear projection layer can be implemented using 1*1*1 convolution to change the feature dimension, where the linear projection layer ultimately maps the feature dimension to the size of num_classes.
[0074] In the above embodiment, the upsampling multiples of the first upsampling layer and the second upsampling layer are the same as the downsampling multiples of the downsampling layer of the same order in the encoding module.
[0075] Features extracted by each order encoding module in the encoding module , , , , The sizes are ,in, , , They are The downsampling rate of the resolution in the H, W, and D directions, is the feature dimension. Generally, as the resolution decreases, the feature dimension doubles, for example The size is , The size is , The size is , The size is , The size is .
[0076] Since the upsampling multiples of the first upsampling layer and the second upsampling layer correspond to the downsampling multiples in the encoding module, the feature size of the second upsampling layer in the first-stage decoding module 11 in the decoding module is the same as the feature size of the stage3 module 9. The size is the same as , the feature size of the second upsampling layer in the second-stage decoding module 12 is the same as the feature size of the stage2 module 8 The size is the same as , the feature size of the second upsampling layer in the third-order decoding module 13 is the same as the feature size of the stage 1 module 7 The size is the same as , the feature size of the first upsampling layer in the fourth-order decoding module 14 is the same as the feature size of the stage0 module 6 The size is the same as .
[0077] The upsampling multiple of each decoding module is the same as the downsampling multiple in its corresponding encoding module, ensuring that the size of the feature map in the decoding process is consistent with the size of the feature map in the corresponding stage of the encoding process, so that the decoding module can effectively reconstruct a high-resolution image from the feature map received from the encoding module. The upsampling operation enables the feature map in the decoding process to gradually recover to the resolution of the original input image, thereby providing more accurate prediction and boundary information in image segmentation.
[0078] In one embodiment, before the second multi-path conversion block layer performs linear transformation on the results of each attention calculation by splicing and fusing, it also includes: using a skip connection layer to calculate the attention components of the image features extracted at the same order as the decoding module in the encoding module and the upsampled image features by linear transformation parameters, and then using each component to perform attention calculation, and splicing and fusing the linear transformation on the calculation results of each attention to integrate the image features of the same order in the encoder and decoder.
[0079] like Figure 1 and Figure 4 As shown, the skip connection layer 3 in the first-order decoding module 11, the second-order decoding module 12, and the third-order decoding module 13 will perform the S operation (Skip Attention). The Skip Attention of this embodiment replaces the traditional concat or sum. The calculation formula of the three components Q, K, and V of Self-Attention is:
[0080] ,
[0081] ,
[0082] Among them, LP represents the linear projection operation. It is through the convolution kernel size conv_kerner_size = stride (the number of pixels moved each time) = Downsampling achieved by convolution; in different attention heads of the same Self-Attention are different, so K and V can capture multi-scale information in Self-Attention. Represents the output of the Shunted Transformer Block in the multi-path transformation block layer in stagel of the encoding module. Represents the output of the upsampling layer corresponding to stagel in the decoding module, , and It is the attention component in the Skip Attention layer. The specific operation process of Skip Attention is:
[0083] ,
[0084] in, represents the i-th attention head, Represents the dimension, is a positional encoding.
[0085] The feed forward layer is relatively simple. It is a two-layer fully connected layer. The activation function of the first layer is Relu, and the second layer does not use an activation function. The corresponding formula is as follows:
[0086] ,
[0087] ,
[0088] in, The parameter is The fully connected layer; is the activation function; Represents the input of the feedforward layer; yes Output: is the output of the feed-forward layer.
[0089] The structure of the skip connection layer is the same as that of the first / second multiplexed block layer. The only difference is that the first / second multiplexed block layer has one input (the feature map after downsampling), and then uses this input to calculate the Query, Key and Value, and then perform subsequent MSSA calculations, while the skip connection layer has two inputs (one is the feature map output by the encoding module). Figure 1 , one is the feature after upsampling in the decoding module Figure 2 , Features Figure 2 Used to calculate Query components, features Figure 1 Used to calculate the Key and Value components, and then perform the required MSSA calculation. , The results of the sum and MSSA are normalized and sent to the feedforward layer to obtain the final output, which is used as the input of the second multiplexing block layer in the decoder).
[0090] By introducing the Self-Attention mechanism in the skip connection layer, multi-scale feature capture and optimization are achieved in the encoding and decoding stages. This approach not only improves the model's ability to understand data, but also effectively improves the model's segmentation performance.
[0091] In one embodiment, the decoding convolution layer in the last-order encoding module, before performing a convolution operation on the upsampled image features, further includes:
[0092] The upsampled image features extracted at the same order as the decoding module in the encoding module are spliced through a skip connection layer, and the spliced image features are convolved to adjust the dimension of the image features.
[0093] like Figure 1 As shown in FIG. 1 , the skip connection layer 3 in the last-order encoding module (fourth-order decoding module 14) will perform the C operation, which adopts the traditional skip-layer connection concat operation, and then adjusts the dimension through 1*1*1 convolution. The concat operation can combine the same-order feature maps in the encoding module 1 and the decoding module 2 to form a new feature map. The features of the encoding module 1 can make up for the missing detail information in the decoding module 2. In this way, the expressiveness of the features can be further improved.
[0094] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0095] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0096] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1A function specified in one or more boxes.
[0097] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0098] Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the invention.
Claims
1. A medical image segmentation method, characterized in that: include: A medical image is segmented using a neural network model, the neural network model comprising an encoding module, a decoding module, a skip connection layer, an expansion layer and a linear projection layer, the encoding module is used to perform multiple downsampling and feature extraction on the medical image to obtain multiple image features of different scales and dimensions, the skip connection layer is used to add the multiple image features to the decoding module respectively, the decoding module is used to perform multiple upsampling operations on the last image feature, and process it with the multiple image features input by the skip connection respectively, the expansion layer is used to perform an upsampling operation on the feature map generated by the decoding module to obtain a feature map with the same resolution as the medical image, and the linear projection layer is used to perform convolution on the feature map generated by the expansion layer to change the dimension of the feature map to obtain a segmentation result of the medical image; The encoding module includes a multi-order encoding module, the first-order encoding module is used to extract features from medical images, and the remaining encoding modules of each order extract features from the image features extracted by the encoding module of the previous order, so as to obtain image features of each order with decreasing scales and increasing dimensions, and the remaining encoding modules of each order include a first multi-path conversion block layer, the first multi-path conversion block layer includes a first multi-path self-attention mechanism layer and a first detail feedforward layer, the first multi-path self-attention mechanism layer is used to capture features of different granularities in the downsampled image features, and the first detail feedforward layer is used to supplement specific detail features in the multi-granularity image features to the local features; The decoding module includes a multi-order decoding module, the last-order encoding module is used to perform an upsampling operation on the image features extracted by the previous order, and the remaining decoding modules each include a second multi-path conversion block layer, the second multi-path conversion block layer includes a second multi-path self-attention mechanism layer and a second detail feed-forward layer, the second multi-path self-attention mechanism layer is used to capture features of different granularities in the upsampled image features, and the second detail feed-forward layer is used to supplement specific detail features in the multi-granularity image features to the local features; Each path self-attention mechanism layer in the first multi-path self-attention mechanism layer and the second multi-path self-attention mechanism layer includes a plurality of attention heads, each of which is responsible for a different query, key, and value, and calculates the attention component according to the linear transformation parameters of the query, key, and value. The first multi-path self-attention mechanism layer and the second multi-path self-attention mechanism layer perform attention calculation on the attention component, and perform splicing and fusion linear transformation on the results of each attention calculation to obtain a multi-granularity image feature; Before the second multi-path conversion block layer performs linear transformation on the results of each attention calculation by splicing and fusing, the image features extracted at the same order as the decoding module in the encoding module and the upsampled image features are respectively calculated using linear transformation parameters through a skip connection layer, and then the attention calculation is performed using each component, and the calculation results of each attention are spliced and fused with linear transformation to integrate the image features of the same order in the encoder and decoder.
2. The method according to claim 1, characterized in that Before using the neural network model to segment medical images, it also includes: Obtain statistical information and network parameter configuration information of medical images, wherein the statistical information includes pixel spacing in medical images, statistical means of all medical image foregrounds, statistical standard deviations of all medical image foregrounds, and average sizes of all medical image foregrounds, and the network parameter configuration information includes the number of medical images, block sizes of segmented images, step sizes of convolutions of various orders, sizes of convolution kernels of various orders, and maximum dimensions of feature representation.
3. The method according to claim 2, characterized in that The first-order encoding module includes an encoding convolution layer, which includes an input layer, a first convolution layer, an activation function and normalization layer, and a second convolution layer. The first convolution layer is used to perform a convolution operation on the medical image input by the input layer according to the convolution kernel size of the corresponding order and the convolution step size of the corresponding order. The activation function and normalization layer are used to perform nonlinear transformation and normalization on the convolved image features. The second convolution layer uses a convolution kernel of size 3 and a convolution step of size 1 to perform a convolution operation on the normalized image features again.
4. The method according to claim 2, characterized in that: The remaining encoding modules of each order also include a downsampling layer, which includes an activation function layer, a normalization layer, and a downsampling convolution layer. The activation function layer is used to perform nonlinear transformation on the image features extracted by the encoding module of the previous order, the normalization layer is used to normalize the image features after nonlinear transformation, and the downsampling convolution layer is used to perform convolution operation on the normalized image features according to the convolution kernel size of the corresponding order and the convolution step size of the corresponding order.
5. The method according to claim 1, characterized in that The first detail feed-forward layer and the second detail feed-forward layer both include a fully connected layer and a detail specific layer. The fully connected layer is used to process global features, and then the detail specific layer is used to process local features. The results of the fully connected layer and the detail specific layer are added and then linearly processed to supplement the specific detail features in the multi-granularity image features into the local features.
6. The method according to claim 1, characterized in that The last-order decoding module includes a first upsampling layer and a decoding convolution layer. The first upsampling layer is used to perform an upsampling operation on the image features extracted in the previous order, and the decoding convolution layer is used to perform a convolution operation on the upsampled image features. The remaining decoding modules also include a second upsampling layer, and the second upsampling layer is used to perform an upsampling operation on the image features extracted in the previous order.
7. The method according to claim 6, characterized in that The decoding convolution layer in the last-order encoding module, before performing a convolution operation on the upsampled image features, further includes: The image features extracted at the same level as the decoding module in the encoding module are spliced with the upsampled image features through a skip connection layer, and the spliced image features are convolved to adjust the dimension of the image features.
8. The method according to claim 6, characterized in that The upsampling multiples of the first upsampling layer and the second upsampling layer are the same as the downsampling multiples of the downsampling layer of the same order in the encoding module.
9. A medical image segmentation device, characterized in that: include: A processor and a memory connected to the processor; wherein the memory stores instructions executable by the processor, and the instructions are executed by the processor so that the processor executes the medical image segmentation method as described in any one of claims 1-8.
