Medical image segmentation method and device, equipment and storage medium
By introducing a U-shaped network architecture with sparse skip connections and global attention mechanism into the medical image segmentation model, the problems of high computational cost and high complexity of existing models are solved, and efficient and accurate segmentation is achieved on small-scale datasets.
Patent Information
- Application Number
- CN202411128008.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-16
- Publication Date
- 2026-03-03
AI Technical Summary
Existing medical image segmentation models are computationally intensive and highly complex, limiting their effectiveness on small datasets.
By employing sparse skip connections between the efficient encoder layer and the enhanced decoder layer, and combining multi-scale feature extraction and global attention mechanisms, a U-shaped network architecture is constructed to compensate for the spatial information loss caused by encoder downsampling, and the upsampling process of the decoder is enhanced through a Transformer network.
While reducing computational load, it improves the accuracy and precision of medical image segmentation, overcomes the shortcomings of CNN networks in global feature extraction, and is suitable for medical image segmentation tasks with small datasets.
Smart Images
Figure CN121600247A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and in particular to a medical image segmentation method, apparatus, device, and storage medium. Background Technology
[0002] Medical image segmentation plays a crucial role in computer-aided monitoring and image-guided disease diagnosis, particularly in lesion examination and disease diagnosis. Convolutional Neural Networks (CNNs) are widely used in computer vision due to their excellent feature extraction capabilities. Existing medical image segmentation networks are mostly based on UNets, which are U-shaped architectures consisting of encoders and decoders built upon CNNs. While UNets can be combined with techniques such as attention mechanisms to improve segmentation accuracy, the inherent locality of convolutional neural networks limits their understanding of the global context of the image, leading to oversegmentation and undersegmentation problems in medical image segmentation.
[0003] To address the oversegmentation and undersegmentation issues of CNN networks, current approaches combine CNNs with other networks such as Transformers to compensate for CNNs' shortcomings in extracting global features. However, networks capable of extracting global features rely on pre-trained parameters on large-scale datasets, exhibiting poor performance on small datasets. They also suffer from high computational cost and model complexity, limiting their application in medical image segmentation tasks. Summary of the Invention
[0004] This invention provides a medical image segmentation method, apparatus, device, and storage medium to address the shortcomings of existing medical image segmentation models, such as high computational load and high model complexity.
[0005] This invention provides a medical image segmentation method, comprising: The medical image to be segmented is input into the high-efficiency encoder layer to extract the shallow features of the medical image and obtain a shallow feature image. Extract multi-scale images from the shallow feature images; Based on the sparse skip connections between the efficient encoder layer and the enhancement decoder layer, the multi-scale image is input into the enhancement decoder layer for segmentation processing to obtain the segmentation result of the medical image; In the sparse skip connection relationship, the Nth layer of the enhanced decoder layer is connected to the 1st to Nth layers of the high-efficiency encoder layer, respectively.
[0006] According to the medical image segmentation method provided by the present invention, the high-efficiency encoder further includes multiple parallel branches, wherein the multiple parallel branches have the same convolution kernel; the multi-scale image extraction of the shallow feature image includes: The shallow feature image is input into the parallel branches for convolution processing to obtain the convolutional feature image output by each branch in the plurality of parallel branches; wherein, in the plurality of parallel branches, the output of the i-th branch is the input of the (i+1)-th branch; By concatenating the convolutional feature images of each branch, a multi-receptive-field feature image is obtained; Global information is supplemented into the multi-receptive field feature image to obtain a multi-scale image.
[0007] According to the medical image segmentation method provided by the present invention, the step of supplementing the multi-receptive field feature image with global information to obtain a multi-scale image includes: Based on the learnable first transformation matrix, the position of each pixel in the convolutional feature image is globally represented to obtain the global attention image corresponding to the convolutional feature image; the number of channels of the multi-receptive field feature image is the same as the number of channels of the global attention image; The multi-receptive field feature image and the global attention image are concatenated and pooled to supplement global information in the multi-receptive feature image, resulting in a multi-scale image.
[0008] According to the medical image segmentation method provided by the present invention, in the sparse skip connection relationship, the input of the i-th layer in the enhancement decoder layer is obtained by upsampling from the (i+1)-th layer; the step of inputting the multi-scale image into the enhancement decoder layer for segmentation processing to obtain the segmentation result of the medical image includes: The multi-scale image is input into the enhancement decoder layer, and the output feature image of the i-th layer in the enhancement decoder layer is enhanced using the multi-scale image to obtain the enhanced feature image of the i-th layer of the enhancement decoder layer; The enhanced feature image and the multi-scale image are concatenated and convolved along the channel dimension to obtain the target feature image of the i-th layer; The target feature image is upsampled to segment the medical image, resulting in segmentation results with multi-level feature representations at different resolutions.
[0009] According to the medical image segmentation method provided by the present invention, the enhancement decoder layer includes a multi-head attention layer; the step of enhancing the output feature image of the i-th layer of the enhancement decoder layer using the multi-scale image to obtain the enhanced feature image of the i-th layer of the enhancement decoder layer includes: Based on the learnable second transformation matrix, the key value parameters are calculated using the multi-scale image, and the query parameters are calculated using the output feature image of the i-th layer in the enhanced decoder layer. The key-value parameters are sampled and compressed to obtain a parameter matrix with a preset resolution; The parameter matrix and the query parameters are input into the multi-head attention layer to capture the long-range dependencies of the multi-scale image from multiple angles, thereby obtaining a dependency image. Obtain the position encoding vector of the output feature image of the i-th layer in the enhancement decoder layer, and embed the position encoding vector into the dependency image to obtain the enhancement feature image of the i-th layer of the enhancement decoder layer.
[0010] According to the medical image segmentation method provided by the present invention, before inputting the medical image to be segmented into the high-efficiency encoder layer, the method further includes: Perform preprocessing operations on the medical images; The preprocessing operations include converting the data format of the medical image and adjusting the image size of the medical image.
[0011] According to the medical image segmentation method provided by the present invention, before inputting the medical image to be segmented into the high-efficiency encoder layer, the method further includes: Acquire initial medical images and perform batch preprocessing on the initial medical images to obtain sample medical images; A sample dataset is constructed based on the aforementioned medical images; The efficient encoder layer and the enhanced decoder layer are iteratively trained using the sample dataset.
[0012] The present invention also provides a medical image segmentation device, comprising the following modules: The first feature extraction module is used to input the medical image to be segmented into the high-efficiency encoder layer, extract the shallow features of the medical image, and obtain a shallow feature image; The second feature extraction module is used to extract multi-scale images of the shallow feature image; The image segmentation module is used to input the multi-scale image into the enhancement decoder layer for segmentation processing based on the sparse skip connections between the high-efficiency encoder layer and the enhancement decoder layer, so as to obtain the segmentation result of the medical image. In the sparse skip connection relationship, the Nth layer of the enhanced decoder layer is connected to the 1st to Nth layers of the high-efficiency encoder layer, respectively.
[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the medical image segmentation method as described above.
[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the medical image segmentation method as described above.
[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the medical image segmentation method as described above.
[0016] The medical image segmentation method, apparatus, device, and storage medium provided by this invention extract multi-scale features from the shallow feature image of the high-efficiency encoder layer. Based on the sparse skip connections between the high-efficiency encoder layer and the enhancement decoder layer, the multi-scale image is input into the enhancement decoder layer for segmentation processing to obtain the segmentation result of the medical image. By constructing a low-complexity U-shaped network architecture for medical image segmentation through the high-efficiency encoder layer, the enhancement decoder layer, and the sparse skip connections between them, a low-complexity U-shaped network architecture for medical image segmentation is constructed. This architecture can compensate for the spatial information loss caused by encoder downsampling, thereby compensating for the detail information that the enhancement decoder layer cannot recover during upsampling, improving feature utilization, and effectively reducing computational load while ensuring image segmentation accuracy and precision. Attached Figure Description
[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0018] Figure 1 This is a flowchart illustrating the medical image segmentation method provided by the present invention.
[0019] Figure 2 This is a schematic diagram of the network architecture for medical image segmentation provided by the present invention.
[0020] Figure 3 This is a schematic diagram of the feature fusion attention layer provided by the present invention.
[0021] Figure 4 This is a schematic diagram of the sparse skip connection relationship provided by the present invention.
[0022] Figure 5This is a schematic diagram of the network structure of a key component of the enhanced decoder layer provided by the present invention.
[0023] Figure 6 This is a schematic diagram of the structure of the medical image segmentation device provided by the present invention.
[0024] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation
[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0026] To address some of the problems existing in medical image segmentation models, this invention provides a medical image segmentation method. It constructs a U-shaped network architecture for medical image segmentation using an efficient encoder layer, an enhanced decoder, and sparse skip connections between the efficient encoder layer and the enhanced decoder layer. The efficient encoder layer extracts target features using convolution and self-attention, and maximizes gradient combinations through gradient backpropagation path analysis to learn the feature representation of the input image. The sparse skip connections pass features from the same and previous encoder layers to the decoder, improving feature utilization and enhancing the decoder's upsampling process. The enhanced decoder layer uses a Transformer network to iteratively enhance the feature map used for result prediction, increasing the information representation of the resolution feature map and mitigating the semantic gap between the encoder and decoder. This not only compensates for the deficiency of convolutional networks in capturing the global dependencies of pixels but also solves the problem of high computational cost caused by introducing Transformer networks into dense prediction tasks, achieving the goal of accurate medical image segmentation.
[0027] Specifically, Figure 1 This is a flowchart illustrating the medical image segmentation method provided by the present invention, as shown below. Figure 1 As shown, the method includes the following steps: Step 100: Input the medical image to be segmented into the high-efficiency encoder layer, extract the shallow features of the medical image, and obtain the shallow feature image; Step 200: Extract multi-scale images of the shallow feature images; Step 300: Based on the sparse skip connections between the high-efficiency encoder layer and the enhancement decoder layer, the multi-scale image is input into the enhancement decoder layer for segmentation processing to obtain the segmentation result of the medical image; In the sparse skip connection relationship, the Nth layer of the enhanced decoder layer is connected to the 1st to Nth layers of the efficient encoder layer respectively (N is a positive integer greater than 1).
[0028] The medical image to be segmented is input into a high-efficiency encoder layer to extract shallow features from the medical image, resulting in a corresponding shallow feature image. Optionally, the medical image to be segmented can be obtained by preprocessing an initial medical image.
[0029] Optionally, the efficient encoder layer may include one or more layers, each of which may be a convolutional layer consisting of convolution, batch normalization (BN), and an activation function, wherein the activation function may be the ReLU function. The medical image to be segmented may be a pre-processed image of a specific size. It is then fed into an efficient encoder layer, and passes through a convolutional layer consisting of convolution, batch normalization, and activation functions. Shallow feature extraction is performed to obtain a shallow feature image. Shallow feature extraction can be expressed by the following formula: (1)
[0030] in, This is a shallow feature image output by the efficient encoder layer. For shallow feature images, The width of the shallow feature image, This represents the number of channels in the shallow feature image.
[0031] Furthermore, multi-scale images are extracted from shallow feature images, including but not limited to multi-scale feature extraction of shallow feature images to obtain feature images of different scales, i.e., multi-scale images.
[0032] Based on the sparse skip connections between the efficient encoder layer and the enhancement decoder layer, the extracted multi-scale images are input into the enhancement decoder layer for segmentation processing to obtain the segmentation results of the medical images.
[0033] The high-efficiency encoder layer and the enhancement decoder layer have a corresponding relationship. This correspondence can be one-to-one. For example, the number of high-efficiency encoder layers and enhancement decoder layers is the same, and one layer in the high-efficiency encoder layer corresponds to one layer in the enhancement decoder layer.
[0034] In the sparse skip connections between the efficient encoder layer and the enhancement decoder layer, the Nth layer of the enhancement decoder layer is connected to the 1st to Nth layers of the efficient encoder layer, respectively. These sparse skip connections connect the efficient encoder layer and the enhancement decoder layer, compensating for the spatial information loss caused by encoder downsampling, thereby mitigating the shortcomings of convolutional networks in global feature extraction.
[0035] In this embodiment, multi-scale feature extraction is performed on the shallow feature image of the efficient encoder layer. Based on the sparse skip connections between the efficient encoder layer and the enhancement decoder layer, the multi-scale image is input into the enhancement decoder layer for segmentation processing to obtain the segmentation result of the medical image. By constructing a low-complexity U-shaped network architecture for medical image segmentation through the efficient encoder layer, the enhancement decoder layer, and the sparse skip connections between them, a low-complexity U-shaped network architecture for medical image segmentation is constructed. This architecture can compensate for the spatial information loss caused by encoder downsampling, thereby compensating for the detail information that the enhancement decoder layer cannot recover during upsampling, improving feature utilization, and effectively reducing computational load while ensuring image segmentation accuracy and precision.
[0036] In one embodiment, the efficient encoder layer includes multiple parallel branches with the same convolutional kernel. Multi-scale feature extraction of the shallow feature image is based on a global attention mechanism. Step 200, extracting the multi-scale image of the shallow feature image, includes: Step 210: Input the shallow feature image into the parallel branches for convolution processing to obtain the convolution feature image output by each branch in the plurality of parallel branches; wherein, in the plurality of parallel branches, the output of the i-th branch is the input of the (i+1)-th branch; Step 220: Concatenate the convolutional feature images of each branch to obtain a multi-receptive field feature image; Step 230: Global information is supplemented into the multi-receptive field feature image to obtain a multi-scale image.
[0037] The shallow feature image is input into multiple parallel branches of the high-efficiency encoder layer and convolved to obtain the convolutional feature image output by each branch. In the parallel branches, the output of the i-th branch is the input of the (i+1)-th branch, that is, the output of the current branch is the input of the next branch.
[0038] The convolutional feature images output from each branch are stitched together to obtain a multi-receptive-field feature image. Global information is then added to the multi-receptive-field feature image to obtain a multi-scale image.
[0039] Furthermore, in step 230, global information supplementation is performed on the multi-receptive field feature image to obtain a multi-scale image, which may also include: Step 231: Based on the learnable first transformation matrix, the position of each pixel in the convolutional feature image is globally represented to obtain the global attention image corresponding to the convolutional feature image; the number of channels of the multi-receptive field feature image is the same as the number of channels of the global attention image; Step 232: The multi-receptive field feature image and the global attention image are concatenated and pooled to supplement the global information of the multi-receptive feature image and obtain a multi-scale image.
[0040] Based on the learnable first transformation matrix, the position of each pixel in the convolutional feature image is globally represented to obtain a global attention image of the convolutional feature image. The number of channels in the multi-receptive field feature image is the same as the number of channels in the global attention image. Based on this global attention image, a multi-scale image of the shallow feature image is obtained. Specifically, the multi-receptive field feature image and the global attention image are added to obtain the corresponding stitched image, and the stitched image is further pooled to downsample the stitched image to obtain the multi-scale image.
[0041] In one embodiment, the parallel branches of the efficient encoder layer are feature integration layers, which can be used to establish global context dependencies and extract multi-scale features, and can fuse global attention features with multi-scale features. Here, context dependencies are represented by globally location-shared vectors, as shown in Equation 2 below: (2)
[0042] In Formula 2, It is the i-th pixel in the h×w plane of x. yes The corresponding global representation, It is the number of pixels in the h×w plane. and is a learnable transformation matrix.
[0043] In the multi-scale feature extraction stage, shallow feature images are... x Given parallel branches with the same kernel size, where the input to the current branch is the output of the previous branch, the calculation formula for parallel branches is as follows, taking a kernel size of 3 as an example: (3)
[0044] in, It is the current branch road The output of the previous branch is used. The outputs of each branch are then concatenated to obtain a multi-receptive-field feature image. Then, unify the multi-receptive field feature images. Channel number and global attention image Consistent, finally multi-receptive field feature images and global attention images Addition as the result of feature extraction Pooling operations are used to obtain multi-scale feature representations at different resolutions, i.e., multi-scale images. .
[0045] In some embodiments, in the sparse skip connection relationship between the efficient encoder and the enhancement decoder, the input of the i-th layer in the enhancement decoder is obtained by upsampling from the (i+1)-th layer. In step 300, based on the sparse skip connection relationship between the efficient encoder and the enhancement decoder, the multi-scale image is input into the enhancement decoder for segmentation processing to obtain the segmentation result of the medical image, including: Step 310: Input the multi-scale image into the enhancement decoder layer, and use the multi-scale image to enhance the output feature image of the i-th layer in the enhancement decoder layer to obtain the enhanced feature image of the i-th layer of the enhancement decoder layer; Step 320: The enhanced feature image and the multi-scale image are concatenated and convolved along the channel dimension to obtain the target feature image of the i-th layer; Step 330: Upsample the target feature image to segment the medical image and obtain segmentation results with multi-level feature representations at different resolutions.
[0046] A multi-scale image is input into the enhancement decoder, which enhances the output feature image of the i-th layer, resulting in the enhanced feature image of the i-th layer. The enhanced feature image and the multi-scale image are concatenated and convolved along the channel dimension to obtain the target feature image of the i-th layer. The target feature image is then upsampled to segment the medical image, yielding segmentation results with multi-level feature representations at different resolutions.
[0047] Furthermore, the enhancement decoder layer includes a multi-head attention layer. In step 320, the output feature image of the i-th layer in the enhancement decoder is enhanced using a multi-scale image to obtain the enhanced feature image of the i-th layer of the enhancement decoder layer, specifically including: Step 321: Based on the learnable second transformation matrix, calculate the key value parameters using the multi-scale image, and calculate the query parameters using the output feature image of the i-th layer in the enhancement decoder layer; Step 322: Sample and compress the key-value parameters to obtain a parameter matrix with a preset resolution; Step 323: Input the parameter matrix and the query parameters into the multi-head attention layer to capture the long-range dependencies of the multi-scale image from multiple angles and obtain a dependency image; Step 324: Obtain the position encoding vector of the output feature image of the i-th layer in the enhancement decoder layer, and embed the position encoding vector into the dependency image to obtain the enhancement feature image of the i-th layer of the enhancement decoder layer.
[0048] Based on the learnable second transformation matrix, key-value parameters are calculated using multi-scale images. These key-value parameters include key parameters (Key) and value parameters (Value). Optionally, the key parameters form a key-value matrix, and the value parameters form a value-value matrix. Query parameters are calculated using the output feature image of the i-th layer in the enhancement decoder. The key-value parameters are then sampled and compressed to obtain a parameter matrix at a preset resolution. This parameter matrix includes a key-value matrix (Key) and a value-value matrix (Value). Optionally, the key parameters in the key-value matrix are sampled and compressed to obtain a compressed key matrix (Key), and the value parameters in the value-value matrix are sampled and compressed to obtain a compressed value-value matrix (Value).
[0049] Furthermore, the parameter matrix and query parameters are input into the multi-head attention layer to capture long-range dependencies of multi-scale images from multiple angles, resulting in a dependency image. The position encoding vector of the output feature image of the i-th layer of the enhancement decoder is then obtained and embedded into the dependency image to supplement its spatial location information, thus obtaining the enhanced feature image of the i-th layer of the enhancement decoder. Here, the position encoding vector is obtained by encoding the pixel positions in the multi-scale image.
[0050] In medical image segmentation, sparse skip connections output multi-scale images. and the enhanced feature map of the i-th layer of the enhanced decoder As input to the enhancement decoder, it captures the long-range relationship between pixels in two feature images.
[0051] Input multi-scale image The key-value parameters, including key and value parameters, are calculated using a learnable transformation matrix to enhance the feature map. The query parameter Query is calculated using a learnable transformation matrix. Then, the key and value parameters are sampled and compressed to obtain a parameter matrix of fixed size (e.g., 8×8). This parameter matrix includes a key parameter matrix and a value parameter matrix. The query parameter Query, the key parameter matrix, and the value parameter matrix are then input into a multi-head attention layer to capture long-range dependencies across multiple scales of the image from multiple angles, resulting in a dependency image, which is then incorporated into a matrix. As a relative positional encoding, it supplements the positional information in the image space of the dependency relationship, resulting in an enhanced feature image. Specifically, it is expressed as follows: (4)
[0052] Where Q is the query parameter Query. The key parameter matrix is key. Let 'value' be the value parameter matrix and 'd' be the dimension of the multi-head attention layer. As Enhanced new feature representations, with multi-scale images The concatenation is performed along the channel dimension and then convolved to obtain the output feature map of the i-th layer of the decoder, as shown in Equation 5 below: (5)
[0053] Repeat the above steps and obtain multi-level feature representations with different resolutions through upsampling, which is the segmentation result of the medical image. The learnable second transformation matrix can be the same as the learnable first transformation matrix.
[0054] In one embodiment, the medical image to be segmented needs to be preprocessed before being input into the efficient encoder layer, and the efficient encoder and the enhancement decoder need to be pre-trained.
[0055] Specifically, for the preprocessing operations of medical images, prior to step 100, the following steps are also included: Step 101: Perform preprocessing operations on the medical image; The preprocessing operations include converting the data format of the medical image and adjusting the image size of the medical image.
[0056] Preprocessing operations are performed on the medical images, including converting the data format of the medical images and adjusting the image size of the medical images to a fixed size image as the input image for the efficient encoder layer.
[0057] Furthermore, for the pre-training of the efficient encoder layer and the enhanced decoder layer, before step 100, the following may also be included: Step 110: Obtain initial medical images and perform batch preprocessing on the initial medical images to obtain sample medical images; Step 120: Construct a sample dataset based on the sample medical images; Step 130: Iteratively train the efficient encoder layer and the enhanced decoder layer using the sample dataset.
[0058] Initial medical images are acquired and batch preprocessed to obtain sample medical images. The batch preprocessing of the initial medical images is the same as the preprocessing of the medical images to be segmented described above, and will not be repeated here. A sample dataset is constructed based on the preprocessed sample medical images, and the efficient encoder and augmented decoder layers are iteratively trained using this sample dataset to obtain pre-trained efficient encoder and augmented decoder layers.
[0059] In some embodiments, an efficient encoder layer, an enhanced decoder layer, and sparse skip connections construct a U-shaped network architecture for medical image segmentation, such as... Figure 2 As shown. The following is combined with... Figures 2 to 5 The medical image segmentation method provided in the embodiments of the present invention will be described.
[0060] exist Figure 2 In this process, the Nth layer of the enhancement decoder layer is connected to the 1st to Nth layers of the high-efficiency encoder layer, respectively, to compensate for the spatial information lost by downsampling in the high-efficiency encoder layer, that is, the detailed information that cannot be recovered by upsampling in the enhancement decoder layer.
[0061] Furthermore, for the encoding stage of the high-efficiency encoder layer, such as Figure 2 As shown, the efficient encoder layer includes a global context modeling layer (Conv GM_block), a multi-feature fusion attention layer, and a max-pooling layer. The global context modeling layer captures the global dependencies of the input image, i.e., global context features, enabling a better understanding of complex structures and details in the image. It then fuses these global context features into the original feature image, enhancing feature representation, and shares the global attention image across all locations. Specifically, the global context modeling layer is primarily used for context modeling, feature transformation, and feature fusion. Context modeling mainly forms context-related features of the input image; feature transformation captures channel orientation dependencies; and feature fusion integrates global context features into features at all locations.
[0062] Figure 2 The Conv FA-block is a fundamental component of the feature fusion attention layer, and its structure is as follows: Figure 3 As shown, the feature fusion attention layer can accurately model and process different channel and pixel-level features, effectively improving image restoration accuracy and visual effects. Figure 3 In this process, the input image x undergoes shallow feature extraction via convolution (conv), batch normalization (BN), and ReLU activation. The extracted shallow feature image is then fed into a feature fusion attention layer, where the FA-block is used to process the shallow features. Specifically, global contextual features are extracted from the shallow feature image, meaning the position of each pixel in the shallow feature image is globally represented to obtain the global attention image. The first stitched image is obtained by performing multi-layer convolution and stitching operations on the shallow feature image with parallel branches. Unified global attention image and the first stitched image The number of channels is counted and then stitched together to obtain the second stitched image. .
[0063] Furthermore, based on Figure 4 The sparse skip connections shown represent the multi-receptive field feature images output by different layers of the efficient encoder layer. After pooling, multi-layer feature representations at different resolutions are obtained, i.e., multi-scale images. .
[0064] In the multi-layer feature representation of the efficient encoder layer, low-resolution feature images focus on semantic information while high-resolution feature images focus on detailed information. Sparse skip connections break the limitation of only incorporating encoder features of the same level, improving feature utilization while implicitly alleviating the semantic gap between encoder and decoder.
[0065] For the decoding stage of the enhanced decoder layer, in Figure 2 In this process, the key component of the enhanced decoder layer is the RTransformer network, whose network structure is as follows: Figure 5 As shown. In Figure 5 In this study, based on a learnable second transformation matrix, multi-scale images are utilized. The key-value parameters are calculated, including a key-value matrix consisting of key parameters and a value-value matrix consisting of value parameters, using the output feature image of the i-th layer of the enhanced decoder layer. The query parameters (Query) are calculated to form a query parameter matrix. Then, the Key and Value are downsampled and compressed to obtain parameter matrices (key and value) of fixed resolution. This is done for the output feature image of the i-th layer of the augmented decoder. The location encoding vector is obtained through location encoding. Using methods such as location embedding, this location encoding vector is input into the multi-head attention layer, thereby embedding the location encoding vector into the dependency image. This supplements the spatial location information of the dependency image, resulting in an enhanced feature image. The enhanced decoder not only reduces the complexity of the RTransformer network, making it suitable for dense segmentation tasks, but also adaptively enhances encoder features at different stages, improving the prediction accuracy of image segmentation.
[0066] In this embodiment, a U-shaped network architecture is constructed using an efficient encoder layer, an enhanced decoder layer, and sparse skip connections. The encoder layer extracts image features, the decoder layer performs pixel-level segmentation, and sparse skip connections between the encoder and decoder compensate for spatial information loss caused by downsampling. During the encoding stage, global attention enriches local and global information. During the decoding stage, an RTransformer network and upsampling restore the spatial resolution of the image, reducing network complexity while achieving accurate segmentation of medical images. Furthermore, the spatial information compensation through sparse skip connections helps reduce the required computation while maintaining accurate segmentation, thus improving segmentation efficiency.
[0067] The medical image segmentation apparatus provided by the present invention is described below. The medical image segmentation apparatus described below can be referred to in correspondence with the medical image segmentation method described above.
[0068] Reference Figure 6 The medical image segmentation apparatus provided in this embodiment of the invention includes: The first feature extraction module 10 is used to input the medical image to be segmented into the high-efficiency encoder layer, extract the shallow features of the medical image, and obtain a shallow feature image. The second feature extraction module 20 is used to extract multi-scale images of the shallow feature image; The image segmentation module 30 is used to input the multi-scale image into the enhancement decoder layer for segmentation processing based on the sparse skip connection relationship between the high-efficiency encoder layer and the enhancement decoder layer, so as to obtain the segmentation result of the medical image. In the sparse skip connection relationship, the Nth layer of the enhanced decoder layer is connected to the 1st to Nth layers of the high-efficiency encoder layer, respectively.
[0069] In one embodiment, the high-efficiency encoder further includes multiple parallel branches, each having the same convolutional kernel; the second feature extraction module 20 is further configured to: The shallow feature image is input into the parallel branches for convolution processing to obtain the convolutional feature image output by each branch in the plurality of parallel branches; wherein, in the plurality of parallel branches, the output of the i-th branch is the input of the (i+1)-th branch; By concatenating the convolutional feature images of each branch, a multi-receptive-field feature image is obtained; Global information is supplemented into the multi-receptive field feature image to obtain a multi-scale image.
[0070] In one embodiment, the second feature extraction module 20 is further configured to: Based on the learnable first transformation matrix, the position of each pixel in the convolutional feature image is globally represented to obtain the global attention image corresponding to the convolutional feature image; the number of channels of the multi-receptive field feature image is the same as the number of channels of the global attention image; The multi-receptive field feature image and the global attention image are concatenated and pooled to supplement global information in the multi-receptive feature image, resulting in a multi-scale image.
[0071] In one embodiment, in the sparse skip connection relationship, the input of the i-th layer in the enhanced decoder layer is obtained by upsampling from the (i+1)-th layer; the image segmentation module 30 is further configured to: The multi-scale image is input into the enhancement decoder layer, and the output feature image of the i-th layer in the enhancement decoder layer is enhanced using the multi-scale image to obtain the enhanced feature image of the i-th layer; The enhanced feature image and the multi-scale image are concatenated and convolved along the channel dimension to obtain the target feature image of the i-th layer; The target feature image is upsampled to segment the medical image, resulting in segmentation results with multi-level feature representations at different resolutions.
[0072] In one embodiment, the enhanced decoder layer includes a multi-head attention layer; the image segmentation module 30 is further configured to: Based on the learnable second transformation matrix, the key value parameters are calculated using the multi-scale image, and the query parameters are calculated using the output feature image of the i-th layer in the enhanced decoder layer. The key-value parameters are sampled and compressed to obtain a parameter matrix with a preset resolution; The parameter matrix and the query parameters are input into the multi-head attention layer to capture the long-range dependencies of the multi-scale image from multiple angles, thereby obtaining a dependency image. Obtain the position encoding vector of the output feature image of the i-th layer in the enhancement decoder layer, and embed the position encoding vector into the dependency image to obtain the enhancement feature image of the i-th layer of the enhancement decoder layer.
[0073] In one embodiment, the medical image segmentation apparatus further includes a preprocessing module for: Perform preprocessing operations on the medical images; The preprocessing operations include converting the data format of the medical image and adjusting the image size of the medical image.
[0074] In one embodiment, the medical image segmentation apparatus further includes a pre-training module for: Acquire initial medical images and perform batch preprocessing on the initial medical images to obtain sample medical images; A sample dataset is constructed based on the aforementioned medical images; The efficient encoder layer and the enhanced decoder layer are iteratively trained using the sample dataset.
[0075] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7 As shown, the electronic device may include: a processor 710, a communications interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communications interface 720, and the memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute steps of a medical image segmentation method, such as: The medical image to be segmented is input into the high-efficiency encoder layer to extract the shallow features of the medical image and obtain a shallow feature image. Extract multi-scale images from the shallow feature images; Based on the sparse skip connections between the efficient encoder layer and the enhancement decoder layer, the multi-scale image is input into the enhancement decoder layer for segmentation processing to obtain the segmentation result of the medical image; In the sparse skip connection relationship, the Nth layer of the enhanced decoder layer is connected to the 1st to Nth layers of the high-efficiency encoder layer, respectively.
[0076] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0077] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to perform the steps of the medical image segmentation methods provided by the above methods, such as including: The medical image to be segmented is input into the high-efficiency encoder layer to extract the shallow features of the medical image and obtain a shallow feature image. Extract multi-scale images from the shallow feature images; Based on the sparse skip connections between the efficient encoder layer and the enhancement decoder layer, the multi-scale image is input into the enhancement decoder layer for segmentation processing to obtain the segmentation result of the medical image; In the sparse skip connection relationship, the Nth layer of the enhanced decoder layer is connected to the 1st to Nth layers of the high-efficiency encoder layer, respectively.
[0078] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the medical image segmentation methods provided by the above methods, including, for example: The medical image to be segmented is input into the high-efficiency encoder layer to extract the shallow features of the medical image and obtain a shallow feature image. Extract multi-scale images from the shallow feature images; Based on the sparse skip connections between the efficient encoder layer and the enhancement decoder layer, the multi-scale image is input into the enhancement decoder layer for segmentation processing to obtain the segmentation result of the medical image; In the sparse skip connection relationship, the Nth layer of the enhanced decoder layer is connected to the 1st to Nth layers of the high-efficiency encoder layer, respectively.
[0079] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0080] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A medical image segmentation method, characterized in that, include: The medical image to be segmented is input into the high-efficiency encoder layer to extract the shallow features of the medical image and obtain a shallow feature image. Extract multi-scale images from the shallow feature images; Based on the sparse skip connections between the efficient encoder layer and the enhancement decoder layer, the multi-scale image is input into the enhancement decoder layer for segmentation processing to obtain the segmentation result of the medical image; In the sparse skip connection relationship, the Nth layer of the enhanced decoder layer is connected to the 1st to Nth layers of the high-efficiency encoder layer, respectively.
2. The medical image segmentation method according to claim 1, characterized in that, The high-efficiency encoder further includes multiple parallel branches, each having the same convolutional kernel; the multi-scale image extraction of the shallow feature image includes: The shallow feature image is input into the parallel branches for convolution processing to obtain the convolutional feature image output by each branch in the plurality of parallel branches; wherein, in the plurality of parallel branches, the output of the i-th branch is the input of the (i+1)-th branch; By concatenating the convolutional feature images of each branch, a multi-receptive-field feature image is obtained; Global information is supplemented into the multi-receptive field feature image to obtain a multi-scale image.
3. The medical image segmentation method according to claim 2, characterized in that, The step of supplementing the multi-receptive-field feature image with global information to obtain a multi-scale image includes: Based on the learnable first transformation matrix, the position of each pixel in the convolutional feature image is globally represented to obtain the global attention image corresponding to the convolutional feature image; the number of channels of the multi-receptive field feature image is the same as the number of channels of the global attention image; The multi-receptive field feature image and the global attention image are concatenated and pooled to supplement global information in the multi-receptive feature image, resulting in a multi-scale image.
4. The medical image segmentation method according to claim 1, characterized in that, In the sparse skip connection relationship, the input of the i-th layer in the enhanced decoder layer is obtained by upsampling from the (i+1)-th layer; The step of inputting the multi-scale image into the enhancement decoder layer for segmentation processing to obtain the segmentation result of the medical image includes: The multi-scale image is input into the enhancement decoder layer, and the output feature image of the i-th layer in the enhancement decoder layer is enhanced using the multi-scale image to obtain the enhanced feature image of the i-th layer of the enhancement decoder layer; The enhanced feature image and the multi-scale image are concatenated and convolved along the channel dimension to obtain the target feature image of the i-th layer; The target feature image is upsampled to segment the medical image, resulting in segmentation results with multi-level feature representations at different resolutions.
5. The medical image segmentation method according to claim 4, characterized in that, The enhanced decoder layer includes a multi-head attention layer; the step of enhancing the output feature image of the i-th layer in the enhanced decoder layer using the multi-scale image to obtain the enhanced feature image of the i-th layer of the enhanced decoder layer includes: Based on the learnable second transformation matrix, the key value parameters are calculated using the multi-scale image, and the query parameters are calculated using the output feature image of the i-th layer in the enhanced decoder layer. The key-value parameters are sampled and compressed to obtain a parameter matrix with a preset resolution; The parameter matrix and the query parameters are input into the multi-head attention layer to capture the long-range dependencies of the multi-scale image from multiple angles, thereby obtaining a dependency image. Obtain the position encoding vector of the output feature image of the i-th layer in the enhancement decoder layer, and embed the position encoding vector into the dependency image to obtain the enhancement feature image of the i-th layer of the enhancement decoder layer.
6. The medical image segmentation method according to claim 1, characterized in that, Before inputting the medical image to be segmented into the high-efficiency encoder layer, the process further includes: Perform preprocessing operations on the medical images; The preprocessing operations include converting the data format of the medical image and adjusting the image size of the medical image.
7. The medical image segmentation method according to claim 1, characterized in that, Before inputting the medical image to be segmented into the high-efficiency encoder layer, the process further includes: Acquire initial medical images and perform batch preprocessing on the initial medical images to obtain sample medical images; A sample dataset is constructed based on the aforementioned medical images; The efficient encoder layer and the enhanced decoder layer are iteratively trained using the sample dataset.
8. A medical image segmentation device, characterized in that, include: The first feature extraction module is used to input the medical image to be segmented into the high-efficiency encoder layer, extract the shallow features of the medical image, and obtain a shallow feature image; The second feature extraction module is used to extract multi-scale images of the shallow feature image; The image segmentation module is used to input the multi-scale image into the enhancement decoder layer for segmentation processing based on the sparse skip connections between the high-efficiency encoder layer and the enhancement decoder layer, so as to obtain the segmentation result of the medical image. In the sparse skip connection relationship, the Nth layer of the enhanced decoder layer is connected to the 1st to Nth layers of the high-efficiency encoder layer, respectively.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the medical image segmentation method as described in any one of claims 1 to 7.
10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the medical image segmentation method as described in any one of claims 1 to 7.