A medical image segmentation method, device and medium

By introducing a multi-level feature extraction enhancement module into the medical image segmentation model, combining downsampling and upsampling network layers, the problem of low segmentation accuracy in the prior art is solved, and efficient recognition and refinement of complex medical images is achieved. Improved segmentation accuracy and recognition capabilities.

CN118552553BActive Publication Date: 2024-12-03GUANGZHOU UNIVERSITY
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
CN202410616680.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-05-17
Publication Date
2024-12-03
Estimated Expiration
2044-05-17

AI Technical Summary

Technical Problem

When faced with multiple complex segmentation tasks, the existing medical image segmentation model has low segmentation accuracy and cannot effectively capture the high-level semantic features and detailed information of the image, resulting in poor performance in multi-task scenarios.

Method used

Using a multi-level feature extraction enhancement module, combining downsampling and upsampling network layers, feature information is extracted and enhanced layer by layer through channel fusion, sparse attention calculation module and FFN network layer, to realize deep mining and refinement of images. Improve the model's ability to identify complex structures.

Benefits of technology

It improves the accuracy and recognition ability of medical image segmentation, can more effectively capture the global and local features of the image, adapt to feature information at different scales and levels, enhances the model's ability to recognize complex structures, and improves the accuracy of segmentation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118552553B_ABST
    Figure CN118552553B_ABST
Patent Text Reader

Abstract

The present application discloses a medical image segmentation method, device and medium, including: inputting an initial feature map into the first downsampling network layer, operating the output features of the first downsampling network layer as the input of the next downsampling network layer, obtaining the first enhanced feature map of each downsampling network layer and the output features of the ith downsampling network layer; fusing the output features of the ith downsampling network layer and the first enhanced feature map of the ith downsampling network layer into the first upsampling network layer, operating the output features of the 1st upsampling network layer and the first enhanced feature map of the ith-1st downsampling network layer as the input of the second feature extraction enhancement module in the next upsampling network layer, obtaining the output features of the ith upsampling network layer; inputting the output features of the ith upsampling network layer into a segmentation head, obtaining a segmented image corresponding to the initial feature map output by the segmentation head. The present application can improve the segmentation accuracy of medical images.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a medical image segmentation method, device and medium. Background Art

[0002] In the field of medical imaging, especially in brain tumor segmentation, the development of technology has made certain progress. The current medical image segmentation technology mainly relies on the use of deep convolutional neural networks (DCNNs) and Transformer architecture, or even the fusion of the two. These advanced models have made remarkable achievements in improving the accuracy of medical image segmentation.

[0003] However, since the existing segmentation models focus on a specific task, although they can perform well in the specific task, when faced with a variety of complex segmentation tasks, the segmentation accuracy of the existing segmentation models is still relatively low. Summary of the invention

[0004] The present application provides a medical image segmentation method, device and storage medium, which can improve the accuracy of medical image segmentation.

[0005] In a first aspect, the present application provides a medical image segmentation method, wherein the image segmentation model includes i downsampling network layers, i upsampling network layers and a segmentation head, wherein i is a positive natural number, the downsampling network layers each include a first feature extraction enhancement module and a downsampling module, and the upsampling network layers each include a second feature extraction enhancement module and an upsampling module, and the method includes:

[0006] Inputting the initial feature map into the first feature extraction enhancement module in the first downsampling network layer to obtain the first enhanced feature map of the first downsampling network layer;

[0007] Inputting the first enhanced feature map of the first downsampling network layer into the downsampling module in the first downsampling network layer to obtain the output feature of the first downsampling network layer;

[0008] The output features of the first downsampling network layer are used as the input of the next downsampling network layer and the operation is repeated until the first enhanced feature map of each downsampling network layer and the output features of the i-th downsampling network layer are obtained;

[0009] The output features of the i-th downsampling network layer and the first enhanced feature map of the i-th downsampling network layer are fused and input into the second feature extraction enhancement module in the first upsampling network layer to obtain the second enhanced feature map of the first upsampling network layer;

[0010] Inputting the second enhanced feature map of the first upsampling network layer into the upsampling module in the first upsampling network layer to obtain the output feature of the first upsampling network layer;

[0011] The output features of the first upsampling network layer and the first enhanced feature map of the i-1th downsampling network layer are used as the input of the second feature extraction enhancement module in the next upsampling network layer, and the operation is repeated until the output features of the i-th upsampling network layer are obtained;

[0012] The output features of the i-th upsampling network layer are input into the segmentation head to obtain the segmentation image corresponding to the initial feature map output by the segmentation head.

[0013] A further technical solution thereof is that the first feature extraction enhancement module and the second feature extraction enhancement module are the same, both including a channel fuser, a sparse attention calculation module and an FFN network layer;

[0014] Inputting the initial feature map into the first feature extraction enhancement module in the first downsampling network layer to obtain the first enhanced feature map of the first downsampling network layer, including:

[0015] Input the initial feature map into the channel fusion device in the first downsampling network layer to obtain the fusion result output by the channel fusion device;

[0016] The fusion result is input into the sparse attention calculation module for self-attention calculation, and the target global feature map output by the sparse attention calculation module is obtained;

[0017] Input the target global feature map into the FFN network layer to obtain the fusion information output by the FFN network layer;

[0018] Based on the fused information, the first enhanced feature map of the first downsampling network layer is obtained.

[0019] A further technical solution is that the channel fuser includes a first convolutional layer, a first depth convolutional layer and a second depth convolutional layer, wherein the convolution kernels of the first depth convolutional layer and the second depth convolutional layer have different sizes;

[0020] The initial feature map is input into the channel fusion device in the first downsampling network layer to obtain the fusion result output by the channel fusion device, including:

[0021] Input the initial feature map into the first convolutional layer to obtain the local features of the initial feature map;

[0022] Inputting the local features into the first depth convolution layer and the second depth convolution layer respectively, obtaining first spatial information and second spatial information outputted by the first depth convolution layer and the second depth convolution layer respectively;

[0023] The first spatial information, the second spatial information and the local features are fused to obtain a fusion result.

[0024] A further technical solution is to obtain a first enhanced feature map of the first downsampling network layer based on the fusion information, including:

[0025] The fusion information and the local features are fused to obtain the first enhanced feature map of the first downsampling network layer.

[0026] A further technical solution is that before fusing the first spatial information, the second spatial information and the local features, the method further includes:

[0027] The first spatial information and the second spatial information are respectively batch normalized to obtain first enhanced spatial information and second enhanced spatial information;

[0028] The first spatial information, the second spatial information and the local features are fused to obtain a fusion result, including:

[0029] The first enhanced spatial information, the second enhanced spatial information and the local features are fused to obtain a fusion result.

[0030] A further technical solution is that before the local features are input into the first deep convolutional layer and the second deep convolutional layer respectively, the method includes:

[0031] The local features are enhanced by using group normalization and GELU activation function to obtain enhanced local features;

[0032] Inputting the local features into the first depth convolution layer and the second depth convolution layer respectively, obtaining the first spatial information and the second spatial information outputted by the first depth convolution layer and the second depth convolution layer respectively, including:

[0033] The enhanced local features are respectively input into the first deep convolutional layer and the second deep convolutional layer to obtain the first spatial information and the second spatial information outputted by the first deep convolutional layer and the second deep convolutional layer respectively.

[0034] Its further technical solution is that the sparse attention calculation module includes average pooling, three third-depth convolution layers, a deconvolution layer and a point convolution layer, and the fusion result is input into the sparse attention calculation module for self-attention calculation to obtain the target global feature map, including:

[0035] Input the fusion result into average pooling to obtain a compressed feature map;

[0036] Input the compressed feature maps into three third-depth convolutional layers respectively to obtain feature vectors corresponding to the outputs of the three third-depth convolutional layers;

[0037] Perform self-attention calculation on the three feature vectors to obtain the initial global feature map;

[0038] Perform group normalization on the initial global feature map to obtain an enhanced initial global feature map;

[0039] The enhanced initial global feature map is input into the deconvolution layer for deconvolution operation to obtain a deconvolution image;

[0040] The deconvolution image is input into the point convolution layer for point convolution operation to obtain the target global feature map.

[0041] A further technical solution is that the three eigenvectors include a first eigenvector, a second eigenvector and a third eigenvector, and the three eigenvectors are self-attentionally calculated to obtain an initial global feature map, including:

[0042] The first eigenvector, the second eigenvector and the third eigenvector are self-attentioned to obtain an initial global feature map.

[0043] In a second aspect, the present application provides a medical image segmentation device, which includes a memory and a processor, the memory is used to store program data, and the processor is used to execute the program data to implement the steps of any of the above methods.

[0044] In a third aspect, the present application provides a computer-readable storage medium, which is used to store a computer program. When the computer program is executed by a processor, it is used to implement the steps of any of the above methods.

[0045] The beneficial effect of the present application is that, different from the prior art, the present application sets a first feature extraction enhancement module in the downsampling network layer and a second feature extraction enhancement module in the upsampling network layer, so that the first feature extraction enhancement module in the downsampling network layer can capture the high-level and semantic features of the image, while the second feature extraction enhancement module in the upsampling network layer focuses on detail recovery and feature refinement. This layered processing not only retains the global context information of the image, but also enhances the recognition ability of local details; and, in the upsampling process of each layer, the upsampling features of the current layer are fused with the enhanced feature map of the previous downsampling layer. This feature-level fusion strategy can integrate feature information of different scales and levels, and enhance the model's recognition ability of complex structures. In addition, by repeatedly transferring and enhancing features between downsampling and upsampling network layers, deep mining and step-by-step refinement of image information are achieved. Each iteration is optimized based on the previous features, which helps the model learn deeper image features, thereby improving the accuracy of segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0047] Figure 1 It is a flowchart of the first embodiment of the medical image segmentation method provided by the present application;

[0048] Figure 2 is a flowchart of a second embodiment of the medical image segmentation method provided by the present application;

[0049] Figure 3 It is a flow chart of the feature extraction enhancement module provided by this application;

[0050] Figure 4 It is a structural schematic diagram of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION

[0051] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It will be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some but not all structures related to the present application are shown in the drawings. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0052] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase at various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0053] In the field of medical imaging, especially in brain tumor segmentation, the development of technology has made certain progress. The current medical image segmentation technology mainly relies on the use of deep convolutional neural networks (DCNNs) and Transformer architecture, or even the fusion of the two. These advanced models have made remarkable achievements in improving the accuracy of medical image segmentation.

[0054] However, since the existing segmentation models focus on a specific task, although they can perform well in the specific task, when faced with a variety of complex segmentation tasks, the segmentation accuracy of the existing segmentation models is still relatively low.

[0055] Therefore, in order to solve the technical problem of low accuracy of image segmentation in the prior art, the present application provides a medical image segmentation method, which can improve the accuracy of medical image segmentation. Please refer to the following embodiments for details.

[0056] See also Figure 1 , Figure 1 1 is a flowchart of the first embodiment of the medical image segmentation method provided by the present application. The image segmentation model includes i downsampling network layers, i upsampling network layers and a segmentation head, wherein i is a positive natural number, the downsampling network layers each include a first feature extraction enhancement module and a downsampling module, and the upsampling network layers each include a second feature extraction enhancement module and an upsampling module, and the method includes the following steps:

[0057] Step 110: Input the initial feature map into the first feature extraction enhancement module in the first downsampling network layer to obtain the first enhanced feature map of the first downsampling network layer.

[0058] Among them, the original image can be first input into the feature extraction module for preliminary extraction to obtain an initial feature map.

[0059] The dimension of the original image may be M×H×W×D, where W, H, and D represent the width, height, and depth of the image, respectively, and M represents the number of input modes of the original image.

[0060] Among them, the first feature extraction enhancement module may include a channel fuser, a sparse attention calculation module and an FFN network layer, etc., and utilizes the deep convolution of different convolution kernels in parallel in the channel fuser to extract the local representation of the initial feature map, and then utilizes the sparse attention calculation module to perform average pooling, self-attention calculation and deconvolution operations on the local representation processed by the channel fuser, so as to extract the global features of the initial feature map, and after self-attention calculation, the global features of the original image are fused into the entire picture through deconvolution, thereby realizing the optimization processing of the original image and retaining the key details of the original image.

[0061] Step 120: Input the first enhanced feature map of the first downsampling network layer into the downsampling module in the first downsampling network layer to obtain the output feature of the first downsampling network layer.

[0062] Step 130: Repeat the operation by using the output features of the first downsampling network layer as the input of the next downsampling network layer until the first enhanced feature maps of each downsampling network layer and the output features of the i-th downsampling network layer are obtained.

[0063] See also Figure 2 The downsampling network layer may include 4 layers, specifically, a 1st downsampling network layer, a 2nd downsampling network layer, a 3rd downsampling network layer and a 4th downsampling network layer, each downsampling network layer is provided with a corresponding first feature extraction enhancement module, specifically, the output feature of the 1st downsampling network layer is input into the first feature extraction enhancement module in the 2nd downsampling network layer for operation to obtain a first enhanced feature map of the 2nd downsampling network layer, the first enhanced feature map of the 2nd downsampling network layer is input into the downsampling module in the 2nd downsampling network layer to obtain the output feature of the 2nd downsampling network layer; the output feature ... The output features of the third downsampling network layer are input into the first feature extraction enhancement module in the fourth downsampling network layer for operation to obtain the first enhanced feature map of the third downsampling network layer, and the first enhanced feature map of the third downsampling network layer is input into the downsampling module in the third downsampling network layer to obtain the output features of the third downsampling network layer; the output features of the third downsampling network layer are input into the first feature extraction enhancement module in the fourth downsampling network layer for operation to obtain the first enhanced feature map of the fourth downsampling network layer, and the first enhanced feature map of the fourth downsampling network layer is input into the downsampling module in the fourth downsampling network layer to obtain the output features of the fourth downsampling network layer.

[0064] Among them, the dimension of the first downsampling network layer can be C×H / 2×W / 2×D / 2; the dimension of the second downsampling network layer can be 2C×H / 4×W / 4×D / 4; the dimension of the third downsampling network layer can be 4C×H / 8×W / 8×D / 8; the dimension of the fourth downsampling network layer can be 8C×H / 16×W / 16×D / 16, where W, H and D represent the width, height and depth of the image respectively, and C represents the number of channels.

[0065] Step 140: The output features of the i-th downsampling network layer and the first enhanced feature map of the i-th downsampling network layer are fused and input into the second feature extraction enhancement module in the first upsampling network layer to obtain the second enhanced feature map of the first upsampling network layer.

[0066] For example, the output features of the 4th downsampling network layer and the first enhanced feature map of the 4th downsampling network layer are fused and input into the second feature extraction enhancement module in the 1st upsampling network layer to obtain the second enhanced feature map of the 1st upsampling network layer.

[0067] Step 150: Input the second enhanced feature map of the first upsampling network layer into the upsampling module in the first upsampling network layer to obtain the output feature of the first upsampling network layer.

[0068] Step 160: Repeat the operation of using the output features of the 1st upsampling network layer and the first enhanced feature map of the i-1th downsampling network layer as inputs of the second feature extraction enhancement module in the next upsampling network layer until the output features of the i-th upsampling network layer are obtained.

[0069] See also Figure 2 The upsampling network layer may include 4 layers, specifically, the 1st upsampling network layer, the 2nd upsampling network layer, the 3rd upsampling network layer and the 4th upsampling network layer. Each upsampling network layer is provided with a corresponding second feature extraction enhancement module. Specifically, the output feature of the 4th downsampling network layer is fused with the first enhanced feature map of the 4th downsampling network layer and input into the second feature extraction enhancement module in the 1st upsampling network layer to obtain the second enhanced feature map of the 1st upsampling network layer. The second enhanced feature map of the 1st upsampling network layer is input into the upsampling module in the 1st upsampling network layer to obtain the output feature of the 1st upsampling network layer.

[0070] The output features of the first upsampling network layer and the first enhanced feature map of the third downsampling network layer are input into the second feature extraction enhancement module in the second upsampling network layer to obtain the second enhanced feature map of the second upsampling network layer. The second enhanced feature map of the second upsampling network layer is input into the upsampling module in the second upsampling network layer to obtain the output features of the second upsampling network layer.

[0071] The output features of the second upsampling network layer and the first enhanced feature map of the second downsampling network layer are input into the second feature extraction enhancement module in the third upsampling network layer to obtain the second enhanced feature map of the third upsampling network layer. The second enhanced feature map of the third upsampling network layer is input into the upsampling module in the third upsampling network layer to obtain the output features of the third upsampling network layer.

[0072] The output features of the 3rd upsampling network layer and the first enhanced feature map of the 1st downsampling network layer are input into the second feature extraction enhancement module in the 4th upsampling network layer to obtain the second enhanced feature map of the 4th upsampling network layer. The second enhanced feature map of the 4th upsampling network layer is input into the upsampling module in the 4th upsampling network layer to obtain the output features of the 4th upsampling network layer.

[0073] Among them, the dimension of the first upsampling network layer can be 8C×H / 16×W / 16×D / 16; the dimension of the second upsampling network layer can be 4C×H / 8×W / 8×D / 82C×H / 4×W / 4×D / 4; the dimension of the third upsampling network layer can be 2C×H / 4×W / 4×D / 4; the dimension of the fourth upsampling network layer can be C×H / 2×W / 2×D / 2, where W, H and D represent the width, height and depth of the image respectively, and C represents the number of channels.

[0074] Step 170: Input the output features of the i-th upsampling network layer into the segmentation head to obtain a segmentation image corresponding to the initial feature map output by the segmentation head.

[0075] The dimension of the obtained segmented image is N×H×W×D, where N represents the type of segmentation.

[0076] This embodiment sets a first feature extraction enhancement module in the downsampling network layer and a second feature extraction enhancement module in the upsampling network layer, so that the first feature extraction enhancement module in the downsampling network layer can capture the high-level and semantic features of the image, while the second feature extraction enhancement module in the upsampling network layer focuses on detail recovery and feature refinement. This layered processing not only retains the global context information of the image, but also enhances the recognition ability of local details; and in the upsampling process of each layer, the upsampling features of the current layer are fused with the enhanced feature map of the previous downsampling layer. This feature-level fusion strategy can integrate feature information of different scales and levels and enhance the model's recognition ability for complex structures. In addition, by repeatedly transferring and enhancing features between the downsampling and upsampling network layers, deep mining and step-by-step refinement of image information are achieved. Each iteration is optimized based on the previous features, which helps the model learn deeper image features, thereby improving the accuracy of segmentation.

[0077] Existing medical image segmentation techniques, despite their excellent performance on specific tasks, are limited by their highly targeted design, resulting in performance degradation when encountering new datasets. This limitation hinders their widespread application in clinical settings and limits their effectiveness in the medical field.

[0078] Overcoming this limitation requires a deep understanding of the visual and morphological challenges inherent in different biomedical features. First, the anatomical structures in medical images exhibit a variety of geometric shapes, such as the branching structure of the airway in sharp contrast to the smooth shape of the liver, requiring segmentation technology to accurately capture and distinguish the unique contours of these structures, which is undoubtedly the first test of segmentation accuracy.

[0079] Second, the difficulty of segmentation is exacerbated by the heterogeneity within different tissues, such as the distinct differences between dense cortical bone and porous cancellous bone in bone tissue, and between muscle tissue and blood-filled chambers in the heart region, emphasizing the necessity of identifying and adapting to the variability of these internal properties during the segmentation process.

[0080] Furthermore, adjacent anatomical structures often exhibit similar visual and density features. For example, the close proximity of the pancreas and stomach makes them difficult to distinguish in imaging. Such situations require the segmentation algorithm to have a high degree of resolution to ensure accurate distinction even when visual information overlaps.

[0081] Based on this, this application introduces a versatile end-to-end medical image segmentation framework to significantly improve the ability to understand and capture the complex relationship between anatomical structures and pathological areas. By adopting hierarchical representation learning and adaptive local-global fusion technology, the ability to recognize geometric configurations, internal structures and unique attributes of different anatomical morphologies is improved to adapt to data sets of different scales, quantities and types. For details, please refer to the second embodiment below.

[0082] See also Figure 2 , Figure 2 It is a flowchart of the second embodiment of the medical image segmentation method provided in this application.

[0083] Step 210: Input the initial feature map into the first feature extraction enhancement module in the first downsampling network layer to obtain the first enhanced feature map of the first downsampling network layer.

[0084] The first feature extraction enhancement module and the second feature extraction enhancement module are the same, both including a channel fusion module, a sparse attention calculation module and an FFN network layer. Step 210 may specifically include the following steps:

[0085] Step 211: Input the initial feature map into the channel fuser in the first downsampling network layer to obtain a fusion result output by the channel fuser.

[0086] The channel fuser includes a first convolutional layer, a first depth convolutional layer and a second depth convolutional layer, wherein the convolution kernel sizes of the first depth convolutional layer and the second depth convolutional layer are different.

[0087] Among them, the convolution kernel size of the first convolution layer is 3×3×3, the convolution kernel size of the first deep convolution layer is 3×3×3, and the convolution kernel size of the second deep convolution layer is 1×1×1.

[0088] Step 211 may specifically include the following:

[0089] Step 11: Input the initial feature map into the first convolutional layer to obtain the local features of the initial feature map.

[0090] Among them, the convolution kernel size of the first convolutional layer is 3×3×3, and it is processed through 3×3×3 convolutional layers to capture detailed local features on various receptive fields.

[0091] Step 12: Input the local features into the first deep convolution layer and the second deep convolution layer respectively to obtain the first spatial information and the second spatial information outputted by the first deep convolution layer and the second deep convolution layer respectively.

[0092] Specifically, the first 3×3×3 deep convolutional layer is used to extract complex spatial information in order to understand the broader context of the image, while the second 1×1×1 deep convolutional layer is used to focus on separating more detailed and localized features.

[0093] The initial feature map is convolved simultaneously by using two deep convolutional layers with different convolution kernel sizes to fully capture image details.

[0094] In some embodiments, before the local features are input into the first deep convolutional layer and the second deep convolutional layer respectively, the method includes:

[0095] 1) Group normalization and GELU activation function are used to enhance local features to obtain enhanced local features.

[0096] Nonlinear activation is performed through group normalization (GroupNorm) and GELU activation function to refine the quality of the extracted features and thus improve the segmentation performance.

[0097] 2) Inputting the enhanced local features into the first deep convolutional layer and the second deep convolutional layer respectively, obtaining the first spatial information and the second spatial information outputted by the first deep convolutional layer and the second deep convolutional layer respectively.

[0098] Step 13: Fuse the first spatial information, the second spatial information and the local features to obtain a fusion result.

[0099] Similarly, in some embodiments, before fusing the first spatial information, the second spatial information, and the local features, the method further includes:

[0100] 1) performing batch normalization processing on the first spatial information and the second spatial information respectively to obtain first enhanced spatial information and second enhanced spatial information;

[0101] 2) Fusing the first enhanced spatial information, the second enhanced spatial information and the local features to obtain a fusion result.

[0102] Step 212: Input the fusion result into the sparse attention calculation module for self-attention calculation, and obtain the target global feature map output by the sparse attention calculation module.

[0103] Among them, the sparse attention calculation module includes average pooling, three third-depth convolutional layers, deconvolution layers, and point convolution layers.

[0104] Among them, the step rate r of average pooling can be (4, 2, 2, 1).

[0105] The convolution kernel size of the three third depth convolution layers is 1×1×1, and the convolution kernel size of the deconvolution layer and the point convolution layer is 1×1×1.

[0106] Step 212 may specifically include the following:

[0107] Step 21: Input the fusion result into average pooling to obtain a compressed feature map.

[0108] Among them, the fusion result Y is processed by an average pooling operation with a step rate r = (4, 2, 2, 1) to compress the feature map, thereby reducing the dimension of the feature map to W / r×H / r×D / r×C, where W, H and D represent the width, height and depth of the image respectively, and C represents the number of channels.

[0109] Step 22: Input the compressed feature map into three third-depth convolutional layers respectively to obtain feature vectors corresponding to the outputs of the three third-depth convolutional layers.

[0110] Step 23: Perform self-attention calculation on the three feature vectors to obtain the initial global feature map.

[0111] Among them, the three eigenvectors include a first eigenvector, a second eigenvector and a third eigenvector. The first eigenvector, the second eigenvector and the third eigenvector are self-attentionally calculated to obtain an initial global feature map.

[0112] Among them, the first eigenvector, the second eigenvector and the third eigenvector may be weight vectors, for example, the first eigenvector is a K weight vector, the second eigenvector is a Q weight vector, and the third eigenvector is a V weight vector.

[0113] Step 24: Perform group normalization on the initial global feature map to obtain an enhanced initial global feature map.

[0114] Step 25: Input the enhanced initial global feature map into the deconvolution layer for deconvolution operation to obtain a deconvolution image.

[0115] The deconvolution layer is used to resize the enhanced initial global feature map back to the dimensions of the original input image. The deconvolution operation integrates the image feature representation extracted by the self-attention operation into the original input image, which not only maintains the input resolution, but also preserves the detailed information inherent in the original image, ensuring the integrity of the global features.

[0116] Step 26: Input the deconvolution image into the point convolution layer for point convolution operation to obtain the target global feature map.

[0117] Among them, the point convolution layer is a 1×1×1 feature fusion layer.

[0118] Step 213: Input the target global feature map into the FFN network layer to obtain the fusion information output by the FFN network layer.

[0119] Step 214: Based on the fusion information, a first enhanced feature map of the first downsampling network layer is obtained.

[0120] The fusion information and the local features are fused to obtain the first enhanced feature map of the first downsampling network layer.

[0121] Step 220: Input the first enhanced feature map of the first downsampling network layer into the downsampling module in the first downsampling network layer to obtain the output feature of the first downsampling network layer.

[0122] Step 230: Repeat the operation by using the output features of the first downsampling network layer as the input of the next downsampling network layer until the first enhanced feature maps of each downsampling network layer and the output features of the i-th downsampling network layer are obtained.

[0123] Step 240: The output features of the i-th downsampling network layer and the first enhanced feature map of the i-th downsampling network layer are fused and input into the second feature extraction enhancement module in the first upsampling network layer to obtain the second enhanced feature map of the first upsampling network layer.

[0124] Step 250: Input the second enhanced feature map of the first upsampling network layer into the upsampling module in the first upsampling network layer to obtain the output feature of the first upsampling network layer.

[0125] Step 260: Repeat the operation of using the output features of the 1st upsampling network layer and the first enhanced feature map of the i-1th downsampling network layer as inputs of the second feature extraction enhancement module in the next upsampling network layer until the output features of the i-th upsampling network layer are obtained.

[0126] Step 270: Input the output features of the i-th upsampling network layer into the segmentation head to obtain a segmentation image corresponding to the initial feature map output by the segmentation head.

[0127] The operation process of the second feature extraction and enhancement module is similar to the steps of the first feature extraction and enhancement module. For details, please refer to the relevant description of the first feature extraction and enhancement module.

[0128] The repeated operations of the downsampling network layer and the upsampling network layer have the same or similar scheme as that of the first embodiment. For details, please refer to the relevant description of the above-mentioned first embodiment, and this application will not go into details here.

[0129] The present application sets up a feature extraction enhancement module including a channel fuser, a sparse attention calculation module and an FFN network layer, so that the channel fuser uses parallel deep convolution of different convolution kernels to extract local features of medical images, and then the sparse attention module effectively extracts global features of medical images by using average pooling, self-attention calculation and deconvolution operations. That is, the hierarchical representation learning and adaptive local-global fusion technology are adopted to effectively improve the recognition ability of geometric configuration, internal structure and unique attributes of different anatomical forms, so that the segmentation model provided by the present invention can adapt to data sets of different scales, quantities and types.

[0130] Combining the above embodiments and Figure 2 , Figure 3 The feature extraction enhancement module (including the first feature extraction enhancement module and the second feature extraction enhancement module) in the medical image segmentation model provided in the present application includes a channel fuser 10, a sparse attention calculation module 20 and an FFN network layer 30, wherein the specific operations and interactions of the channel fuser 10, the sparse attention calculation module 20 and the FFN network layer 30 can refer to the following process:

[0131] S1: Input feature map.

[0132] That is, the feature map is input into the channel fuser.

[0133] S2: The feature map passes through a 3×3×3 convolutional layer to obtain local features.

[0134] That is, the first convolutional layer with a convolution kernel size of 3×3×3 is used to perform a convolution operation on the feature map to obtain local features.

[0135] S3: Group normalization (GroupNorm) and GELU activation function are used to process local features to obtain enhanced local features.

[0136] S4: Input a 3×3×3 deep convolutional layer to obtain the first spatial information.

[0137] That is, the first deep convolution layer with a convolution kernel size of 3×3×3 is used to extract complex spatial information of local features to obtain the first spatial information.

[0138] S5: Input a 1×1×1 deep convolutional layer to obtain the second spatial information.

[0139] At the same time, the second deep convolution layer with a convolution kernel size of 1×1×1 is used to perform convolution processing on the local features to obtain the second spatial information.

[0140] S6: Normalize the first spatial information.

[0141] S7: Normalize the second spatial information.

[0142] For S6 and S7, the first spatial information and the second spatial information are batch normalized respectively to obtain the first enhanced spatial information and the second enhanced spatial information; so as to fuse the first enhanced spatial information, the second enhanced spatial information and the local features to obtain a fusion result.

[0143] S8: Input the fusion result into average pooling to obtain a compressed feature map.

[0144] That is, the fusion result is input into the average pooling in the sparse attention calculation module to obtain a compressed feature map.

[0145] S9: Input the compressed feature map into three 1×1×1 convolutional layers for processing to obtain K, Q, and V feature vectors.

[0146] That is, the compressed feature maps are respectively input into three third-depth convolutional layers to obtain feature vectors corresponding to the outputs of the three third-depth convolutional layers. For example, the first feature vector is a K weight vector, the second feature vector is a Q weight vector, and the third feature vector is a V weight vector.

[0147] S10: Perform self-attention calculation on the three feature vectors to obtain the initial global feature map.

[0148] S11: Perform group normalization on the initial global feature map to obtain an enhanced initial global feature map.

[0149] S12: Input the enhanced initial global feature map into the deconvolution layer for deconvolution operation to obtain a deconvolution image.

[0150] S13: Input the deconvolution image into a 1×1×1 point convolution layer for convolution operation to obtain the final global feature, i.e., the target global feature map.

[0151] Then the target global feature map is input into the FFN network layer to obtain the fusion information output by the FFN network layer.

[0152] Among them, the FFN network layer includes two 1×1×1 point convolutional layers. The specific operations of the FFN network layer include the following:

[0153] 1) Input the target global feature map into the 1×1×1 point convolution layer for point convolution operation to obtain preliminary fusion information;

[0154] 2) The preliminary fusion information is batch normalized to obtain enhanced fusion information;

[0155] 3) Using GELU activation function to process the enhanced fusion information;

[0156] 4) Input the fused information processed by the GELU activation function into the 1×1×1 point convolution layer for point convolution operation;

[0157] 5) The fused information obtained after the convolution operation is batch normalized to obtain the final fused information, that is, the fused information output by the FFN network layer.

[0158] By introducing the channel fuser and sparse attention calculation module, the channel fuser uses parallel deep convolution of different convolution kernels to extract local features of medical images. Then the sparse attention module effectively extracts the global features of medical images by utilizing average pooling, self-attention calculation and deconvolution operations. That is, the hierarchical representation learning and adaptive local-global fusion technology are adopted to effectively improve the recognition ability of geometric configuration, internal structure and unique attributes of different anatomical forms.

[0159] Corresponding to the medical image segmentation method of the above embodiment, the present application also provides a medical image segmentation device, which includes a first input unit, a downsampling unit, a first circulation unit, a second input unit, an upsampling unit, a second circulation unit and a segmentation unit.

[0160] A first input unit is used to input the initial feature map into the first feature extraction enhancement module in the first downsampling network layer to obtain a first enhanced feature map of the first downsampling network layer;

[0161] A downsampling unit, used for inputting the first enhanced feature map of the first downsampling network layer into the downsampling module in the first downsampling network layer to obtain the output feature of the first downsampling network layer;

[0162] The first cycle unit is used to repeatedly operate the output features of the first downsampling network layer as the input of the next downsampling network layer until the first enhanced feature map of each downsampling network layer and the output features of the i-th downsampling network layer are obtained;

[0163] The second input unit is used to fuse the output features of the i-th downsampling network layer with the first enhanced feature map of the i-th downsampling network layer and input them into the second feature extraction enhancement module in the first upsampling network layer to obtain the second enhanced feature map of the first upsampling network layer;

[0164] An upsampling unit, used for inputting the second enhanced feature map of the first upsampling network layer into the upsampling module in the first upsampling network layer to obtain the output feature of the first upsampling network layer;

[0165] The second cycle unit is used to repeatedly operate the output features of the first upsampling network layer and the first enhanced feature map of the i-1th downsampling network layer as inputs of the second feature extraction enhancement module in the next upsampling network layer until the output features of the i-th upsampling network layer are obtained;

[0166] The segmentation unit is used to input the output features of the i-th upsampling network layer into the segmentation head to obtain a segmentation image corresponding to the initial feature map output by the segmentation head.

[0167] It can be understood that the above-mentioned units are also used to implement the technical solution of any embodiment in the present application.

[0168] The present application also provides an image enhancement system, which includes a memory and a processor, wherein the memory stores a computer program; the processor is used to implement the medical image segmentation method provided by any one of the aforementioned method embodiments when executing the computer program.

[0169] See also Figure 4 , Figure 4 The structure of an embodiment of a computer-readable storage medium provided by the present application is shown in FIG. 7 . The computer-readable storage medium 70 is used to store a computer program 71. When the computer program 71 is executed by a processor, it is used to implement the following method steps:

[0170] Inputting the initial feature map into the first feature extraction enhancement module in the first downsampling network layer to obtain the first enhanced feature map of the first downsampling network layer;

[0171] Inputting the first enhanced feature map of the first downsampling network layer into the downsampling module in the first downsampling network layer to obtain the output feature of the first downsampling network layer;

[0172] The output features of the first downsampling network layer are used as the input of the next downsampling network layer and the operation is repeated until the first enhanced feature map of each downsampling network layer and the output features of the i-th downsampling network layer are obtained;

[0173] The output features of the i-th downsampling network layer and the first enhanced feature map of the i-th downsampling network layer are fused and input into the second feature extraction enhancement module in the first upsampling network layer to obtain the second enhanced feature map of the first upsampling network layer;

[0174] Inputting the second enhanced feature map of the first upsampling network layer into the upsampling module in the first upsampling network layer to obtain the output feature of the first upsampling network layer;

[0175] The output features of the first upsampling network layer and the first enhanced feature map of the i-1th downsampling network layer are used as the input of the second feature extraction enhancement module in the next upsampling network layer, and the operation is repeated until the output features of the i-th upsampling network layer are obtained;

[0176] The output features of the i-th upsampling network layer are input into the segmentation head to obtain the segmentation image corresponding to the initial feature map output by the segmentation head.

[0177] It can be understood that when the computer program 71 is executed by the processor, it is also used to implement the technical solution of any embodiment of the present application.

[0178] In the several layers of implementation provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only illustrative, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as two layers of units or components can be combined or integrated into another layer of the system, or some features can be ignored or not executed.

[0179] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one layer or distributed on two layers of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of this embodiment.

[0180] In addition, each functional unit in each layer implementation mode of the present application can be integrated into a layer of processing units, or each layer unit can exist physically separately, or two or more layers of units can be integrated into a layer of units. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0181] If the integrated units in the above other implementations are implemented in the form of software functional units and sold or used as independent products, they can be stored in a layer of computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a layer of storage medium, including several instructions to enable a computer device (which can be a layer of personal computers, servers, or network devices, etc.) or a processor (processor) to execute all or part of the steps of the implementation methods of each layer of the present application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM, Read-On lyMemory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0182] The above are only implementation methods of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A medical image segmentation method, characterized in that: The image segmentation model includes 4 downsampling network layers, 4 upsampling network layers and a segmentation head, wherein the downsampling network layers each include a first feature extraction enhancement module and a downsampling module, and the upsampling network layers each include a second feature extraction enhancement module and an upsampling module, and the method includes: Inputting the initial feature map into the first feature extraction enhancement module in the first downsampling network layer to obtain the first enhanced feature map of the first downsampling network layer; Inputting the first enhanced feature map of the first downsampling network layer into the downsampling module in the first downsampling network layer to obtain the output feature of the first downsampling network layer; The output features of the first downsampling network layer are used as the input of the next downsampling network layer to repeat the operation until the first enhanced feature map of each downsampling network layer and the output features of the i-th downsampling network layer are obtained; The output features of the i-th downsampling network layer and the first enhanced feature map of the i-th downsampling network layer are fused and input into the second feature extraction enhancement module in the first upsampling network layer to obtain the second enhanced feature map of the first upsampling network layer; Inputting the second enhanced feature map of the first upsampling network layer into the upsampling module in the first upsampling network layer to obtain output features of the first upsampling network layer; Repeat the operation by using the output features of the first upsampling network layer and the first enhanced feature map of the i-1th downsampling network layer as inputs of the second feature extraction enhancement module in the next upsampling network layer until the output features of the i-th upsampling network layer are obtained, where i=2, 3, 4; Inputting the output features of the 4th upsampling network layer into the segmentation head to obtain a segmentation image corresponding to the initial feature map output by the segmentation head; The first feature extraction enhancement module and the second feature extraction enhancement module are the same, both including a channel fusion module, a sparse attention calculation module and an FFN network layer; The step of inputting the initial feature map into the first feature extraction enhancement module in the first downsampling network layer to obtain the first enhanced feature map of the first downsampling network layer includes: Inputting the initial feature map into the channel fuser in the first downsampling network layer to obtain a fusion result output by the channel fuser; The fusion result is input into the sparse attention calculation module to perform self-attention calculation to obtain a target global feature map; Inputting the target global feature map into the FFN network layer to obtain fusion information output by the FFN network layer; Based on the fusion information, a first enhanced feature map of the first downsampling network layer is obtained; The channel fuser includes a first convolutional layer, a first depth convolutional layer, and a second depth convolutional layer, wherein the convolution kernel sizes of the first depth convolutional layer and the second depth convolutional layer are different; The step of inputting the initial feature map into a channel fuser in the first downsampling network layer to obtain a fusion result output by the channel fuser includes: Inputting the initial feature map into the first convolutional layer to obtain local features of the initial feature map; Inputting the local features into the first deep convolution layer and the second deep convolution layer respectively, to obtain first spatial information and second spatial information outputted by the first deep convolution layer and the second deep convolution layer respectively; The first spatial information, the second spatial information and the local features are fused to obtain a fusion result.

2. The medical image segmentation method according to claim 1, characterized in that: The step of obtaining a first enhanced feature map of the first downsampling network layer based on the fusion information includes: The fusion information and the local features are fused to obtain a first enhanced feature map of the first downsampling network layer.

3. The medical image segmentation method according to claim 1, characterized in that: Before fusing the first spatial information, the second spatial information, and the local features, the method further includes: Performing batch normalization processing on the first spatial information and the second spatial information respectively to obtain first enhanced spatial information and second enhanced spatial information; The fusing the first spatial information, the second spatial information and the local features to obtain a fusion result includes: The first enhanced spatial information, the second enhanced spatial information, and the local feature are fused to obtain a fusion result.

4. The medical image segmentation method according to claim 1, characterized in that: Before inputting the local features into the first deep convolutional layer and the second deep convolutional layer respectively, the method comprises: The local features are enhanced by using group normalization and GELU activation function to obtain enhanced local features; The inputting the local features into the first deep convolution layer and the second deep convolution layer respectively to obtain first spatial information and second spatial information outputted by the first deep convolution layer and the second deep convolution layer respectively includes: The enhanced local features are respectively input into the first deep convolution layer and the second deep convolution layer to obtain first spatial information and second spatial information outputted by the first deep convolution layer and the second deep convolution layer respectively.

5. The medical image segmentation method according to claim 1, characterized in that: The sparse attention calculation module includes average pooling, three third depth convolution layers, a deconvolution layer and a point convolution layer; The step of inputting the fusion result into the sparse attention calculation module to perform self-attention calculation to obtain a target global feature map comprises: Inputting the fusion result into the average pooling to obtain a compressed feature map; Inputting the compressed feature map into three third depth convolutional layers respectively to obtain feature vectors corresponding to outputs of multiple third depth convolutional layers; Perform self-attention calculation on the three feature vectors to obtain an initial global feature map; Performing group normalization processing on the initial global feature map to obtain an enhanced initial global feature map; Inputting the enhanced initial global feature map into the deconvolution layer for deconvolution operation to obtain a deconvolution image; The deconvolution image is input into the point convolution layer to perform a point convolution operation to obtain a target global feature map.

6. The medical image segmentation method according to claim 5, characterized in that: The three feature vectors include a first feature vector, a second feature vector and a third feature vector, and the three feature vectors are self-attention-calculated to obtain an initial global feature map, including: Perform self-attention calculation on the first eigenvector, the second eigenvector, and the third eigenvector to obtain an initial global feature map.

7. A medical image segmentation device, characterized in that: The medical image segmentation device comprises a memory and a processor, the memory is used to store program data, and the processor is used to execute the program data to implement the medical image segmentation method according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program data, and when the program data is executed by a processor, it is used to implement the medical image segmentation method according to any one of claims 1 to 6.

Citation Information

Patent Citations

  • Semantic segmentation-based traffic road adhesion coefficient prediction model and prediction method

    CN116503834A

  • Multi-scale lightweight real-time semantic segmentation method and system

    CN116612288A

  • Multi-branch image semantic segmentation method and system based on AM and feature fusion

    CN116681889A

  • Three-dimensional medical image segmentation method and system based on adaptive feature fusion network

    CN117058160A

  • Medical image segmentation method based on boundary perception and attention mechanism

    CN117078930A