An ultrasound image segmentation method based on CNN and Transformer
By combining the self-attention mechanism of CNN and Transformer, U-Net and Swin-Transformer encoders are used for feature extraction and fusion, the problems of low ultrasonic image segmentation accuracy and poor robustness are solved, and high-precision, stable and efficient image segmentation are achieved.
Patent Information
- Application Number
- CN202310921204.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-26
- Publication Date
- 2025-08-22
- Estimated Expiration
- 2043-07-26
AI Technical Summary
When the existing ultrasonic image segmentation method faces factors such as noise, artifacts and low contrast, the segmentation accuracy is low and the robustness is poor, making it difficult to accurately extract the region of interest.
Combining the self-attention mechanism of convolutional neural network (CNN) and Transformer, features are extracted through U-Net and Swin-Transformer encoders, and deep fusion is used to achieve efficient extraction and splicing of features.
It improves the accuracy and robustness of ultrasonic image segmentation, can accurately divide the target areas, adapt to different image quality, and has high computing efficiency, meeting clinical application needs.
Smart Images
Figure CN116758102B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of medical image segmentation, and specifically relates to an ultrasound image segmentation method based on CNN and Transformer. Background Art
[0002] Ultrasound imaging is a technology that uses ultrasound equipment to transmit ultrasonic waves and receive reflected signals from the human body, processing them to produce images of internal organs. It has the advantages of being non-invasive, radiation-free, convenient, and low-cost. Ultrasound imaging accounts for a large proportion of medical imaging, accounting for approximately 74.44% of the total number of medical imaging devices in use. It is widely used in fields such as cardiology, obstetrics and gynecology, oncology, and neurology, and plays a key role in the diagnosis and treatment of diseases. However, due to the characteristics of ultrasound imaging, such as low contrast, noise interference, and the diversity of disease morphology, size, and location, accurately extracting regions of interest from ultrasound images remains a challenge. Currently, the recognition and segmentation of ultrasound images mainly rely on experienced doctors. Therefore, developing an algorithm that can automatically segment ultrasound images has great practical value and practical significance.
[0003] The current mainstream medical image segmentation methods mainly include four categories: threshold-based segmentation, region-based segmentation, edge-based segmentation, and deep learning-based segmentation.
[0004] Among them, the threshold-based segmentation method is a simple and effective method that divides the image according to the grayscale value or color value of the pixels in the image. However, it is less effective when the grayscale value or color value in the image is uneven or there is noise.
[0005] Region-based segmentation is a method that divides an image based on the similarity or connectivity between pixels in the image. It can handle situations where the grayscale value or color value in the image is uneven or there is noise, but it is less effective when there is a complex background or unclear target boundaries in the image.
[0006] Edge-based segmentation is a method that detects the boundary of an object based on the change in grayscale or color value between the object and the background in an image. It can handle situations where there is a complex background or unclear boundary of the object in the image, but it is less effective when there is noise or discontinuous boundary of the object in the image.
[0007] The deep learning-based segmentation method is a method that uses deep neural networks to learn the feature representation between the target and background in the image, and predicts whether each pixel belongs to the target or the background based on the feature representation. Among them, convolutional neural networks are widely used in medical image segmentation. However, convolutional neural networks are very sensitive to factors such as noise, artifacts, and low contrast in the image. These factors will affect the ability of convolutional neural networks to extract image features, thereby reducing the accuracy and robustness of segmentation. Summary of the Invention
[0008] The purpose of the present invention is to solve the problems of low ultrasound image segmentation accuracy and high sensitivity to factors such as noise, artifacts, and low contrast in the above-mentioned prior art, and to provide an ultrasound image segmentation method based on CNN and Transformer.
[0009] To achieve the above objectives, the technical solution adopted in the present invention is to combine the traditional convolution operation and the self-attention mechanism, making full use of the advantages of both in extracting features and capturing global dependencies, thereby proposing an ultrasound image segmentation method based on CNN and Transformer.
[0010] Which includes the following steps:
[0011] (1) Obtaining ultrasound images;
[0012] The original image sequence acquisition method is as follows: a medical image X is obtained by a medical ultrasound imaging device, which has a spatial resolution of H×W and C channels.
[0013] (2) The original ultrasound image is passed through the image Unet encoder to obtain the feature map;
[0014] In a U-Net encoder, the encoder consists of multiple convolutional and pooling layers to extract image features. After an image is fed into the encoder, each pooling layer halves the image's resolution and doubles the number of feature channels. The encoder output is a low-resolution, high-channel feature map.
[0015] The encoder extracts image features layer by layer. Its structure consists of four stages, each consisting of two 3×3 convolutions, batch normalization and activation, and a downsampling layer using 2×2 max pooling. With each stage, the output feature map is scaled down by half, while the channel dimension is doubled. After feature extraction by the U-net encoder, the feature vector has dimensions H / 16×W / 16×1024.
[0016] (3) The original ultrasound image is passed through the image SwinT encoder to obtain the feature map for input image serialization. Patch segmentation decomposes the input image X into N image blocks of size P×P through convolution operation and flattens them into a sequence
[0017] Then the linear embedding layer will perform convolution operations on Perform a linear transformation and map it to a space of dimension C.
[0018] In the subsequent Swin-Transformer block, there are two modules: window self-attention and sliding window self-attention. Through these two modules, we can get:
[0019]
[0020]
[0021]
[0022]
[0023] in with x l Represents the output of the l-th layer window self-attention module and the multi-layer perceptron module, with x l+1 is the output of the l+1-layer sliding window self-attention module and the multilayer perceptron module, and LN is layer normalization. After four stages of stacking the SwinT module, the feature map of the ultrasound image is finally obtained.
[0024] (4) Feature fusion is performed on the feature maps of the two channels in steps (2) and (3). After feature extraction by the Swin-T encoder and the U-Net encoder, two feature maps are obtained. The features obtained by the two paths are deeply fused through the GDFF module. The specific formula of this module is as follows:
[0025] F conv =DC(TP(F Swin-T )) (5)
[0026] F fuse =G(Convcat(F conv ,F U-net )) (6)
[0027] Where DC represents two layers of convolution, batch normalization, and activation, TP represents the transposed convolution module, G represents the gating mechanism, and x sw with x u These are the features of different scales extracted by the Swin-T encoder and the U-Net encoder respectively;
[0028] (5) The layer-by-layer features obtained in step (2) and the fused bottom-layer features obtained in step (4) are combined with the decoder. During the decoding process, the features extracted by the gated convolution feature fusion module and the features extracted by the upsampled Swin-T encoder are firstly converted to ×2 upsampling by transposed convolution. The features extracted by the corresponding layer of the U-Net encoder module are concatenated and double-layered convolution, batch normalization and activation operations are performed. After obtaining the output of this layer, the next layer decoder will again upsample the features of the previous layer with the Swin-T encoder, and concatenate and convolve with the U-Net encoder module of the corresponding layer. In this way, in the next three stages, the dimensions of And the feature vector of H×W×64, after two 3×3 convolutions, the final segmentation prediction result with the dimension of H×W×1 is obtained.
[0029] The present invention is a deep learning-based ultrasound image segmentation method that can effectively identify and extract regions of interest in ultrasound images, such as fetuses, placentas, tumors, etc. Compared with existing threshold, edge, and region growing-based technologies, the present invention has the following advantages: (1) High ultrasound image segmentation accuracy, which can accurately delineate the boundaries of the target region and avoid the problem of over-segmentation or under-segmentation; (2) Strong robustness, which can adapt to changes in ultrasound image quality, noise, contrast, and other factors, ensuring the stability and consistency of the segmentation results; (3) High computational efficiency, which utilizes the parallel computing power of deep learning models to quickly process large amounts of ultrasound image data and meet the needs of clinical applications. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] Figure 1 This is a flowchart of an ultrasound image segmentation method based on CNN and Transformer in the present invention.
[0031] Figure 2 It is a flow chart of the GDFF feature fusion method of the present invention. DETAILED DESCRIPTION
[0032] like Figure 1 、 Figure 2 As shown, this embodiment implements an ultrasound image segmentation method based on CNN and Transformer, including the following steps:
[0033] (1) Obtaining ultrasound images;
[0034] The original image sequence acquisition method is as follows: a medical image X is obtained by a medical ultrasound imaging device, which has a spatial resolution of H×W and C channels.
[0035] (2) The original ultrasound image is passed through the image Unet encoder to obtain the feature map;
[0036] In a U-Net encoder, the encoder consists of multiple convolutional and pooling layers to extract image features. After an image is fed into the encoder, each pooling layer halves the image's resolution and doubles the number of feature channels. The encoder output is a low-resolution, high-channel feature map.
[0037] The encoder extracts image features layer by layer. Its structure consists of four stages, each consisting of two 3×3 convolutions, batch normalization and activation, and a downsampling layer using 2×2 max pooling. With each stage, the output feature map is scaled down by half, while the channel dimension is doubled. After feature extraction by the U-net encoder, the feature vector has dimensions H / 16×W / 16×1024.
[0038] (3) The original ultrasound image is passed through the image SwinT encoder to obtain the feature map for input image serialization. Patch segmentation decomposes the input image X into N image blocks of size P×P through convolution operation and flattens them into a sequence
[0039] Then the linear embedding layer will perform convolution operations on Perform a linear transformation and map it to a space of dimension C.
[0040] In the subsequent Swin-Transformer block, there are two modules: window self-attention and sliding window self-attention. Through these two modules, we can get:
[0041]
[0042]
[0043]
[0044]
[0045] in with x l Represents the output of the l-th layer window self-attention module and the multi-layer perceptron module, with x l+1 is the output of the l+1-layer sliding window self-attention module and the multilayer perceptron module, and LN is layer normalization. After four stages of stacking the SwinT module, the feature map of the ultrasound image is finally obtained.
[0046] (4) Feature fusion is performed on the feature maps of the two channels in steps (2) and (3). After feature extraction by the Swin-T encoder and the U-Net encoder, two feature maps are obtained. The features obtained by the two paths are deeply fused through the GDFF module. The specific formula of this module is as follows:
[0047] F conv =DC(TP(F Swin-T )) (5)
[0048] F fuse =G(Concat(F conv ,F U-net )) (6)
[0049] Where DC represents two layers of convolution, batch normalization, and activation, TP represents the transposed convolution module, G represents the gating mechanism, and x sw with x u These are the features of different scales extracted by the Swin-T encoder and the U-Net encoder respectively;
[0050] (5) The layer-by-layer features obtained in step (2) and the fused bottom-layer features obtained in step (4) are combined with the decoder. During the decoding process, the features extracted by the gated convolution feature fusion module and the features extracted by the upsampled Swin-T encoder are firstly converted to ×2 upsampling by transposed convolution. The features extracted by the corresponding layer of the U-Net encoder module are concatenated and double-layered convolution, batch normalization and activation operations are performed. After obtaining the output of this layer, the next layer decoder will again upsample the features of the previous layer with the Swin-T encoder, and concatenate and convolve with the U-Net encoder module of the corresponding layer. In this way, in the next three stages, the dimensions of And the feature vector of H×W×64, after two 3×3 convolutions, the final segmentation prediction result with the dimension of H×W×1 is obtained.
Claims
1. A method for ultrasound image segmentation based on CNN and Transformer, characterized by: The following steps are involved: (1) Obtaining ultrasound images; The original image sequence acquisition method is as follows: a medical image X is obtained by a medical ultrasound imaging device, which has a spatial resolution of H×W and C channels; (2) The original ultrasound image is passed through the image Unet encoder to obtain the feature map; In the U-Net encoder, the encoder consists of multiple convolutional layers and pooling layers to extract image features. After the image is fed into the encoder, the image resolution is halved and the number of feature channels is doubled after each pooling layer. The output of the encoder is a low-resolution, high-channel feature map. (3) The original ultrasound image is passed through the image SwinT encoder to obtain the feature map for input image serialization. In the subsequent Swin-Transformer block, there are two modules, window self-attention and sliding window self-attention. Through these two modules, we can obtain: in with x l Represents the output of the l-th layer window self-attention module and the multi-layer perceptron module, with x l+1 is the output of the l+1 layer sliding window self-attention module and the multi-layer perceptron module, and LN is layer normalization. After four stages of stacking the SwinT module, the feature map of the ultrasound image is finally obtained. (4) Feature fusion is performed on the feature maps of the two channels in steps (2) and (3). After feature extraction by the Swin-T encoder and the U-Net encoder, features of different scales are obtained. The features of different modules at different levels are deeply fused through the GDFF module. The formula of the GDFF module is as follows: F conv =DC(TP(F swin-T )) (5) F fuse =G(Concat(F conv ,F U-net )) (6) Where DC represents two layers of convolution, batch normalization, and activation, TP represents the transposed convolution module, G represents the gating mechanism, and F swin-T With F U-net These are the features of different scales extracted by the Swin-T encoder and the U-Net encoder respectively; (5) The layer-by-layer features obtained in step (2) and the fused bottom-layer features obtained in step (4) are combined with the decoder and the features extracted by the corresponding layer of the U-Net encoder module for splicing and double-layer convolution, batch normalization and activation operations; after obtaining the output of this layer, the next layer decoder will again upsample the features of the previous layer with the Swin-T encoder, and splice and convolve with the U-Net encoder module of the corresponding layer again, and then obtain the result after two 3×3 convolutions.