A medical ultrasound image segmentation method

Abstract

The present invention relates to a medical ultrasound image segmentation method, which belongs to the technical field of electronic digital data processing, and comprises the following steps: S1: collecting relevant medical ultrasound data sets; S2: reading images in the medical ultrasound data sets, processing the images into the same size, and enhancing the training set by using operations such as random rotation and horizontal flipping; S3: building a medical ultrasound image segmentation model TMU‑NeXt based on deep learning, and selecting appropriate hyperparameters and training settings; S4: when the model is trained to the point where the loss no longer decreases, i.e., converges, the training is stopped and the model is saved; S5: using relevant indicators to compare and evaluate the sample effect of the model segmentation. The method can improve the segmentation index when performing medical ultrasound image segmentation, and has more accurate segmentation while having less computational complexity and inference time than other more advanced CNN‑Transformer hybrid framework methods.

Description

Technical Field

[0001] The invention belongs to the technical field of electronic digital data processing, and in particular relates to a medical ultrasonic image segmentation method. Background Art

[0002] Ultrasound (US) is a non-invasive imaging technology that can effectively evaluate and diagnose pathological lesions of various diseases and provide real-time visualization and analysis of patients' organs. Its low cost, strong portability, safety, and comfortable operation make it the first choice, especially for breast screening of women under 40 years old with dense breasts. Not only in breast examinations, but also in thyroid examinations, fetal condition assessments, liver examinations, etc., ultrasound can also be particularly effective in assisting doctors to make correct diagnoses and treatment plans. The ultrasound imaging methods currently used in hospitals for clinical use include B mode, shear wave elastography SWE (Shear-wave Elastography), color Doppler (Color-Doppler) and other multimodal modes.

[0003] In order to help doctors correctly interpret ultrasound images, researchers use automatic segmentation technology to automatically segment ultrasound images, improve the efficiency of reading films, and save labor costs. In early studies, researchers mostly used traditional segmentation methods. For example, Kekre and Shrinath proposed a vector quantization-based method to generate probability images based on the frequency of occurrence of pixel values ​​and use clustering algorithms to segment tumor areas; Huang et al. proposed a robust graph-based segmentation algorithm to improve segmentation performance. Although the segmentation accuracy has been improved to a certain extent, it is time-consuming and introduces too many human intervention factors.

[0004] With the development of deep learning (DL), convolutional neural networks (CNN) have shown excellent performance in automatic feature extraction and ultrasound segmentation in medical image analysis. The deep learning-based model frequently used in medical image processing is based on the U-Net architecture. The CNN model based on the UNet core architecture is one of the most typical breast tumor segmentation networks. The use of this framework has been proven to be advantageous because it integrates high-level features from the decoder and low-level features from the encoder. This integration helps to retrieve specific lesion details even in complex backgrounds. After the advent of the UNet structure, researchers have created various U-shaped networks based on it. Zhou et al. proposed UNet++, which narrowed the semantic gap between the encoder and decoder sub-networks by introducing a series of dense connections of convolutions, achieving better segmentation performance. This is an improvement on the restrictive skip connection in U-Net that only needs to fuse feature maps of the same scale. Later, Byra et al. were inspired by Li et al. and proposed SK-UNet, a convolutional neural network based on the selective kernel U-Net, for breast mass segmentation. However, the original selective kernel convolution ignores the feature selection under different receptive fields in the spatial dimension. Ning et al. proposed a saliency-guided morphology-aware U-shaped network SMUNet, which learns foreground saliency and background saliency representations respectively under the guidance of the prior information of the saliency map, and then provides contextual information contained in the background to assist foreground segmentation. The network framework that mainly relies on convolutional neural networks (CNNs) has inherent limitations in establishing the interaction of global and long-range semantic information, and suffers from serious false detections and missed detections. This is because due to the locality of the convolution operation, CNNs tend to focus on local information, which may be detrimental to the segmentation of ultrasound images, because the lesion location information in ultrasound images is often affected by complex background grayscale and contrast (e.g., heterogeneous echoes).

[0005] In recent years, attention learning has become a popular method for focusing on important features in deep learning. Hu et al. proposed the squeeze excitation (SE) block as an adaptive method to recalibrate channel feature responses. Yu et al. proposed a channel attention method for semantic segmentation, which solves the class inconsistency problem by selecting discriminative features of channels. Zhong et al. developed the squeeze-and-attention network (SANet), which aims to enhance the prediction of single pixels and the spatial attention of pixel groups. Woo et al. constructed a convolutional block attention module (CBAM) to integrate channel and spatial attention, and achieve adaptive emphasis or suppression of feature maps to obtain refined intermediate features. Many attention-based models have achieved excellent results in ultrasound (US) segmentation. Chen et al. designed a hybrid adaptive module in the proposed AAU-Net to aggregate information from multiple kernels to achieve adaptive receptive field size of neurons. The hybrid adaptive module can adaptively select receptive fields of different scales from channel and spatial dimensions. Inspired by ConvMixer, Tang et al. used large convolution kernels to mix remote spatial positions to obtain global context information and proposed an efficient fully convolutional image segmentation network CMUNet, which includes a ConvMixer module and a multi-scale attention gate. The ConvMixer module is used to extract global context information, and the multi-scale attention gate suppresses irrelevant features and enhances valuable features.

[0006] Transformer has been shown to be a promising technique in natural language processing (NLP), which can capture long-range dependencies through the self-attention mechanism between input patches and is able to extract global contextual information. It has also been increasingly used in medical image segmentation and has achieved great success. Dosovitskiy et al. introduced a transformer-based image recognition model, Vision transformer (ViT), which replaced all layers of CNN with transformer blocks and surpassed CNN for the first time. Liu et al. proposed a Swin transformer with a shifted window strategy based on Vit, which exploited the idea of ​​convolutional sliding windows and significantly reduced the computational parameters of multi-head self-attention (MHSA). The Swin transformer has demonstrated its applicability in various computer vision tasks. However, using transformers for medical image segmentation will introduce many new challenges. First, it requires a wider training dataset for transformers to perform better; in addition, high resource requirements for computing and storage are necessary; finally, it is limited in capturing local feature information.

[0007] To address these issues, a CNN-transformer hybrid framework was proposed to effectively utilize global and local information by replacing convolutional layers or superimposing features extracted by both. Cao et al. proposed Swin-UNet, which used Swin transformer blocks as encoders and decoders of the U-Net architecture and achieved excellent performance in two-dimensional medical image segmentation. Chen et al. proposed TransUNet, which combined a CNN-transformer hybrid encoder to extract local and global features, exceeding the performance achieved by using each component alone. Bi et al. proposed BPAT-Unet based on transformer and UNet for ultrasound thyroid nodule segmentation

[57] . The network used a novel self-attention pooling method to design a bounded point supervision module to enhance boundary features, and placed the transformer at the bottleneck of the network to fully integrate high-frequency local and low-frequency global features. However, it should be noted that most of the work on transformer-based networks focuses on improving network performance, without paying attention to computational complexity and inference time, which are essential in real-world applications.

[0008] Therefore, at this stage, it is necessary to design a medical ultrasound image segmentation method to solve the above problems. Summary of the invention

[0009] The purpose of the present invention is to provide a medical ultrasound image segmentation method for solving the technical problems existing in the above-mentioned prior art, which can improve the segmentation index when performing medical ultrasound image segmentation, achieve more accurate segmentation and have less computational complexity and inference time than other more advanced CNN-Transformer hybrid framework methods.

[0010] To achieve the above object, the technical solution of the present invention is:

[0011] A medical ultrasound image segmentation method comprises the following steps:

[0012] S1: Collect relevant medical ultrasound datasets;

[0013] S2: Read images from the medical ultrasound dataset, process the images to the same size, and enhance the training set using operations such as random rotation and horizontal flipping;

[0014] S3: Build a deep learning-based medical ultrasound image segmentation model TMU-NeXt and select appropriate hyperparameters and training settings;

[0015] S4: When the model training reaches the point where the loss no longer decreases, that is, convergence, stop training and save the model;

[0016] S5: Use relevant indicators to evaluate the sample segmentation effect of the model.

[0017] Furthermore, building a medical ultrasound segmentation model in step S3 includes the following steps:

[0018] S31: The backbone network adopts a U-shaped network, including five encoding layers, five decoding layers and an intermediate module; the data for training in the network is set to be an image matrix with a dimension of channel × height × width, where channel × height × width is recorded as C × H × W. First, the encoding layer part. The input of the first encoding layer is a 3 × H × W image pixel normalized matrix, and the output is not only the input of the second encoding layer, but also the first input of the intermediate module; the outputs of the second, third, and fourth encoding layers are the inputs of the third, fourth, and fifth encoding layers and the second, third, and fourth inputs of the intermediate module respectively; and the decoding layer corresponds to the encoding layer, the input of the first decoding layer is the output of the fifth encoding layer, and the outputs of the first, second, third, and fourth decoding layers are added to the fourth, third, second, and first outputs of the intermediate module respectively as the inputs of the second, third, fourth, and fifth decoding layers; finally, the output of the fifth decoding layer is a weight matrix with a channel of 1 and the same size as the input, i.e., H × W;

[0019] S32: In step S31, the first, second and third encoding layers include sequentially cascaded convolutional layers, batch normalization layers, non-parameter attention layers, activation layers and maximum pooling layers; the third and fourth decoding layers include sequentially cascaded convolutional layers, batch normalization layers, interpolation layers, activation layers and non-parameter attention layers, wherein the interpolation layers use bilinear interpolation upsampling; the fifth decoding layer includes cascaded convolution blocks with a convolution kernel of 1×1 and interpolation layers, wherein the 1×1 convolution block changes the input channel to 1, and the interpolation layer restores the image size to the same as the first encoding layer through bilinear interpolation; the fourth and fifth encoding layers include not only the same layers as the first, second and third encoding layers, but also cascaded labeled multi-layer perceptron layers, and the first and second decoder layers include not only the same decoding layers as the third and fourth decoding layers, but also cascaded labeled multi-layer perceptron layers.

[0020] Furthermore, the middle part in step S31 is composed of four cascaded channel cross-fusion attention modules; the channel cross-fusion attention module includes a multi-scale feature embedding layer, a multi-head cross-attention layer, a normalization layer and a multi-layer perceptron layer; the four inputs L1, L2, L3, and L4 are in the multi-scale feature embedding layer, where , , reshape the features into a sequence of unfolded plane image blocks of sizes P, P / 2, P / 4, and P / 8 for labeling, so that the image blocks are mapped to the same area at these four scales; after passing through the normalization layer, the labels of these four layers Li are spliced ​​as the key vector key and the value vector value, and then input into the multi-head cross attention layer together with the normalized query vector query; in the multi-head cross attention layer, the query vector, key vector, and value vector generate a similarity matrix Mi, the value vector is weighted by the similarity matrix, the sum of the exponential power of the head number is calculated and the average is taken, and the residual connection is used as the output; the output of the previous layer passes through the subsequent normalization layer and multi-layer perceptron layer and performs residual connection, and the final output result is input into the next channel cross fusion attention module.

[0021] Furthermore, in step S32, the labeled multilayer perceptron layer includes a movable multilayer perceptron layer, a deep convolutional layer, an activation layer, a normalization layer, and a residual connection; in the movable multilayer perceptron layer, the feature is moved and projected onto the tag, the tag first changes the channel, and then the tag passes through a movable multilayer perceptron of moving width or height, wherein the hidden dimension of the multilayer perceptron is a hyperparameter; in the movable multilayer perceptron layer, the channel axis of the convolutional feature is first moved so that the multilayer perceptron only focuses on the preset position of the convolutional feature; there are two movable multilayer perceptrons in the labeled multilayer perceptron layer, one moves across width and the other moves across height.

[0022] Furthermore, in step S4, the loss function Loss adopts the combination of Dice coefficient and binary cross entropy BCE, and the proportional coefficient α ranges from 0 to 1. is the predicted value, is the target value;

[0023] .

[0024] Furthermore, in step S5, the index ratio includes the dice coefficient and the Jaccard index.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] An innovative CNN-Transformer hybrid framework network that integrates multi-layer perceptron and channel cross-fusion attention into a U-shaped network is proposed. It can improve the segmentation index when performing medical ultrasound image segmentation, achieve more accurate segmentation, and have less computational complexity and inference time than other more advanced CNN-Transformer hybrid framework methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a schematic diagram of the local structure of the segmentation model described in S32 in the implementation method of the present invention.

[0028] Figure 2 This is a visual comparison diagram of the segmentation results obtained by the implementation method of the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention, that is, the embodiments described are only part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0030] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative work are within the scope of protection of the present invention. It should be noted that relational terms such as the terms "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0031] Moreover, the terms "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus that includes a list of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or apparatus. In the absence of more restrictions, an element defined by the phrase "comprises a ..." does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0032] Embodiment 1:

[0033] A method for segmenting a medical ultrasound image comprises the following steps:

[0034] S1: Collect datasets;

[0035] Collect relevant medical ultrasound datasets.

[0036] S2: Image preprocessing;

[0037] Read images from the ultrasound dataset, process the images to the same size, and use operations such as random rotation and horizontal flipping to enhance the training set to ensure the model training effect.

[0038] S3: Model training;

[0039] Build the deep learning-based medical ultrasound image segmentation model TMU-NeXt and select appropriate hyperparameters and training settings.

[0040] S4: Model saving;

[0041] When the model training reaches the point where the loss no longer decreases, that is, converges, stop training and save the model. The loss function Loss uses a combination of the Dice coefficient and the binary cross entropy. The proportional coefficient α ranges from 0 to 1. In this embodiment, it takes 0.5. is the predicted value, is the target value.

[0042]

[0043] S5: Performance evaluation;

[0044] Using relevant indicators such as Dice Coefficient, Jaccard index ( IoU ) is used to evaluate the sample effect of model segmentation. The specific meaning is shown in Table 1.

[0045]

[0046]

[0047]

[0048] Table 1

[0049] Furthermore, the medical ultrasound segmentation model built in S3 includes the following processes:

[0050] S31: The backbone network uses a U-shaped network, including five encoding layers, five decoding layers and an intermediate module. The data for training in the network is set to be an image matrix with a dimension of channel × height × width (C×H×W). First, the encoding layer part. The input of the first encoding layer is a 3×H×W image pixel normalization matrix. The output is not only the input of the second encoding layer, but also the first input of the intermediate module; and so on. The outputs of the second, third, and fourth encoding layers are the inputs of the third, fourth, and fifth encoding layers and the second, third, and fourth inputs of the intermediate module. The decoding layer corresponds to the encoding layer. The input of the first decoding layer is the output of the fifth encoding layer. The outputs of the first, second, third, and fourth decoding layers are added to the fourth, third, second, and first outputs of the intermediate module as the inputs of the second, third, fourth, and fifth decoding layers; finally, the output of the fifth decoding layer is a weight matrix with a channel of 1 and the same size as the input, that is, H×W.

[0051] S32: Figure 1In S31, the first, second, and third encoding layers contain sequentially cascaded convolutional layers, batch normalization layers, non-parameterized attention layers, activation layers, and maximum pooling layers. After these layers, the input channels increase and the image size decreases. Similarly, the third and fourth decoding layers contain sequentially cascaded convolutional layers, batch normalization layers, interpolation layers, activation layers, and non-parameterized attention layers. The interpolation layers use bilinear interpolation upsampling. Using bilinear interpolation instead of the commonly used transposed convolution can reduce the parameters. After these layers, the input channels increase. The image size is reduced and doubled; the fifth decoding layer contains cascaded convolution blocks with a convolution kernel of 1×1 and an interpolation layer, wherein the 1×1 convolution block makes the input channel become 1, and the interpolation layer restores the image size to the same as the first encoding layer through bilinear interpolation; the fourth and fifth encoding layers contain the same layers as the first, second, and third encoding layers, and are also cascaded with a labeled multilayer perceptron layer. Similarly, the first and second decoder layers contain the same decoding layers as the third and fourth decoding layers, and are also cascaded with a labeled multilayer perceptron layer.

[0052] The middle part of S31 consists of four cascaded channel cross-fusion attention modules. The channel cross-fusion attention module structure includes a multi-scale feature embedding layer, a multi-head cross-attention layer, a normalization layer, and a multi-layer perceptron layer. The four inputs L1, L2, L3, L4 In the multi-scale feature embedding layer, the features are reshaped into a sequence of unfolded plane image blocks of sizes P, P / 2, P / 4, and P / 8 for labeling, so that the image blocks can be mapped to the same area at these four scales. This process maintains the original channel dimension; after passing through the normalization layer, the labels of these four layers Li (i∈1,2,3,4) are concatenated as the key vector key and the value vector value, and then input into the multi-head cross attention layer together with the normalized query vector query; in the multi-head cross attention layer, the query vector, key vector, and value vector generate a similarity matrix Mi, the value vector is weighted by the similarity matrix, the sum of the exponential power of the head number is averaged, and the residual connection is used as the output; the output of the previous layer passes through the subsequent normalization layer and multi-layer perceptron layer and performs residual connection, and the final output result is input into the next channel cross fusion attention module.

[0053] The tokenized multilayer perceptron layer in S32 includes a movable multilayer perceptron layer, a deep convolution layer, an activation layer, a normalization layer, and a residual connection. In the movable multilayer perceptron layer, the features are moved and projected onto the tags. The tags first change the channel, and then these tags are passed through the movable multilayer perceptron of the moving width or height, where the hidden dimension of the multilayer perceptron is a hyperparameter; the deep convolution layer uses deep convolution to help encode the location information of the multilayer perceptron features, and can also improve efficiency with fewer parameters than UNet. In the aforementioned movable multilayer perceptron layer, the channel axis of the convolution feature is first moved so that the multilayer perceptron only focuses on the preset position of the convolution feature, guiding this layer to be more local; there are two movable multilayer perceptrons in the tokenized multilayer perceptron layer, one moves the features across the width, and the other moves across the height, just like the axial attention mechanism, dividing the features into multiple different areas, and then moving them to a certain position according to the specified axis, which helps to create a random window that guides the local along the axis.

[0054] Embodiment 2:

[0055] The method in Example 1 is used to perform breast tumor ultrasound image segmentation on the public breast ultrasound image segmentation datasets BUSI and UDIAT, and the segmentation categories are lesions and normal tissues. The computer environment of this experiment: the operating system is Windows 11; an NVIDIA RTX3060 GPU, and the software platform is python 3.9 and Pytorch 1.12. This example randomly selects 70% from the public data set as the training set, 20% as the validation set, and the remaining 10% as the test set. The experiment uses IoU and Dice as evaluation criteria. As shown in Table 2, the IoU and Dice coefficients of this method in the ultrasound segmentation task reached 0.7604 and 0.8513 respectively, and the accuracy is higher than that of the more advanced methods. The test set image segmentation effect is shown in the figure. Figure 2 As shown, the present invention can better locate and segment clearer and more accurate boundaries.

[0056]

[0057] Table 2

[0058] The above are preferred embodiments of the present invention. Any changes made according to the technical solution of the present invention, as long as the resulting functions do not exceed the scope of the technical solution of the present invention, belong to the protection scope of the present invention.

Claims

1. A medical ultrasound image segmentation method, characterized in that: The following steps are involved: S1: Collect relevant medical ultrasound datasets; S2: Read images from the medical ultrasound dataset, process the images to the same size, and use random rotation and horizontal flipping operations to enhance the training set; S3: Build the deep learning-based medical ultrasound image segmentation model TMU-NeXt, select hyperparameters and training settings; S4: When the model training reaches the point where the loss no longer decreases, that is, convergence, stop training and save the model; S5: Use relevant indicators to evaluate the sample segmentation effect of the model; In step S3, a medical ultrasound segmentation model is constructed, including the following steps: S31: The backbone network adopts a U-shaped network, including five encoding layers, five decoding layers and an intermediate module; the data for training in the network is set to be an image matrix with a dimension of channel × height × width, where channel × height × width is recorded as C × H × W. First, the encoding layer part. The input of the first encoding layer is a 3 × H × W image pixel normalized matrix, and the output is not only the input of the second encoding layer, but also the first input of the intermediate module; the outputs of the second, third, and fourth encoding layers are the inputs of the third, fourth, and fifth encoding layers and the second, third, and fourth inputs of the intermediate module respectively; and the decoding layer corresponds to the encoding layer, the input of the first decoding layer is the output of the fifth encoding layer, and the outputs of the first, second, third, and fourth decoding layers are added to the fourth, third, second, and first outputs of the intermediate module respectively as the inputs of the second, third, fourth, and fifth decoding layers; finally, the output of the fifth decoding layer is a weight matrix with a channel of 1 and the same size as the input, i.e., H × W; S32: The first, second and third encoding layers in step S31 include sequentially cascaded convolutional layers, batch normalization layers, non-parameter attention layers, activation layers and maximum pooling layers; the third and fourth decoding layers include sequentially cascaded convolutional layers, batch normalization layers, interpolation layers, activation layers and non-parameter attention layers, wherein the interpolation layers use bilinear interpolation upsampling; the fifth decoding layer includes cascaded convolution blocks with a convolution kernel of 1×1 and interpolation layers, wherein the 1×1 convolution block changes the input channel to 1, and the interpolation layer restores the image size to the same as the first encoding layer through bilinear interpolation; the fourth and fifth encoding layers include the same layers as the first, second and third encoding layers, and also cascaded labeled multi-layer perceptron layers, and the first and second decoder layers include the same decoding layers as the third and fourth decoding layers, and also cascaded labeled multi-layer perceptron layers; The middle part of step S31 is composed of four cascaded channel cross-fusion attention modules; the channel cross-fusion attention module includes a multi-scale feature embedding layer, a multi-head cross-attention layer, a normalization layer and a multi-layer perceptron layer; the four inputs L1, L2, L3, and L4 are in the multi-scale feature embedding layer, where , , reshape the features into a sequence of unfolded plane image blocks of sizes P, P / 2, P / 4, and P / 8 for labeling, so that the image blocks are mapped to the same area at these four scales; after passing through the normalization layer, the labels of these four layers Li are spliced ​​as the key vector key and the value vector value, and then input into the multi-head cross attention layer together with the normalized query vector query; in the multi-head cross attention layer, the query vector, key vector, and value vector generate a similarity matrix Mi, the value vector is weighted by the similarity matrix, the sum of the exponential power of the head number is calculated and the average is taken, and the residual connection is used as the output; the output of the previous layer passes through the subsequent normalization layer and multi-layer perceptron layer and performs residual connection, and the final output result is input into the next channel cross fusion attention module.

2. A medical ultrasound image segmentation method according to claim 1, characterized in that: In step S32, the labeled multilayer perceptron layer includes a movable multilayer perceptron layer, a deep convolutional layer, an activation layer, a normalization layer, and a residual connection; in the movable multilayer perceptron layer, the feature is moved and projected onto the tag, the tag first changes the channel, and then the tag passes through a movable multilayer perceptron of moving width or height, wherein the hidden dimension of the multilayer perceptron is a hyperparameter; in the movable multilayer perceptron layer, the channel axis of the convolutional feature is first moved so that the multilayer perceptron only focuses on the preset position of the convolutional feature; there are two movable multilayer perceptrons in the labeled multilayer perceptron layer, one moves across width and the other moves across height.

3. The medical ultrasound image segmentation method according to claim 1, characterized in that: In step S4, the loss function Loss uses a combination of the Dice coefficient and the binary cross entropy BCE, and the proportional coefficient α ranges from 0 to 1. is the predicted value, is the target value; 。 4. The medical ultrasound image segmentation method according to claim 1, characterized in that: In step S5, the index ratio includes the dice coefficient and the Jaccard index.

Images