A small object medical ultrasound image segmentation method based on coarse and fine granularity attention

Abstract

The present invention discloses a small target medical ultrasound image segmentation method based on coarse-grained and fine-grained attention, comprising: S1, collecting medical ultrasound images and performing data preprocessing as a data set; S2, using the data set to train a multi-scale coarse-grained-fine-grained channel attention U-Net model to obtain a trained multi-scale coarse-grained-fine-grained channel attention U-Net model; S3, inputting the medical ultrasound image to be segmented into the trained multi-scale coarse-grained-fine-grained channel attention U-Net model to obtain a small target segmentation result. The present application improves the current segmentation accuracy of small target ultrasound images, uses a gating signal and a multi-scale convolution to expand the feature scale in a jump connection, and then uses an SE module and a multi-scale channel attention module to perform coarse-grained and fine-grained channel processing, and provides a multi-scale feature fusion weighting solution.

Description

Technical Field

[0001] The present application relates to the technical field of target image processing, and in particular to a small target medical ultrasound image segmentation method based on coarse and fine granularity attention. Background Art

[0002] Deep learning is a subset of machine learning that is based on the study of artificial neural networks, especially those with multi-layer structures, also known as deep neural networks (DNNs). Deep learning models are able to learn complex patterns and high-level features of data, which makes them very effective when processing images, sounds, text, and other complex data types. In the field of medical image segmentation, deep learning techniques have become mainstream because they are able to provide higher segmentation accuracy than traditional methods. Medical image segmentation is crucial for disease diagnosis and treatment planning, as it can identify and locate organs or lesions in images. Deep learning models are able to learn complex patterns from large amounts of training data, thereby achieving accurate segmentation of regions of interest in medical images. Medical ultrasound images are a common type of segmented image in the field of medical image segmentation. Medical ultrasound images, also known as ultrasound imaging or ultrasound scanning, are a medical diagnostic technology that uses high-frequency sound waves to generate images of internal structures of the body.

[0003] At present, the difficulty of medical ultrasound image segmentation is the segmentation accuracy of small targets. Medical ultrasound images are different from general images. General images are rich in low-level features (such as texture, color, shape), and the high-level features composed of low-level features are obvious. However, the low-level features of medical ultrasound images are relatively not rich. For example, ultrasound images of diseases such as carotid plaques, breast tumors, and anorectal fistulas have a single color tone, small targets, and unclear texture differences, resulting in low efficiency of manual segmentation and low accuracy of segmentation technology based on deep learning.

[0004] The existing technology for medical image analysis problems is mostly based on the encoder-decoder structure network U-Net, which improves accuracy by proposing various attention mechanisms in the jump connection. The jump connection helps the network obtain high-resolution semantic features by bridging the finer detail features of the lower layer and the coarse-resolution high-level semantic features. However, in the existing technology, it is not taken into account that when the convolutional neural network is used as a feature extraction tool for downsampling, some information of small targets is easily lost in the convolution calculation. Therefore, the feature extraction in the jump connection is insufficient, and the fusion method of multi-scale features is relatively rough, resulting in insufficient low-level features captured in the learning process of small target ultrasound image segmentation, and many shapes are not well-known. The details of irregular images, unclear boundaries, and small targets are not learned. At the same time, although the current channel attention mechanism has the function of filtering noisy channels, some channel attention mechanisms can retain local information to a certain extent while filtering noisy channels, thereby limitedly improving the detection or segmentation accuracy of small target images. However, when faced with multi-scale features of small targets, either too much emphasis is placed on filtering noisy channels, resulting in some important information of some small target features being filtered out, or too much emphasis is placed on capturing local information, resulting in incomplete filtering of noisy channels. It is difficult to simultaneously meet the needs of capturing local information and filtering noisy channels. Therefore, for small target ultrasound images, the effect of any current attention mechanism is limited. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the present invention provides a small target medical ultrasound image segmentation method based on coarse and fine granularity attention, which solves the problems of insufficient low-level features learned in the convolutional neural network learning process and inefficient feature fusion in jump connections.

[0006] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is: a small target medical ultrasound image segmentation method based on coarse and fine granular attention, which solves the problem that in the process of convolutional neural network learning, the small target features are lost due to convolution calculation, resulting in insufficient learned low-level features and inefficient feature fusion in jump connection, including:

[0007] S1. Collect medical ultrasound images and perform data preprocessing as a data set;

[0008] S2. Use the data set to train a multi-scale coarse-grained-fine-grained channel attention U-Net model to obtain a trained multi-scale coarse-grained-fine-grained channel attention U-Net model;

[0009] S3. Input the medical ultrasound image to be segmented into the trained multi-scale coarse-grained-fine-grained channel attention U-Net model to obtain the small target segmentation result.

[0010] Further: S2 includes:

[0011] S21. Input the data set into the multi-scale coarse-grained-fine-grained channel attention U-Net model. Through forward propagation, starting from the input of the model, the output of each neuron is calculated layer by layer to obtain the predicted value of the model.

[0012] S22. According to the predicted value of the model, the error between the classification probability map and the label is calculated by the composite loss function, and the error between the classification probability map and the label is back-propagated through the gradient to obtain the weight parameter of the network model;

[0013] S23. Through the early stopping mechanism, the model parameters with the highest Dice evaluation value during the training process are saved to obtain the trained multi-scale coarse-grained-fine-grained channel attention U-Net model.

[0014] Further: In S22, the expression of the composite loss function Loss(p,t) is: 其中，wpos和wneg分别是正类和负类的权重，pi和ti是每个像素位置的预测值和目标值，i表示在每个图像中遍历的每个像素位置的索引，smooth为防止除零情况的非零极小 数。

[0015] Further: S3 includes:

[0016] S31, inputting the medical ultrasound image to be segmented into the trained multi-scale coarse-grained-fine-grained channel attention U-Net model;

[0017] S32, gradually extracting features of the input image through the encoder to obtain a high-level feature map;

[0018] Among them, as the encoder level goes deeper, the size of the high-level feature map gradually decreases, and the number of feature channels gradually increases;

[0019] S33, performing a step-by-step upsampling operation on the high-level feature map through a decoder with the same number of layers as the encoder, restoring the high-level feature map to the size of the original input image, and obtaining a test image;

[0020] S34. Convolve the test image, fuse the test image into 1 channel, and obtain a binary image through the Sigmoid function as the small target segmentation result.

[0021] Further: S32 includes:

[0022] S321, performing convolution and pooling operations on the input image in sequence through the encoder;

[0023] S322, performing convolution and pooling operations on the convolved and pooled image through an encoder to obtain a feature map;

[0024] S323, repeatedly execute S322 until the predetermined number of convolutions and poolings is reached to complete the downsampling, and perform a convolution operation on the downsampled features as a high-level feature map.

[0025] Further: S33 includes:

[0026] S331, performing an upsampling operation on the high-level feature map through a decoder to obtain an upsampled image;

[0027] S332, input the upsampled image as a gated signal into the multi-scale coarse-grained-fine-grained channel attention module, input the feature map extracted by the corresponding level encoder as an input feature into the multi-scale coarse-grained-fine-grained channel attention module, perform adaptive weight feature fusion, and obtain a fused weighted feature map;

[0028] S333, sequentially concatenating and fusing the fused weighted feature map and the upsampled image input decoder to obtain a feature fusion map;

[0029] S334, performing an upsampling operation on the feature fusion image through a decoder to obtain a new upsampled image;

[0030] S335. Repeat S332-S334 until the new up-sampled image is restored to the size of the original input image, thereby obtaining a test image.

[0031] Further: In S332, the relationship between the decoder and the encoder of the corresponding level is:

[0032] k+j=n

[0033] Where k is the number of decoder layers, j is the corresponding number of encoder layers, and n is the total number of encoder or decoder layers.

[0034] Further: In S332, the method for processing the gating signal and the incoming features by the multi-scale coarse-grained-fine-grained channel attention module includes:

[0035] S3321, performing 3*3 convolution and 5*5 convolution on the gate signal g respectively to obtain gate signal g3 and gate signal g5;

[0036] S3322, perform 3*3 convolution and 5*5 convolution on the input feature x respectively to obtain the input feature x3 and the input feature x5;

[0037] S3323, concatenate the gated signal g3, the gated signal g5, the incoming feature x3, and the incoming feature x5 to obtain a feature group cat;

[0038] S3324, input the feature group cat into the SE module to obtain an SE weighted feature map;

[0039] S3325, perform a 3*3 convolution operation on the SE weighted feature map to obtain a feature group catse;

[0040] S3326, input the feature group catse into the multi-scale channel attention module for local and global attention processing to obtain the weight s;

[0041] S3327. According to the weight s, a fused weighted feature map z is obtained, and its expression is:

[0042] z=x3*s+x5*(1-s)

[0043] Among them, x3 represents the incoming feature x3, and x5 represents the incoming feature x5.

[0044] Further: In S3326, the expression of weight s is:

[0045]

[0046] Among them, σ() is the Sigmoid activation function, ⊙ represents element-by-element multiplication, Conv1×1 represents a 1*1 convolution operation, ReLU() is the ReLU function, AvgPool() represents the average pooling operation, and BN() represents the batch normalization layer.

[0047] The beneficial effects of the present invention are:

[0048] 1. Extract multi-scale features through convolution operation with multi-scale convolution kernels through gated signal mechanism, thereby compensating for the information lost in convolution calculation of small target objects;

[0049] 2. Through two channel attention modules (SE module and multi-scale channel attention module), the extracted multi-scale features are weighted from coarse-grained to fine-grained. The noisy channels that lose information are filtered out through coarse-grained processing to obtain high-quality features, and then local information and global information are captured simultaneously through fine-grained processing. This solves the problem of small target features being lost due to convolution calculations during convolutional neural network learning, resulting in insufficient learned low-level features and inefficient feature fusion in jump connections. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 Flowchart of the small target medical ultrasound image segmentation method based on coarse and fine grained attention.

[0051] Figure 2 Schematic diagram of the multi-scale coarse-grained-fine-grained channel attention U-Net model.

[0052] Figure 3 Schematic diagram of the multi-scale coarse-grained-fine-grained channel attention module.

[0053] Figure 4 Schematic diagram of SE module.

[0054] Figure 5 Schematic diagram of the multi-scale channel attention module. DETAILED DESCRIPTION

[0055] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0056] like Figure 1 As shown, in one embodiment of the present invention, a small target medical ultrasound image segmentation method based on coarse and fine granularity attention is provided, comprising:

[0057] S1. Collect medical ultrasound images and perform data preprocessing as a data set;

[0058] S2. Use the data set to train a multi-scale coarse-grained-fine-grained channel attention U-Net model to obtain a trained multi-scale coarse-grained-fine-grained channel attention U-Net model;

[0059] S3. Input the medical ultrasound image to be segmented into the trained multi-scale coarse-grained-fine-grained channel attention U-Net model to obtain the small target segmentation result.

[0060] Optionally, after S1, additional steps may be performed:

[0061] S11. Divide the small object medical ultrasound image dataset into training set, validation set and test set.

[0062] S12. Normalize the training set and validation set, perform random rotation and flipping, and scale them to a size of 224*224.

[0063] Specifically, in this embodiment, S2 includes:

[0064] S21. Input the data set into the multi-scale coarse-grained-fine-grained channel attention U-Net model. Through forward propagation, starting from the input of the model, the output of each neuron is calculated layer by layer to obtain the predicted value of the model.

[0065] S22. According to the predicted value of the model, the error between the classification probability map and the label is calculated by the composite loss function, and the error between the classification probability map and the label is back-propagated through the gradient to obtain the weight parameter of the network model;

[0066] S23, through the early stopping mechanism, save the model parameters that obtain the highest Dice evaluation value during the training process, and obtain the trained multi-scale coarse-grained-fine-grained channel attention U-Net model;

[0067] If the additional steps after S1 are adopted, the model parameters that obtain the highest Dice evaluation value on the validation set during the training process are saved, and the trained multi-scale coarse-grained-fine-grained channel attention U-Net model can be obtained.

[0068] Specifically, the Dice evaluation value is a similarity index used to evaluate image segmentation models. It is mainly used to measure the degree of overlap between the segmentation area predicted by the model and the true segmentation area (ground truth). The Dice coefficient ranges from 0 to 1, where 1 means that the predicted segmentation is completely consistent with the true segmentation, and 0 means there is no overlap. The Dice formula is:

[0069] Dice = 2 ∙ TP ÷ (2 ∙ TP + FP + FN)

[0070] Among them, TP is true positive, FP is false positive, and FN is false negative, all of which are related to the binary segmentation of the test image.

[0071] In S22, the expression of the composite loss function Loss(p,t) is:

[0072]

[0073] Among them, wpos and wneg are the weights of the positive and negative classes respectively, pi and ti are the predicted value and target value of each pixel position, i represents the index of each pixel position traversed in each image, and smooth is a non-zero minimum number to prevent division by zero.

[0074] like Figure 2 As shown, in this embodiment, a multi-scale coarse-grained-fine-grained channel attention U-Net model with 4 encoder and decoder layers is provided. Taking this model as an example, S3 includes:

[0075] S31, inputting the medical ultrasound image to be segmented into the trained multi-scale coarse-grained-fine-grained channel attention U-Net model;

[0076] S32, gradually extracting features of the input image through the encoder to obtain a high-level feature map;

[0077] Among them, as the encoder level goes deeper, the size of the high-level feature map gradually decreases, and the number of feature channels gradually increases;

[0078] S33, performing a step-by-step upsampling operation on the high-level feature map through a decoder with the same number of layers as the encoder, restoring the high-level feature map to the size of the original input image, and obtaining a test image;

[0079] S34. Convolve the test image, fuse the test image into 1 channel, and obtain a binary image through the Sigmoid function as the small target segmentation result.

[0080] This application does not limit the specific number of layers of the encoder and decoder, and the number of layers depends on the specific required accuracy.

[0081] Specifically, S32 includes:

[0082] S321, performing convolution and pooling operations on the input image in sequence through the encoder;

[0083] S322, performing convolution and pooling operations on the convolved and pooled image through an encoder to obtain a feature map;

[0084] S323, repeatedly execute S322 until the predetermined number of convolutions and poolings is reached to complete the downsampling, and perform a convolution operation on the downsampled features as a high-level feature map.

[0085] Specifically, S33 includes:

[0086] S331, performing an upsampling operation on the high-level feature map through a decoder to obtain an upsampled image;

[0087] S332, input the upsampled image as a gated signal into the multi-scale coarse-grained-fine-grained channel attention module, input the feature map extracted by the corresponding level encoder as an input feature into the multi-scale coarse-grained-fine-grained channel attention module, perform adaptive weight feature fusion, and obtain a fused weighted feature map;

[0088] The relationship between the decoder and the encoder at the corresponding level is:

[0089] k+j=n

[0090] Where k is the number of decoder layers, j is the corresponding number of encoder layers, and n is the total number of encoder or decoder layers;

[0091] S333, sequentially concatenating and fusing the fused weighted feature map and the upsampled image input decoder to obtain a feature fusion map;

[0092] S334, performing an upsampling operation on the feature fusion image through a decoder to obtain a new upsampled image;

[0093] S335. Repeat S332-S334 until the new up-sampled image is restored to the size of the original input image, thereby obtaining a test image.

[0094] Specifically, Figure 3 As shown, in S332, the method for processing the gating signal and the incoming features by the multi-scale coarse-grained-fine-grained channel attention module includes:

[0095] S3321, performing 3*3 convolution and 5*5 convolution on the gate signal g respectively to obtain gate signal g3 and gate signal g5;

[0096] S3322, perform 3*3 convolution and 5*5 convolution on the input feature x respectively to obtain the input feature x3 and the input feature x5;

[0097] S3323, concatenate the gated signal g3, the gated signal g5, the incoming feature x3, and the incoming feature x5 to obtain a feature group cat;

[0098] S3324, input the feature group cat into the SE module to obtain an SE weighted feature map;

[0099] S3325, perform a 3*3 convolution operation on the SE weighted feature map to obtain a feature group catse;

[0100] S3326, input the feature group catse into the multi-scale channel attention module for local and global attention processing to obtain the weight s;

[0101] The expression of weight s is:

[0102]

[0103] Among them, σ() is the Sigmoid activation function, ⊙ represents element-by-element multiplication, Conv1×1 represents a 1*1 convolution operation, ReLU() is the ReLU function, BN() represents a batch normalization layer, and AvgPool() represents an average pooling operation;

[0104] S3327. According to the weight s, a fused weighted feature map z is obtained, and its expression is:

[0105] z=x3*s+x5*(1-s)

[0106] Among them, x3 represents the incoming feature x3, and x5 represents the incoming feature x5.

[0107] In this embodiment, the SE module and the multi-scale channel attention module are respectively as follows: Figure 4 and Figure 5As shown in the figure, the SE module compresses features along the spatial dimension through global average pooling, converts each two-dimensional feature channel into a real number, and captures the global distribution of the response on the feature channel to complete the coarse-grained processing; because the size of small targets is small and the ultrasound image itself is not rich enough, it is easy for small targets to lose low-level features during the convolution process, that is, noise occurs, and the noise part is different under different convolution calculations. In the case of single-scale features, although the SE module can filter out this part of the noise channel from the perspective of global distribution, it is also easy to ignore some important low-level features in the process of global average pooling. This method is not effective for small targets with insufficient feature richness. However, in the case of multi-scale features, different scale features can make up for the information lost in the convolution calculation, but it will also increase the number of noise channels. At this time, the SE module can make better use of the efficient filtering ability of the noise channel. Although some important features may still be filtered out, this part of the loss is acceptable because the number of features of multi-scale features is rich enough.

[0108] The coarse-grained features are fused through convolution and then processed in a multi-scale channel attention module for fine-grained processing. The incoming features are converted into a C*1*1 (C*W*H, where C is the number of channels, W is the width, and H is the height) feature map through global average pooling, and then the global attention of C*1*1 is finally obtained through point convolution to capture the global context information. At the same time, the local attention of C*W*H is obtained through direct point convolution to obtain the local context information. The global context is added to the local context through the operation of step-by-step element addition to reduce the loss of context information. Compared with the SE module, the multi-scale channel attention module pays more attention to local information, captures the low-level features of small targets more efficiently, and weights the channels more carefully and efficiently. The low-level features in the segmentation of small target ultrasound images are more important, so the segmentation effect is better.

[0109] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A small target medical ultrasound image segmentation method based on coarse and fine granularity attention, characterized in that: include: S1. Collect medical ultrasound images and perform data preprocessing as a data set; S2. Use the data set to train a multi-scale coarse-grained-fine-grained channel attention U-Net model to obtain a trained multi-scale coarse-grained-fine-grained channel attention U-Net model; S2 includes: S21. Input the data set into the multi-scale coarse-grained-fine-grained channel attention U-Net model. Through forward propagation, starting from the input of the model, the output of each neuron is calculated layer by layer to obtain the predicted value of the model. S22. According to the predicted value of the model, the error between the classification probability map and the label is calculated by the composite loss function, and the error between the classification probability map and the label is back-propagated through the gradient to obtain the weight parameter of the network model; The expression of the composite loss function Loss(p,t) is: Among them, w pos and w neg are the weights of the positive and negative classes, respectively, p i and t i are the predicted value and target value of each pixel position, i represents the index of each pixel position traversed in each image, and smooth is a non-zero minimum number to prevent division by zero; S23, through the early stopping mechanism, save the model parameters that obtain the highest Dice evaluation value during the training process, and obtain the trained multi-scale coarse-grained-fine-grained channel attention U-Net model; S3, inputting the medical ultrasound image to be segmented into the trained multi-scale coarse-grained-fine-grained channel attention U-Net model to obtain the small target segmentation result; S3 includes: S31, inputting the medical ultrasound image to be segmented into the trained multi-scale coarse-grained-fine-grained channel attention U-Net model; S32, gradually extracting features of the input image through the encoder to obtain a high-level feature map; Among them, as the encoder level goes deeper, the size of the high-level feature map gradually decreases, and the number of feature channels gradually increases; S33, performing a step-by-step upsampling operation on the high-level feature map through a decoder with the same number of layers as the encoder, restoring the high-level feature map to the size of the original input image, and obtaining a test image; S33 includes: S331, performing an upsampling operation on the high-level feature map through a decoder to obtain an upsampled image; S332, input the upsampled image as a gated signal into the multi-scale coarse-grained-fine-grained channel attention module, input the feature map extracted by the corresponding level encoder as an input feature into the multi-scale coarse-grained-fine-grained channel attention module, perform adaptive weight feature fusion, and obtain a fused weighted feature map; The relationship between the decoder and the encoder at the corresponding level is: k+j=n Where k is the number of decoder layers, j is the corresponding number of encoder layers, and n is the total number of encoder or decoder layers; In S332, the multi-scale coarse-grained-fine-grained channel attention module processes the gating signal and the incoming features including: S3321, performing 3*3 convolution and 5*5 convolution on the gate signal g respectively to obtain gate signal g3 and gate signal g5; S3322, perform 3*3 convolution and 5*5 convolution on the input feature x respectively to obtain the input feature x3 and the input feature x5; S3323, concatenate the gated signal g3, the gated signal g5, the incoming feature x3, and the incoming feature x5 to obtain a feature group cat; S3324, input the feature group cat into the SE module to obtain an SE weighted feature map; S3325, perform 3*3 convolution operation on the SE weighted feature map to obtain the feature group cat se ; S3326, cat the feature group se Input the multi-scale channel attention module for local and global attention processing to obtain the weight s; The expression of weight s is: s=how much se ⊙σ(BN(ReLU(BN(Conv 1x1 (How se )))))+ BN(ReLU(BN(Conv 1x1 (AvgPool(cat se , Among them, σ() is the Sigmoid activation function, ⊙ represents element-by-element multiplication, Conv 1×1 Represents a 1*1 convolution operation, ReLU() is a ReLU function, AvgPool() represents an average pooling operation, and BN() represents a batch normalization layer; S3327. According to the weight s, a fused weighted feature map z is obtained, and its expression is: z=x3*s+x5*(1-s) Among them, x3 represents the incoming feature x3, and x5 represents the incoming feature x5; S333, sequentially concatenating and fusing the fused weighted feature map and the upsampled image input decoder to obtain a feature fusion map; S334, performing an upsampling operation on the feature fusion image through a decoder to obtain a new upsampled image; S335, repeating S332-S334 until the new up-sampled image is restored to the size of the original input image, thereby obtaining a test image; S34. Convolve the test image, fuse the test image into 1 channel, and obtain a binary image through the Sigmoid function as the small target segmentation result.

2. The small target medical ultrasound image segmentation method based on coarse and fine granularity attention according to claim 1 is characterized in that: S32 includes: S321, performing convolution and pooling operations on the input image in sequence through the encoder; S322, performing convolution and pooling operations on the convolved and pooled image through an encoder to obtain a feature map; S323, repeatedly execute S322 until the predetermined number of convolutions and poolings is reached to complete the downsampling, and perform a convolution operation on the downsampled features as a high-level feature map.

Images