Self-attention medical image segmentation device based on edge high-frequency texture features

Abstract

The application discloses an edge high-frequency texture feature-based self-attention medical image segmentation device, which comprises: images filtered by a Gaussian filtering module are sequentially input into a self-attention module based on a multilayer perceptron and a high-frequency information extraction module based on Fourier transform; the self-attention module based on the multilayer perceptron is a three-layer structure, input features are respectively subjected to the multilayer perceptron and a 1*1 convolution unit to obtain first, second and third feature information vectors after processing; the first and second feature information vectors are multiplied, and the result after multiplication is subjected to a Softmax function to enhance the generalization ability of the network; the result and the third feature information vector are subjected to the above steps repeatedly; finally, dimension transformation work is performed to ensure that the input and output dimensions are consistent; the high-frequency information extraction module based on the Fourier transform converts the image from the time domain to the frequency domain, extracts object edge feature information through high-pass filtering, simultaneously uses the Fourier transform and the inverse Fourier transform to ensure the unity of the dimensions, and is subjected to histogram equalization processing.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image segmentation of machine learning, and particularly relates to an edge high-frequency texture feature based self-attention medical image segmentation device. BACKGROUND

[0002] Medical image segmentation is a complex and key step in the field of medical image processing and analysis, which aims to segment the part with certain special meaning in the medical image and extract relevant features to provide reliable basis for clinical diagnosis and treatment and pathological research, and assist doctors to make more accurate diagnosis.

[0003] Due to the complexity of medical images, a series of problems such as unevenness and individual differences need to be solved in the segmentation process, so that general image segmentation methods are difficult to be directly applied to medical image segmentation. At present, medical image segmentation is still developing from manual segmentation or semi-automatic segmentation to full-automatic segmentation. In traditional medical image segmentation methods, most of them are based on the way of graphics. Commonly used methods include threshold-based segmentation method, edge detection-based segmentation method and clustering-based segmentation method. The threshold-based segmentation method refers to dividing the gray histogram of the image into several classes through one or more thresholds, and considering that the pixels with the same gray value in the image have the same attribute. The key of the algorithm is to obtain the optimal threshold. The edge detection-based segmentation method refers to describing the object edge through the set of pixels with spatial mutation of gray in the image, and describing and detecting the edge through the first and second derivatives. For example: Roberts, Prewitt and Sobel operators. The clustering-based segmentation method belongs to unsupervised learning method, which reveals the inherent properties and laws through learning of unmarked training samples. In the actual training process, the samples in the data set are usually divided into several usually disjoint subsets, each subset is called a "cluster". Similar pixel points are classified into the same cluster, and different clusters are classified into different clusters. Common clustering methods include K-Means algorithm and fuzzy C-means clustering algorithm.

[0004] With the continuous development of computer power and the rise of artificial intelligence, most of the emerging medical image segmentation devices are based on neural networks. Common methods include encoder-decoder based, attention mechanism based, context based, etc. Among them, the encoder-decoder based is to divide the entire network into an encoder and a decoder, the network first encodes the image, and then decodes the target area through the decoder to obtain the segmentation result. Common encoder-decoder networks such as FCN (fully convolutional semantic segmentation network) and U-Net (U-shaped semantic segmentation network). The attention mechanism based is to adjust the attention degree of the network to the target area through the attention mechanism. Common segmentation based on attention mechanism, such as self-attention mechanism and Transformer. The context based is to analyze the association between the target area and the surrounding context in the image to segment the lesion area in the medical image.

[0005] Of course, there are still many problems in the current medical image segmentation, such as many noise points in the image, low image contrast, and unclear segmentation edge details, which will affect the final segmentation result. Therefore, in future work, these problems need to be solved to improve the accuracy of medical image segmentation. SUMMARY

[0006] The application provides a self-attention medical image segmentation device based on edge high-frequency texture features, which combines traditional graphics methods with neural networks, adopts an encoder-decoder structure, and introduces a series of mathematical methods such as self-attention mechanism and Fourier transform to enhance the sensitivity of the network to object edge details, thereby enhancing the network performance and improving the segmentation effect, providing strong data support for subsequent doctor diagnosis, as described in detail below:

[0007] A self-attention medical image segmentation device based on edge high-frequency texture features, the device comprises:

[0008] The image filtered by the Gaussian filtering module is sequentially input into the self-attention module based on the multi-layer perception and the high-frequency information extraction module based on the Fourier transform;

[0009] The self-attention module based on the multi-layer perception is a three-layer structure, the input features are respectively processed by the multi-layer perception, 1*1 convolution unit, to obtain the first, second and third feature information vectors after processing; then the first and second feature information vectors are multiplied, and the result after multiplication is enhanced using the Softmax function to enhance the generalization ability of the network; and the result and the third feature information vector repeat the above steps; finally, the dimension transformation work is performed to ensure that the input and output dimensions are consistent;

[0010] The Fourier transform-based high-frequency information extraction module converts the image from the time domain to the frequency domain, extracts the object edge feature information through high-pass filtering, and simultaneously uses Fourier transform and inverse Fourier transform to ensure the uniformity of the dimension and is processed through histogram equalization.

[0011] The processing result of the histogram equalization is:

[0012]

[0013] The cdf(v) function is used to count the number of feature vectors with a value of v, L represents the number of series, the round function represents the down rounding function, cdf(min) is the number of feature vectors with the minimum value, and cdf(max) is the number of feature vectors with the maximum value.

[0014] The technical scheme provided by the application has the beneficial effects that:

[0015] 1. The application proposes a new efficient feature sampling based on a multi-layer perception and a self-attention mechanism to provide spatial position information for subsequent feature segmentation.

[0016] 2. The application designs a Fourier transform-based high-frequency information extraction module that can extract high-frequency detail information to further optimize the results of lesion edge segmentation.

[0017] 3. The multi-layer perception and self-attention mechanism and the high-frequency information extraction module of the application design a new network structure for medical image segmentation, which can improve the network segmentation effect while optimizing the object edge texture details and further improve the network's effect on medical image segmentation. DETAILED DESCRIPTION

[0018] Figure 1 It is a network architecture diagram for self-attention medical image segmentation based on edge high-frequency texture features.

[0019] Figure 2 It is a schematic diagram of a self-attention module based on a multi-layer perception.

[0020] Figure 3 It is a schematic diagram of a Fourier transform-based high-frequency information extraction module.

[0021] Figure 4 It is a visual display diagram of different filtering methods.

[0022] Figure 5 It is a visual comparison effect diagram of different methods. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical scheme and advantages of the present application more clear, the embodiments of the present application are described in further detail below.

[0024] Medical image segmentation has always been a key research direction in the field of graphics, which plays a crucial role in solving the problems of shortage and imbalance of medical resources. However, the medical image segmentation still faces the problem of unclear segmentation of object edge details, which brings a series of challenges to the contour analysis of target lesions and the later auxiliary diagnosis of doctors.

[0025] The embodiment of the present application mainly describes a self-attention medical image segmentation device based on edge high-frequency texture features, and the network structure diagram of the device is as shown in Figure 1

[0026] The network structure mainly consists of three parts: a Gaussian filtering module, a self-attention module based on a multi-layer perceptron (MLPA), and a high-frequency information extraction module based on Fourier transform (FHF). The image is first filtered by the Gaussian filtering module, where a Gaussian filtering function is used to perform image denoising, the main purpose of which is to remove noise points in the medical image, thereby providing a high-quality clear image for subsequent work. The image filtered by the Gaussian filter is respectively subjected to the self-attention module based on the multi-layer perceptron and the high-frequency information extraction module based on the Fourier transform. In the self-attention module based on the multi-layer perceptron, the present embodiment replaces the 1*1 convolution in the traditional self-attention mechanism with a multi-layer perceptron, thereby improving the richness of network information extraction and enhancing the extraction capability of network information. In the high-frequency information extraction module based on Fourier transform, the present embodiment converts the image from time domain to frequency domain, and extracts object edge feature information through high-pass filtering. Then the two parts of features are combined to obtain fused features and are up-sampled to obtain the final output result.

[0027] I. Gaussian filtering module

[0028] In view of the problem that there are many noise points in the process of collecting and generating medical images, the present embodiment first uses Gaussian filtering for image preprocessing to improve image quality.

[0029] In the process of imaging in the field of medical images, signals will be disturbed due to refraction and scattering, and there will be problems such as dark current, signal loss, etc. in the mutual conversion of information of the collection device. This results in that the medical image collection result will have a lot of noise. In view of this problem, the image is first processed by Gaussian filtering in the present embodiment to reduce the interference of noise in the image on the segmentation result.

[0030] ​Gaussian filter is mainly used for processing Gaussian noise problem, and is a linear smoothing filter. Gaussian noise usually refers to isolated pixels and pixel blocks in an image that cause visual effects. By using a Gaussian filter function, noise in a medical image can be effectively removed, the picture quality is improved, and a high-quality input image is provided for subsequent medical image segmentation. The specific formula of a two-dimensional Gaussian function is as follows:

[0031]

[0032] wherein x and y represent the values of the horizontal coordinate and the vertical coordinate respectively, and sigma represents the width of the Gaussian function. The larger the sigma, the wider the function image span, and vice versa.

[0033] In a specific implementation, the embodiment of the application implements Gaussian filtering in a way of a discrete window sliding window convolution. First, a convolution kernel with a size of 3*3 is set, then a center point to be calculated is fixed as an origin, and then the surrounding points are assigned weights according to a normal distribution function, and a weighted average value is calculated to obtain a final result. Through the Gaussian filtering module, noise problems caused by signal interference and loss in the image can be effectively filtered, and high-quality pictures are provided for subsequent segmentation.

[0034] II. Self-attention module based on multi-layer perceptron

[0035] In view of the problem that a traditional neural network has insufficient information extraction capability for medical images, the embodiment of the application designs a self-attention module based on a multi-layer perceptron, which can enhance the sensitivity of the network to a target segmentation region and improve the segmentation capability of the network for medical targets. The overall design architecture of the module is as shown in Figure 2

[0036] In a traditional self-attention mechanism, a 1*1 convolution unit is used for sampling work at each layer. However, in view of the complex and multi-element condition of medical image information, the traditional method cannot extract rich image information. Therefore, in the present module, a multi-layer perceptron module capable of extracting rich information is combined with the self-attention idea, and a self-attention module based on a multi-layer perceptron is proposed.

[0037] The self-attention module of the multi-layer perceptron is a three-layer structure. The input features are respectively processed by the multi-layer perceptron and the 1*1 convolution unit, and the processed feature information vectors are respectively a, b and c. Then, the vector a and the vector b are multiplied, and the result after multiplication is processed by a Softmax function to enhance the generalization capability of the network. The result is repeated with the vector c. Finally, dimension transformation is performed to ensure that the input and output dimensions are consistent.

[0038] ​Through the self-attention module based on the multi-layer perception designed in the embodiment of the application, the network can extract more abundant image feature information, reduce the information loss caused by the network in the down-sampling process, and in combination with the attention mechanism, improve the attention degree of the network to the target segmentation region, and enhance the overall segmentation capability of the network.

[0039] III. High-frequency information extraction module based on Fourier transform

[0040] In view of the problem of blurred edge texture segmentation of medical images, the embodiment of the application designs a high-frequency information extraction module based on Fourier transform to specifically enhance the network's ability to perceive high-frequency information of the edge of the segmented object. In the module, multiple filtering methods are used to extract high-frequency information in the image, to enhance the network's ability to segment the edge details of the target object, and the overall architecture diagram is as shown in Figure 3 .

[0041] First, the feature information is converted from the time domain to the frequency domain, and here the Fourier transform is used for domain conversion. Fourier transform believes that a periodic function contains multiple frequency components, and any function can be synthesized by adding multiple periodic functions. The specific formula is as follows:

[0042]

[0043] Where, ω represents frequency, t represents time, e -iwt represents a complex function. Through a set of special functions as an orthogonal basis, the original function is converted in advance to obtain the projection of the original function on each group of basis functions.

[0044] In the image processing process, it is generally believed that high-frequency components are places where image intensity changes sharply, that is, the commonly known edge information; low-frequency components are places where image intensity changes gently, that is, places where more color blocks are connected. Therefore, after Fourier transform, high-frequency information is filtered by using a high-pass filter to enhance the edge detail feature information, and the calculation formula of the high-pass filter is as follows:

[0045] f(x i )=α*f(x i-1 )+α*(x i -x i-1 ) (3)

[0046] Where, x i represents the sampling data, f(x i ) represents the filtering result, and alpha < 1, which is used to strengthen the difference between the samples.

[0047] The high-frequency information extracted after the feature information passes through the high-pass filter also needs to be inverse Fourier transformed to ensure that its dimension is consistent with the input dimension, so that effective information enhancement can be performed. The formula of inverse Fourier transform is as follows:

[0048]

[0049] Wherein, ω represents frequency, ω0 represents initial frequency.

[0050] After Fourier transform, high-pass filter and inverse Fourier transform, the feature vector containing high-frequency information is obtained. The feature vector containing high-frequency information is combined with the original feature vector through the way of skip connection, so as to enhance the edge texture details of the object. Then, the local contrast information is enhanced through histogram equalization, and the specific formula is as follows:

[0051]

[0052] Wherein, the cdf(v) function is used to count the number of feature vectors with value v, L represents the number of series, the round function represents the down rounding function, cdf(min) is the number of feature vectors with the minimum value, and cdf(max) is the number of feature vectors with the maximum value.

[0053] Through the high-frequency information extraction module based on Fourier transform designed in the embodiment of the application, the high-frequency information in the feature can be effectively extracted, wherein the edge feature information contained can improve the segmentation ability of the network for edge details. At the same time, Fourier transform and inverse Fourier transform are used to ensure the uniformity of the dimension. Finally, through the processing of histogram equalization, the local performance of the model can be further improved, and the segmentation effect can be improved.

[0054] Experimental details

[0055] 1. Training details:

[0056] All models are trained on an NVIDIA 1080Ti with 11GB of memory. In the training process, the parameters are randomly initialized, and the RMSProp algorithm is used as the optimizer, and the initial learning rate is 0.0001. Unless specified, the size of the input image is adjusted to 800*800 pixels, and 1e-8 weight decay factor and 0.9 momentum drop are used as training parameters.

[0057] 2. Dataset:

[0058] Embodiments of the present application adopt Drosophila cell segmentation data as a data set. In terms of data enhancement, random mirroring and random resizing between 0.5-2 are adopted for all data sets, and additionally random rotation between-10-10 degrees and random Gaussian blur are added as a benchmark. This comprehensive data enhancement scheme enables the network to have anti-overfitting capability.

[0059] 3. Evaluation index:

[0060] The present method is evaluated by four widely used indicators: Dice coefficient, pixel accuracy (PA), sensitivity (Sen) and volume overlap error (VOE). Considering the segmented region (positive) and background (negative), the true positive (TP), true negative (TN), false positive (FP) and false negative (FN) terms are calculated.

[0061] (1) Dice is used as the evaluation index, which is usually used to calculate the similarity of two samples, and the calculation formula is as follows:

[0062]

[0063] (2) Pixel accuracy (PA) is the ratio of all correctly classified pixels to the total number of pixels, and the calculation formula is as follows:

[0064]

[0065] (3) Sensitivity is the ratio of the number of correctly predicted positive samples to the total number of true samples, and the calculation formula is as follows:

[0066]

[0067] (4) VOE represents the volume overlap error, and the calculation formula is as follows:

[0068]

[0069] 4. Ablation experiment:

[0070] Ablation experiments are conducted on the Gaussian filtering module, the self-attention module based on multi-layer perception, and the high-frequency information extraction module based on Fourier transform, to prove the effectiveness of the proposed modules.

[0071] 1) Gaussian filtering module:

[0072] In this part, the performance improvement of the Gaussian filtering module is verified, and the Gaussian filtering is compared with other filtering methods such as mean filtering and median filtering. The backbone network used here is U-Net, and the specific results are shown in Table 1.

[0073] Table 1 shows the ablation experiment results for different filtering methods

[0074] Model name PA Sen VOE Dice U-Net 0.867 0.921 0.154 0.910 U-Net+mean filter 0.866 0.919 0.151 0.911 U-Net+median filter 0.875 0.927 0.144 0.917 U-Net+Gaussian filter 0.883 0.935 0.132 0.922

[0075] As can be seen from Table 1, the network segmentation performance can be improved by using Gaussian filtering, mainly because the noise interference in the image can be removed by using the filtering method, and the generation of noise conforms to the Gaussian distribution rule, so that the effect of Gaussian filtering is better than that of other filtering methods. In order to more intuitively show the filtering effect, the present embodiment performs visual display on each result, as shown in Figure 4 As can be seen from the figure, the image result of Gaussian filtering is better than the other two filtering methods in detail processing, and can effectively remove the noise in the original image.

[0076] 2) Self-attention module based on multi-layer perception:

[0077] In this part, it is verified that the self-attention module based on multi-layer perception can improve the performance of the model, and it is also verified that the connection convolution kernel size of the multi-layer perception connected convolution layer can affect the network capability, and the specific results are shown in Table 2, wherein the MLPA module represents the self-attention module based on multi-layer perception.

[0078] Table 2 shows the ablation experiment results of different convolution kernel sizes

[0079]

[0080]

[0081] As can be seen from Table 2, with the continuous increase of the convolution kernel, the performance of the model decreases, which may be due to the information missing caused by using large convolution kernel for information filtering and extraction after multi-layer perception, thereby affecting the final segmentation result. It is proved by the ablation experiment that the effect is best when the convolution kernel of 1*1 is selected.

[0082] 3) High-frequency information extraction module based on Fourier transform:

[0083] In this part, it is verified that the high-frequency information extraction module based on Fourier transform can improve the performance of the model, and the specific results are shown in Table 3, wherein the FHF module represents the high-frequency information extraction module based on Fourier transform.

[0084] Table 3 shows the ablation experiment results of different modules

[0085]

[0086] As can be seen from Table 3, four combinations are designed for comparison, including U-Net+Gaussian filter, U-Net+Gaussian filter+MLPA module, U-Net+Gaussian filter+FHF module, and U-Net+Gaussian filter+MLPA module+FHF module. Through comparison of experimental results, the embodiments of the present application prove the effectiveness of each module, and the modules can also enhance each other. Finally, the network combining the MLPA module and the FHF module designed in the embodiments of the present application achieves the best effect.

[0087] 5. Quantitative experiment

[0088] In order to better verify the effect of the model proposed in the embodiments of the present application, comparative experiments are performed with other models, including FCN with ResNet as the backbone network, and U-Net, Attention-UNet and UNet++ with U-Net as the backbone network. Through experimental analysis, the present method is superior to other algorithms.

[0089] Table 4. Comparison of results between different models

[0090] Model name PA Sen VOE Dice U-Net 0.867 0.921 0.154 0.910 FCN 0.875 0.932 0.145 0.921 Attention-UNet 0.897 0.949 0.114 0.936 UNet++ 0.884 0.937 0.133 0.925 Algorithm in this paper 0.915 0.951 0.101 0.954

[0091] 6. Qualitative experiment

[0092] In order to further show the superiority of the present method, the embodiments of the present application visually compare the experimental results of the above comparison, and the specific results are shown in Figure 5

[0093] In summary, the self-attention medical image segmentation device based on edge high-frequency texture features proposed in the embodiments of the present application has the following three contributions:

[0094] 1) An improved self-attention module based on multilayer perceptron is proposed. In order to solve the information loss problem caused by sampling in the self-attention mechanism, the embodiments of the present application introduce a multilayer perceptron for efficient feature sampling, which provides spatial position information support for subsequent feature segmentation.

[0095] 2) A high-frequency information extraction module based on Fourier transform is proposed. The module combines traditional image analysis knowledge of image edge texture features, and efficiently combines multiple feature extraction methods. It can effectively extract high-frequency feature information in image information, and provide edge detail texture information support for segmentation results.

[0096] 3) A new network structure for medical image segmentation is proposed. The structure effectively combines the advantages of traditional graphics and modern neural networks, and can improve the network segmentation effect while optimizing the object edge texture details, which is very important for medical image segmentation.​

[0097] The effectiveness of the self-attention module based on the multi-layer perception and the high-frequency information extraction module based on the Fourier transform is proved through a series of experiments, and the advancement of the new network model is verified by comparing with other network models. The new network model not only provides a new idea for further improving the medical image segmentation effect, but also provides technical support for the construction of a new medical image auxiliary judgment platform.

[0098] In the embodiments of the present application, the types of the devices are not limited unless otherwise specified, and any device that can complete the above functions can be used.

[0099] Those skilled in the art can understand that the drawings are only schematic diagrams of a preferred embodiment, and the above-mentioned serial numbers of the embodiments of the present application are only for description, not representing the advantages and disadvantages of the embodiments.

[0100] The above is only a preferred embodiment of the present application, and does not limit the present application. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. An edge high-frequency texture feature based self-attention medical image segmentation device, characterized in that, The device comprises: The image filtered by the Gaussian filtering module is sequentially input into a self-attention module based on a multi-layer perception and a high-frequency information extraction module based on Fourier transform; The self-attention module based on the multi-layer perception is a three-layer structure, and the input features are processed by the multi-layer perception and a 1*1 convolution unit to obtain first, second and third feature information vectors; then the first and second feature information vectors are multiplied, and the result is used as an input of a Softmax function to enhance the generalization ability of the network; the result is repeatedly processed with the third feature information vector; finally, a dimension transformation is performed to ensure the consistency of the input and output dimensions; The high-frequency information extraction module based on the Fourier transform converts the image from the time domain to the frequency domain, extracts the edge feature information of the object by high-pass filtering, and uses the Fourier transform and the inverse Fourier transform to ensure the consistency of the dimensions and perform histogram equalization processing.

2. The self-attention medical image segmentation device based on edge high-frequency texture features according to claim 1, characterized in that, The processing result of the histogram equalization is: wherein the cdf(v) function is used to count the number of feature vectors with a value of v, L represents the number of series, the round function represents a down-round function, cdf(min) is the number of feature vectors with a minimum value, and cdf(max) is the number of feature vectors with a maximum value.

Images