An image anomaly feature segmentation method based on improved U-Net

Abstract

The application discloses an image abnormal feature segmentation method based on an improved U-Net. A residual convolution module is taken as a basic unit, and an image abnormal feature segmentation method based on the improved U-Net is proposed. The residual convolution module is introduced in the coding and decoding parts of the U-Net to extract features of an image target. In combination with an improved spatial attention mechanism, the attention degree of a target region is strengthened. Finally, a multi-level feature strengthening module is used to fully fuse spatial information of low-level features and semantic information of high-level features. A hybrid loss is used to optimize the segmentation result during network training. The medical image abnormal feature segmentation method can improve the segmentation precision of medical image abnormal features.

Description

Technical Field

[0001] This invention relates to the field of deep learning, specifically to an image anomaly feature segmentation method based on an improved U-Net, and more particularly to a medical image lesion segmentation method. Background Technology

[0002] Medical imaging is an effective medical method for diagnosing, treating, evaluating, and preventing various serious diseases. Currently, experienced doctors can use medical images and other auxiliary information to identify lesion areas in images. However, highly skilled doctors are very scarce, and manual image interpretation consumes a lot of doctors' time and energy. The large amount of repetitive image interpretation also leads to a waste of medical resources. Moreover, medical images are inherently blurry, and manual interpretation is also subject to the doctor's subjective bias, causing a series of problems such as misidentification.

[0003] With the development of computer technology, some researchers have attempted to solve these problems using image processing techniques and machine learning methods. However, traditional image processing methods, such as thresholding, struggle to handle complex images and suffer from low segmentation accuracy. Although machine learning methods have made significant progress in image segmentation tasks, they still have drawbacks such as high resource consumption, weak generalization ability, and lack of consistency and continuity in segmentation results.

[0004] Deep learning offers advantages in medical image lesion segmentation, including high accuracy, high automation, strong learning ability, strong contextual understanding, strong multimodal processing capabilities, and good transferability. These advantages make deep learning a crucial technology in medical image segmentation, playing a vital role in medical diagnosis and treatment. Medical image segmentation can automatically delineate lesion regions in images, assisting doctors in diagnosis and treatment planning. A nuclear medicine bone transfer segmentation method based on a lightweight convolutional neural network is presented, using an improved feature extraction convolutional module and comparing it with ResNet, Inception, and other structures to demonstrate the algorithm's effectiveness. An automatic segmentation method for arthritis lesions in SPECT bone scintigraphy based on Mask R-CNN is also presented, using a traditional segmentation model with clustering techniques. The effectiveness of this segmentation model in terms of accuracy, recall, and mean intersection-over-union ratio (MCR) for arthritis lesion segmentation is verified. Finally, a deep learning-based automatic segmentation method for hotspot transfer in chest SPECT images achieves beneficial results by fine-tuning the structure of U-Net and Mask R-CNN to construct the segmentation model. Summary of the Invention

[0005] 1. Purpose of the invention:

[0006] The purpose of this invention is to propose an image abnormality feature segmentation method based on an improved U-Net, which can automatically segment the lesion area in medical images, assist doctors in making more accurate diagnoses of patients' conditions, and greatly improve the doctors' diagnostic efficiency.

[0007] 2. Technical Solution:

[0008] A method for segmenting image anomalies based on an improved U-Net, characterized by the following steps:

[0009] Step 1: Acquire images, preprocess them, and divide them into training and test sets;

[0010] Step 2: Construct an improved U-Net model, which consists of an encoder and decoder with multi-scale feature fusion:

[0011] Step 2.1: Construct the encoder section, which includes the following structural components:

[0012] First, a multi-scale feature enhancement module is constructed; then, a main encoder feature extraction module is constructed; and finally, a main encoder feature enhancement module is constructed.

[0013] The multi-scale feature enhancement module consists of two branches. The two branches use 3×3 and 5×5 convolution kernels respectively for feature extraction. The results of the convolution of the two branches are added together and fused.

[0014] The main encoder feature extraction module consists of UD1 and UD2. Two UD2 modules are used after the first two UD1 modules. x By residual convolution module U x The system consists of three parts connected in series: an improved spatial attention module, a downsampling module, and a 3D model (x = 1, 2).

[0015] The calculation process for the residual convolution module U1 is as follows:

[0016]

[0017]

[0018] x 11 =concat(x1,x2)

[0019]

[0020] x 22 =concat(x3,x in )

[0021]

[0022] xout =x4+x in

[0023] The calculation process of the residual convolution module U2 is as follows:

[0024]

[0025]

[0026]

[0027] x 11 =concat(x2,x3)

[0028]

[0029] x 22 =concat(x1,x4)

[0030]

[0031] x 33 =concat(x5,x in )

[0032]

[0033] x out =x6+x in

[0034] In the formula: Conv 1×1 (x in ), Conv 3×3 (x in ), Conv 5×5 (x in ), Conv 7×7 (x in () represents 1×1 convolution, 3×3, 5×5, and 7×7 convolution operations, respectively, x in The input image for the feature extraction module is x1, x2, x3, x4, x5, and x6, which are the results after convolution. 11 x 22 x 33 The result after feature channel concatenation, x out This is the output feature map of the residual convolution module; concat means feature concatenation, ReLU means activation function, and BN means batch normalization.

[0035] The improved spatial attention module calculation process is as follows:

[0036]

[0037] x2=x1-Avg 3x3 (x1)

[0038] x3=x1-Avg 5x5 (x1)

[0039] x4 = concat(x1, x2, x3)

[0040]

[0041] x6=Avg 2x2 (x5)

[0042] x7 = Maxpool 2x2 (x5)

[0043] x8 = concat(x6, x7)

[0044]

[0045] x out =x in *x9

[0046] In the formula: Conv 1×1 (x i ) represents a 1×1 convolution operation, Conv 3×3 (x in ) represents a 3×3 convolution operation, x in The input image for the feature extraction module is shown. x2 and x3 are the results of local mean normalization, x4 is the result after concatenation of local mean normalized values, x5 is the result after channel compression, x6 is the result after average pooling, x7 is the result after max pooling, x8 is the result after concatenation of average and max pooling, and x9 is the calculated attention coefficient. out The attention weights are calculated; concat represents feature concatenation, Maxpool... 2x2 This represents 2×2 max pooling, Avg 3x3 Avg 5x5 Represents 3×3 and 5×5 average pooling, ReLU represents the activation function, and Sigmoid represents the activation function;

[0047] The feature enhancement module calculation process is as follows:

[0048] x1=x i+1 *Sigmoid(x i+1 )

[0049] x2=x i+2 *Sigmoid(x i+2 )

[0050] x3=x i+3 *Sigmoid(x i+3 )

[0051] x4 = x1 + x2 + x3

[0052] x out =x i +(1-Sigmoid(x i ))*x4

[0053] In the formula: x i Input feature image for the current layer, x i+1 x i+2 x i+3 For other levels of input features, x1, x2, and x3 are the results after feature enhancement at other levels, and x4 represents the weighted result of features at other levels; x out This represents the result of fusing features from different levels; Sigmoid represents the activation function.

[0054] Step 2.2: Construct the decoder section, which consists of the following structure:

[0055] First, an upsampling module is constructed, followed by a feature recovery module. The decoding process is symmetrical to the encoding process. Features are sequentially upsampled from the encoder's low-level features for feature fusion and recovery. Upsampling uses bilinear interpolation, fusion uses concat concatenation, and recovery uses a residual convolution module. x The module consists of the first two layers with x=2, the last two layers with x=1, and finally a 1×1 convolution to obtain the output.

[0056] Step 2.3: Construct a hybrid loss function to calculate the training loss, use the Adam optimizer to optimize the network training process, and set the model quantization evaluation method;

[0057] The hybrid loss function combines cross-entropy loss and Focal Loss, and its expression is as follows:

[0058] L H =0.8L ce +0.2L fl

[0059] In the formula: L H L represents the total loss. ce Represents cross-entropy loss; L fl Represents Focal Loss;

[0060] The formula for cross-entropy loss is:

[0061] L ce=-(ylog(p)+(1-y)log(1-p)),

[0062] In the formula: l ce y represents the value of the cross-entropy loss function, p is the probability that the model predicts the positive class, and y is the manually labeled ground truth label, which is 0 for background images and 1 for foreground images.

[0063] L fl The formula for focus loss is:

[0064] FL(p k )=-α(1-p k ) γ lg(p k ),

[0065] In the formula, α is a constant. When γ increases, α needs to decrease slightly, (1-p k ) γ p is the modulation coefficient. k This represents the probability that the predicted sample belongs to the k-th class;

[0066] The average intersection-over-union ratio (OCU) is used as a quantitative evaluation method for the model. The formula for calculating the average OCU is:

[0067]

[0068] In the formula: TP is a true positive, representing a positive sample predicted as a positive sample; FP is a false positive, representing a positive sample predicted as a negative sample; FN is a true negative, representing a positive sample predicted as a negative sample; TN is a false negative, representing a negative sample predicted as a negative sample; TP, FP, FN, and TN are all positive integers.

[0069] Step 3: Input the preprocessed test set data from Step 1 into the network model built in Step 2 for training, and evaluate the segmentation effect of the model by average intersection-over-union comparison, and save the best network model as the Best Model.

[0070] Step 4: Input the images from the test set into the Best Model to obtain the segmented images.

[0071] Innovation points:

[0072] Compared to the original U-Net model, this invention replaces the basic convolutional modules with efficient residual modules, improving the ability to capture multi-scale image anomaly features and training efficiency. An improved spatial attention mechanism is designed, which first performs local mean normalization on the original feature map, thereby subtracting background information and highlighting the target or region of interest, better capturing the spatial features of the image. A multi-level feature enhancement module is designed to effectively fuse low-level and high-level features, achieving information complementarity at different levels.

[0073] Beneficial effects:

[0074] Compared with the current mainstream image segmentation methods, this invention enhances the ability to extract different organ or lesion regions, and effectively integrates the spatial detail features of low-level features and the rich semantic information of high-level features. The segmentation effect of the method of this invention is better than the current mainstream segmentation algorithms, and has certain practical application value. Attached Figure Description

[0075] Figure 1 The improved U-Net model diagram constructed for this invention mainly includes an encoder, a decoder, and a feature fusion output part;

[0076] Figure 2 Figure U1 shows the residual convolution module, which mainly uses convolution kernels of different sizes and residual ideas to extract features in the shallow layers of the encoder and decoder.

[0077] Figure 3 Figure U2 shows the residual convolutional module, which mainly uses convolutional kernels of different sizes and residual concepts to extract features in the deep layers of the encoder and decoder.

[0078] Figure 4 To improve the spatial attention module diagram, local mean normalization is introduced to increase the model's attention to the spatial location of lesions;

[0079] Figure 5 To enhance the module graph for multi-level features, fully integrate low-level detailed features and deep semantic information;

[0080] Figure 6 A graph showing the loss and miou values ​​during model training;

[0081] Figure 7 The first column shows the partial segmentation results; Image represents a medical image, Label represents an expert-labeled lesion mask; the remaining columns show the segmentation results of mainstream segmentation models, and Improved U-Net represents the model proposed in this method. Detailed Implementation

[0082] The present invention will now be described in detail with reference to the accompanying drawings. Specific embodiments of the present invention are as follows:

[0083] An image anomaly feature segmentation method based on improved U-Net:

[0084] Step 1: Obtain images and perform standardization processing; divide the preprocessed image dataset into training and test sets in a 7:3 ratio;

[0085] Step 2: Construct an improved U-Net model, which consists of an encoder and decoder with multi-scale feature fusion:

[0086] Step 2.1: Construct the encoder section, which includes the following structural components:

[0087] First, a multi-scale feature enhancement module is constructed; then, a main encoder feature extraction module is constructed; and finally, a main encoder feature enhancement module is constructed.

[0088] The multi-scale feature enhancement module consists of two branches. The two branches use 3×3 and 5×5 convolution kernels respectively for feature extraction. The results of the convolution of the two branches are added together and fused.

[0089] The main encoder feature extraction module consists of UD1 and UD2. Two UD2 modules are used after the first two UD1 modules. x By residual convolution module U x The system consists of three parts connected in series: an improved spatial attention module, a downsampling module, and a 3D model (x = 1, 2).

[0090] The calculation process for the residual convolution module U1 is as follows:

[0091]

[0092]

[0093] x 11 =concat(x1,x2)

[0094]

[0095] x 22 =concat(x3,x in )

[0096]

[0097] x out =x4+x in

[0098] The calculation process of the residual convolution module U2 is as follows:

[0099]

[0100]

[0101]

[0102] x 11 =concat(x2,x3)

[0103]

[0104] x 22 =concat(x1,x4)

[0105]

[0106] x 33 =concat(x5,x in )

[0107]

[0108] x out =x6+x in

[0109] In the formula: Conv 1×1 (x in ), Conv 3×3 (x in ), Conv 5×5 (x in ), Conv 7×7 (x in () represents 1×1 convolution, 3×3, 5×5, and 7×7 convolution operations, respectively, x in The input image for the feature extraction module is x1, x2, x3, x4, x5, and x6, which are the results after convolution. 11 x 22 x 33 The result after feature channel concatenation, x out This is the output feature map of the residual convolution module; concat means feature concatenation, ReLU means activation function, and BN means batch normalization.

[0110] The improved spatial attention module calculation process is as follows:

[0111]

[0112] x2=x1-Avg 3x3 (x1)

[0113] x3=x1-Avg 5x5 (x1)

[0114] x4 = concat(x1, x2, x3)

[0115]

[0116] x6=Avg 2x2 (x5)

[0117] x7 = Maxpool 2x2 (x5)

[0118] x8 = concat(x6, x7)

[0119]

[0120] x out =x in *x9

[0121] In the formula: Conv 1×1 (x i ) represents a 1×1 convolution operation, Conv 3×3 (x in ) represents a 3×3 convolution operation, x in The input image for the feature extraction module is shown. x2 and x3 are the results of local mean normalization, x4 is the result after concatenation of local mean normalized values, x5 is the result after channel compression, x6 is the result after average pooling, x7 is the result after max pooling, x8 is the result after concatenation of average pooling and max pooling, and x9 is the calculated attention coefficient. out The attention weights are calculated; concat represents feature concatenation, Maxpool... 2x2 This represents 2×2 max pooling, Avg 3x3 Avg 5x5 Represents 3×3 and 5×5 average pooling, ReLU represents the activation function, and Sigmoid represents the activation function;

[0122] The feature enhancement module calculation process is as follows:

[0123] x1=x i+1 *Sigmoid(x i+1 )

[0124] x2=x i+2 *Sigmoid(x i+2 )

[0125] x3=x i+3 *Sigmoid(x i+3 )

[0126] x4 = x1 + x2 + x3

[0127] x out =x i +(1-Sigmoid(x i ))*x4

[0128] In the formula: x i Input feature image for the current layer, x i+1 x i+2 x i+3For other levels of input features, x1, x2, and x3 are the results after feature enhancement at other levels, and x4 represents the weighted result of features at other levels. out This represents the result of fusing features from different levels; Sigmoid represents the activation function.

[0129] Step 2.2: Construct the decoder section, which consists of the following structure:

[0130] First, an upsampling module is constructed, followed by a feature recovery module. The decoding process is symmetrical to the encoding process. Features are sequentially upsampled from the encoder's low-level features for feature fusion and recovery. Upsampling uses bilinear interpolation, fusion employs concat concatenation, and recovery uses a residual convolution module. x The module consists of the first two layers with x=2, the last two layers with x=1, and finally a 1×1 convolution to obtain the output.

[0131] Step 2.3: Construct a hybrid loss function to calculate the training loss, use the Adam optimizer to optimize the network training process, and set the model quantization evaluation method;

[0132] The hybrid loss function combines cross-entropy loss and Focal Loss, and its expression is as follows:

[0133] L H =0.8L ce +0.2L fl

[0134] In the formula: L H L represents the total loss. ce Represents cross-entropy loss; L fl Represents Focal Loss;

[0135] The formula for cross-entropy loss is:

[0136] L ce =-(ylog(p)+(1-y)log(1-p)),

[0137] In the formula: L ce y represents the value of the cross-entropy loss function, p is the probability that the model predicts the positive class, and y is the manually labeled true label, where 0 represents the background and 1 represents the foreground lesion.

[0138] L fl The formula for focus loss is:

[0139] FL(p k )=-α(1-p k ) γ lg(p k ),

[0140] In the formula, α is a constant. When γ increases, α needs to decrease slightly, (1-p k ) γ p is the modulation coefficient. k This represents the probability that the predicted sample belongs to the k-th class;

[0141] The average intersection-over-union ratio (OCU) is used as a quantitative evaluation method for the model. The formula for calculating the average OCU is:

[0142]

[0143] In the formula: TP is a true positive, representing a positive sample predicted as a positive sample; FP is a false positive, representing a positive sample predicted as a negative sample; FN is a true negative, representing a positive sample predicted as a negative sample; TN is a false negative, representing a negative sample predicted as a negative sample; TP, FP, FN, and TN are all positive integers.

[0144] Step 3: Input the preprocessed test set data from Step 1 into the network model built in Step 2 for training, and evaluate the segmentation effect of the model by average intersection-over-union comparison, and save the best network model as the Best Model.

[0145] Step 4: Input the images from the test set into the Best Model to obtain the segmented images.

[0146] Example of the effect of the implementation method of the present invention Figure 7 The table below presents the segmentation effects of various models and the method of this invention on lesions in medical images, as shown in Table 1:

[0147] Table 1. Average intersection-union ratios of different algorithms

[0148] U-Net 0.4032 At_U-Net 0.4201 Dense_U-Net 0.4364 Trans_U-Net 0.4417 Algorithm of this invention 0.4763

[0149] As can be seen from the table above, the medical image lesion segmentation method based on the improved U-Net proposed in this invention has the highest average crossover-union ratio and the best performance.

Claims

1. An image anomaly feature segmentation method based on an improved U-Net, characterized in that... Includes the following steps: Step 1: Acquire images, preprocess them, and divide them into training and test sets; Step 2: Construct an improved U-Net model, which consists of an encoder and decoder with multi-scale feature fusion: Step 2.1: Construct the encoder section, which includes the following structural components: First, a multi-scale feature enhancement module is constructed; then, a main encoder feature extraction module is constructed; and finally, a main encoder feature enhancement module is constructed. The multi-scale feature enhancement module consists of two branches. The two branches use 3×3 and 5×5 convolution kernels respectively for feature extraction. The results of the convolution of the two branches are added together and fused. The main encoder feature extraction module consists of UD1 and UD2. Two UD2 modules are used after the first two UD1 modules. x By residual convolution module U x The system consists of three parts connected in series: an improved spatial attention module, a downsampling module, and a 3D model (x = 1, 2). The calculation process for the residual convolution module U1 is as follows: x 11 =concat(x1,x2) x 22 =concat(x3,x in ) x out =x4+x in The calculation process of the residual convolution module U2 is as follows: x 11 =concat(x2,x3) x 22 =concat(x1,x4) x 33 =concat(x5,x in ) x out =x6+x in In the formula: Conv 1×1 (x in ), Conv 3×3 (x in ), Conv 5×5 (x in ), Conv 7×7 (x in () represents 1×1 convolution, 3×3, 5×5, and 7×7 convolution operations, respectively, x in The input image for the feature extraction module is x1, x2, x3, x4, x5, and x6, which are the results after convolution. 11 x 22 x 33 The result after feature channel concatenation, x out This is the output feature map of the residual convolution module; concat means feature concatenation, ReLU means activation function, and BN means batch normalization. The improved spatial attention module calculation process is as follows: x2=x1-Avg 3x3 (x1) x3=x1-Avg 5x5 (x1) x4 = concat(x1, x2, x3) x6=Avg 2x2 (x5) x7=Maxpool 2x2 (x5) x8 = concat(x6, x7) x out =x in *x9 In the formula: Conv 1×1 (x i ) represents a 1×1 convolution operation, Conv 3×3 (x in ) represents a 3×3 convolution operation, x in The input image for the feature extraction module is shown. x2 and x3 are the results of local mean normalization, x4 is the result after concatenation of local mean normalized values, x5 is the result after channel compression, x6 is the result after average pooling, x7 is the result after max pooling, x8 is the result after concatenation of average and max pooling, and x9 is the calculated attention coefficient. out The attention weights are calculated; concat represents feature concatenation, Maxpool... 2x2 This represents 2×2 max pooling, Avg 3x3 Avg 5x5 Represents 3×3 and 5×5 average pooling, ReLU represents the activation function, and Sigmoid represents the activation function; The feature enhancement module calculation process is as follows: x1=x i+1 *Sigmoid(x i+1 ) x2=x i+2 *Sigmoid(x i+2 ) x3=x i+3 *Sigmoid(x i+3 ) x4 = x1 + x2 + x3 x out =x i +(1-Sigmoid(x i ))*x4 In the formula: x i Input feature image for the current layer, x i+1 x i+2 x i+3 For other levels of input features, x1, x2, and x3 are the results after feature enhancement at other levels, and x4 represents the weighted result of features at other levels; x out This represents the result of fusing features from different levels; Sigmoid represents the activation function. Step 2.2: Construct the decoder section, which consists of the following structure: First, an upsampling module is constructed, followed by a feature recovery module. The decoding process is symmetrical to the encoding process. Features are sequentially upsampled from the encoder's low-level features for feature fusion and recovery. Upsampling uses bilinear interpolation, fusion uses concat concatenation, and recovery uses a residual convolution module. x The module consists of the first two layers with x=2, the last two layers with x=1, and finally a 1×1 convolution to obtain the output. Step 2.3: Construct a hybrid loss function to calculate the training loss, use the Adam optimizer to optimize the network training process, and set the model quantization evaluation method; The hybrid loss function combines cross-entropy loss and Focal Loss, and its expression is as follows: L H <0.8L ce +0.2L fl In the formula: L H L represents the total loss. ce Represents cross-entropy loss; L fl Represents Focal Loss; The formula for cross-entropy loss is: L ce =-(ylog(p)+(1-y)log(1-p)), In the formula: L ce y represents the value of the cross-entropy loss function, p is the probability that the model predicts the positive class, and y is the manually labeled ground truth label, which is 0 for background images and 1 for foreground images. L fl The formula for focus loss is: FL(p k )m-α(1-p k ) γ 1g(p k ), In the formula, α is a constant. When γ increases, α needs to decrease slightly, (1-p k ) γ p is the modulation coefficient. k This represents the probability that the predicted sample belongs to the k-th class; The average intersection-over-union ratio (OCU) is used as a quantitative evaluation method for the model. The formula for calculating the average OCU is: In the formula: TP is a true positive, representing a positive sample predicted as a positive sample; FP is a false positive, representing a positive sample predicted as a negative sample; FN is a true negative, representing a positive sample predicted as a negative sample; TN is a false negative, representing a negative sample predicted as a negative sample; TP, FP, FN, and TN are all positive integers. Step 3: Input the preprocessed test set data from Step 1 into the network model built in Step 2 for training, and evaluate the segmentation effect of the model by average intersection-over-union comparison, and save the best network model as the Best Model. Step 4: Input the images from the test set into the Best Model to obtain the segmented images.

Images