Anterior cruciate ligament image segmentation method fusing expansion convolution and residual mixed attention

Abstract

The invention discloses an anterior cruciate ligament image segmentation method fusing expansion convolution and residual mixed attention. The anterior cruciate ligament image segmentation method comprises the following steps of 1, data set acquisition and image preprocessing; 2, expanding the preprocessed data set by using an image enhancement mode; step 3, the U-Net model is improved, and a DRH-UNet segmentation model is established: on the basis of the U-Net, an RH Block is designed, and expansion convolution is introduced; 4, model training and hyper-parameter optimization: training the model by using the expanded data set, and meanwhile, trying different hyper-parameter combinations to train a DRH-UNet segmentation model with optimal performance; and 5, carrying out segmentation test on the test set by using the DRH-UNet segmentation model with the optimal performance obtained in the step 4 to obtain a segmentation result. According to the method, image preprocessing and data set expansion are carried out on the knee joint MRI, the conditions of data imbalance, overfitting and the like are effectively prevented, and the segmentation effect is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of anterior cruciate ligament diagnosis in image processing and medical image analysis, and particularly to an anterior cruciate ligament image segmentation method integrating dilated convolution and residual hybrid attention. Background Art

[0002] The anterior cruciate ligament is a dense connective tissue band extending from the femur to the tibia, which plays a key role in maintaining the stability of the knee joint during human activities (including running, jumping, and rotation). Anterior cruciate ligament injury is a common and serious injury in sports, which not only causes joint effusion, muscle weakness, and a significant decline in motor function, but also, in the long term, the clinical sequelae of anterior cruciate ligament injury will increase the risk of osteoarthritis. Therefore, quickly and accurately diagnosing whether the anterior cruciate ligament is injured is extremely crucial for the treatment and rehabilitation of patients. In recent years, due to the fact that magnetic resonance imaging (MRI) can clearly display the soft tissue structure in the knee joint without radiation, it has become an important auxiliary means for clinicians to diagnose diseases, evaluate prognosis, and formulate surgical plans, and is an effective method for clinically diagnosing anterior cruciate ligament injury, which can help doctors determine the location and degree of the injury. However, in the MRI images of the knee joint, the area presented by the anterior cruciate ligament is small and difficult to distinguish, resulting in time-consuming and laborious clinical film reading by doctors. Therefore, introducing medical image segmentation technology based on computer vision to achieve efficient and accurate segmentation of the anterior cruciate ligament helps to assist doctors in making faster and more accurate diagnoses and decisions.

[0003] Traditional research on medical image segmentation methods usually relies on edge detection, statistical shape models, graph cuts, etc. Although these traditional methods show certain innovation, they still need to be improved in terms of accuracy, efficiency, and adaptability. Therefore, how to achieve efficient and accurate segmentation remains the key point of research.

[0004] In recent years, image segmentation methods based on deep learning have achieved remarkable results in the field of medical image segmentation. Among them, the classic medical image segmentation model U-Net has been widely favored in the field of medical image segmentation due to its excellent performance, strong adaptability, and high efficiency. This model adopts a U-shaped topological structure composed of a symmetric encoder-decoder, uses multi-level skip connections to achieve cross-layer feature fusion, and at the same time integrates shallow texture information at different levels in the encoding stage through horizontal connections, which is beneficial to the segmentation of images. However, for the segmentation task of soft tissues such as the anterior cruciate ligament with characteristics of low contrast and significant morphological variation, traditional U-Net still has limitations in capturing fine-grained features and long-range context modeling, and it needs to be improved to enhance its feature representation ability for the anterior cruciate ligament, thereby improving the segmentation accuracy of the anterior cruciate ligament. Summary of the Invention

[0005] To overcome the limitations and low precision of U-Net in anterior cruciate ligament segmentation, the present invention provides a method for segmenting anterior cruciate ligament images that integrates dilated convolution and residual hybrid attention by improving the U-Net model, so as to make full use of the feature information of the anterior cruciate ligament and the advantages of the U-Net model in feature extraction, and improve the segmentation precision of the anterior cruciate ligament. It is necessary to develop a method for segmenting anterior cruciate ligament images that integrates dilated convolution and residual hybrid attention, including the following steps:

[0006] Step 1: Dataset acquisition and image preprocessing;

[0007] Step 2: Expand the preprocessed dataset using image enhancement methods;

[0008] Step 3: Improve the U-Net model and establish a DRH-UNet segmentation model: On the basis of U-Net, design an RHBlock and introduce dilated convolution;

[0009] Step 4: Model training and hyperparameter optimization: Use the expanded dataset to train the model, and at the same time try different combinations of hyperparameters to train the DRH-UNet segmentation model with the best performance;

[0010] Step 5: Use the DRH-UNet segmentation model with the best performance obtained in Step 4 to perform segmentation tests on the test set to obtain segmentation results.

[0011] Preferably, the image preprocessing in Step 1 refers to screening out the images in which the anterior cruciate ligament is clearly and completely shown from the obtained sagittal MRI images of the knee joint, and using the image annotation tool Labelme to annotate them to obtain a mask label, that is, the true label of the segmentation.

[0012] Preferably, the expansion of the preprocessed dataset using image enhancement methods in Step 2 refers to expanding the dataset obtained after image preprocessing by using data enhancement methods such as translation, small-angle rotation, and adding noise to prevent overfitting caused by a small amount of data.

[0013] Preferably, in step 3, the U-Net model is improved to establish a DRH-UNet segmentation model, which mainly includes: designing an RH Block on the skip connection between the encoder and decoder of the U-Net model. This module first calculates channel attention and spatial attention on the input features in sequence to obtain channel attention features and spatial attention features, and multiplies them with their input features respectively; then obtains the preliminarily processed features through a convolutional layer, and uses a residual connection to fuse the preliminarily processed features with the original input; finally, performs a non-linear transformation on the fused feature map through a parametric rectified linear unit (PReLU) activation function to obtain the final output; where the channel attention mechanism, spatial attention mechanism, and PReLU activation function are defined as follows:

[0014] Mc(F c )=σ(MLP(AvgPool(F c ))+MLP(MaxPool(F c )))#(2)

[0015] Ms(F s )=σ(f 7×7 [AvgPool(F s );MaxPool(F s )])#(3)

[0016] PReLU(x)=max{x,ax}#(4)

[0017] Among them, in formulas (2) and (3), σ represents the Sigmoid activation function, MLP represents the multi-layer perceptron, AvgPool and MaxPool respectively represent global average pooling and global maximum pooling, f 7×7 represents a convolution operation with a convolution kernel size of 7×7, F c and F s respectively represent the feature inputs of the channel attention mechanism calculation function Mc and the spatial attention mechanism calculation function Ms; in formula (4), x represents the input of the non-linear activation function PReLU, and a represents the gradient of the negative value part. Therefore, the output of the RH Block is H TY :

[0018] H TY (x)=PReLU(Conv3(Ms(Mc(x)))+x)#(5)

[0019] Among them, Conv3 is a 3×3 convolution, Mc and Ms are channel and spatial attention calculations, PReLU is the activation function, and x is the original input.

[0020] Preferably, the U-Net model is improved in step 3 to establish a DRH-UNet segmentation model, mainly including: in each layer of the encoder and decoder of the U-Net model, the second ordinary convolution is replaced by an expanded convolution; compared with ordinary convolution, the expanded convolution introduces a hyperparameter of the expansion rate, which changes the sampling method of the convolution kernel when processing data; when the expansion rate is 1, the expanded convolution degenerates into ordinary convolution; and when the expansion rate is greater than 1, that is, n, the adjacent sampling points of the convolution kernel will be spaced by n-1 data points, and sampling is performed at non-continuous positions of the input matrix, while keeping the physical size of the convolution kernel unchanged and without adding additional parameters.

[0021] Preferably, the model training and hyperparameter optimization in step 4 refers to trying different hyperparameter combinations during the model training process. The hyperparameter combination can be a learning rate, a batch size, and a regularization parameter to find the optimal hyperparameter combination and train the DRH-UNet segmentation model with the best performance.

[0022] Preferably, the segmentation test in step 5 refers to performing a segmentation test on the test set using the DRH-UNet segmentation model with the best performance obtained in step 4 to obtain segmentation results of relevant performance evaluation indicators: DSC coefficient and HD coefficient.

[0023] The purpose of the present invention is to provide a

[0024] Beneficial effects of the present invention: The present invention performs image preprocessing and data set expansion on knee joint MRI, effectively preventing data imbalance, overfitting, etc., and enhancing the segmentation effect;

[0025] Based on the U-Net model, this paper designs RHBlock, which performs dual attention calculations on the channel and spatial dimensions of the image. It also introduces dilated convolution to expand the model's receptive field, enhancing the ability to characterize the anterior cruciate ligament's characteristic regions and their structural relationship with other physiological tissues of the knee joint, effectively improving anterior cruciate ligament segmentation and providing preliminary pathological information for anterior cruciate ligament diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 Schematic diagram of the process of the anterior cruciate ligament image segmentation method of the present invention;

[0027] Figure 2 This is a flow chart of model training and verification of the present invention;

[0028] Figure 3 This is a structural diagram of the DRH-UNet segmentation model of the present invention;

[0029] Figure 4 RHBlock structure diagram designed for the present invention;

[0030] Figure 5 The figure shows the comparison of the segmentation results of the present invention on the test set. DETAILED DESCRIPTION

[0031] The present invention provides a method for anterior cruciate ligament image segmentation that integrates dilated convolution and residual mixed attention. Figure 1 Its flow chart includes the following steps:

[0032] Step 1: Dataset acquisition and image preprocessing: Obtain a knee joint MRI dataset, select MR images showing complete and clear anterior cruciate ligaments, and annotate the images.

[0033] The MRNet dataset is the knee joint MRI data released by Stanford University Medical Center. The dataset contains 1,370 knee MRI tests, which were collected at a magnetic field strength of 3.0T or 1.5T. Each case has MRI in three anatomical orientations: sagittal, coronal, and axial. The dataset is divided into 1,130 training sets, 120 validation sets, and 120 test sets. Since Stanford University Medical Center only released the training and validation sets to the public, the data used in the experiment totaled 1,250 MRI tests. In addition, given that sagittal MRI can fully and clearly display the anterior cruciate ligament, the present invention only uses sagittal data and performs the following processing:

[0034] First, the case detection data were converted into two-dimensional slice image sequences, namely portable network graphics (PNG) format files with a size of 256×256. Then, the middle 15% of slices of each MRI were extracted to ensure that the images showing the anterior cruciate ligament were complete and clear. After that, images that could fully and clearly show the anterior cruciate ligament were screened. Finally, the image annotation tool Labelme was used to annotate them and obtain the corresponding mask labels. A total of 1500 images and their corresponding mask labels were obtained.

[0035] Step 2: Dataset expansion: Use image augmentation to expand the dataset.

[0036] In order to avoid the overfitting problem that may occur due to too few samples and enrich the diversity of the data, the present invention expands the dataset through three data augmentation methods: small-angle random rotation, translation, and noise addition, and finally obtains 6,000 images, which are divided into training set and test set with a ratio of 8:2.

[0037] Step 3: Improve the U-Net model and establish the DRH-UNet segmentation model. Based on the U-Net, design the RH-Block and introduce dilated convolution.

[0038] The DRH-UNet segmentation model structure obtained by improving the U-Net is as follows Figure 3 shown. It mainly consists of an encoder, a decoder, and skip connections with RH Blocks added. Each rectangular box in the figure corresponds to a multi-channel feature map, and the height, width, and number of channels are at the lower left and top of the rectangular box. The connection arrows between the encoder and the decoder represent skip connections, and the remaining different arrows represent different operation processes. The operation process represented by each arrow will be specifically pointed out when the encoder, decoder, and skip connections are introduced in detail below.

[0039] As Figure 3 shown, in the encoder stage of the DRH-UNet segmentation model, the two arrows in each layer represent standard convolution and dilated convolution in sequence. Among them, the dilated convolution has an additional hyperparameter of dilation rate compared to the standard convolution, which changes the sampling method of the convolution kernel when processing data; when the dilation rate is 1, the dilated convolution degenerates into ordinary convolution; when the dilation rate is greater than 1 and is n, there will be n - 1 data points spaced between adjacent sampling points of the convolution kernel, that is, sampling is performed at non-consecutive positions of the input matrix while keeping the physical size of the convolution kernel unchanged and not adding additional parameters. The dilation rate adopted in the embodiments of the present invention is 2, that is, when the convolution kernel samples on the input matrix, there will be one data point spaced between adjacent sampling points. And the downward arrow between adjacent layers represents the downsampling operation, that is, the max pooling layer. After the feature map is downsampled, its number of channels will double, and the height and width will be reduced by half. The specific numerical changes in the number of channels, height, and width have been marked in the figure. The processing result of each layer of the encoder is H BM-n (x):

[0040] H BM-n (x) = DownSample(DConv(Conv3(x)))#(1)

[0041] Among them, H BM-n (x) is the output of the nth layer of the encoder (n ≤ 4). When n is 1, x is the input of the image matrix; when n > 1, x is the output of the previous layer of the encoder. Conv3 is a 3×3 convolution, DConv is a dilated convolution with a dilation rate of 2, and DownSample is downsampling.

[0042] On the skip connections between the encoder and the decoder, the present invention designs RH Blocks, and different numbers of RH Blocks are added to the skip connections at different levels. The features extracted in the encoder stage are processed by the RH Blocks and then passed to the decoder end to be concatenated with the features at the corresponding levels. The structure of the RH Block is as follows Figure 4As shown, this module first calculates channel attention and spatial attention for the input features in sequence, obtaining channel attention features and spatial attention features, and multiplying them with their respective input features; then it obtains the preliminarily processed features through a convolutional layer and uses residual connection to fuse the preliminarily processed features with the original input; finally, it performs a non-linear transformation on the fused feature map through a parametric rectified linear unit (PReLU) activation function to obtain the final output. The channel attention mechanism, spatial attention mechanism, and PReLU activation function are defined as follows:

[0043] Mc(F c ) = σ(MLP(AvgPool(F c )) + MLP(MaxPool(F c ))) #(2)

[0044] Ms(F s ) = σ(f 7×7 [AvgPool(F s );MaxPool(F s )]) #(3)

[0045] PReLU(x) = max{x, ax} #(4)

[0046] Among them, in formulas (2) and (3), σ represents the Sigmoid activation function, MLP represents the multi-layer perceptron, AvgPool and MaxPool respectively represent global average pooling and global max pooling, f 7×7 represents a convolution operation with a convolution kernel size of 7×7, F c and F s respectively represent the feature inputs of the channel attention mechanism calculation function Mc and the spatial attention mechanism calculation function Ms; in formula (4), x represents the input of the non-linear activation function PReLU, and a represents the gradient of the negative value part. The processing result of each layer of skip connection is H TY-n (x):

[0047] H TY-n (x) = PReLU(Conv3(Ms(Mc(x))) + x) #(5)

[0048] Among them, H TY-n (x) is the output of repeating the RH Block n times on the nth layer of skip connection, Conv3 is a 3×3 convolution, Mc and Ms are channel and spatial attention calculations, PReLU is the activation function, and x is the original input.

[0049] In the decoder stage, the two arrows in each layer respectively represent standard convolution and dilated convolution. Different from this, the upward arrow between every two layers represents the upsampling operation, i.e., bilinear interpolation. After the feature map is upsampled, its number of channels remains unchanged, while its height and width double. However, this feature map will be concatenated with the encoder-side features passed through the skip connection. Therefore, the number of channels will still double compared with the previous layer, and the specific numerical changes in the number of channels, height, and width are also marked in the figure. The processing result of each layer of the decoder is H JM-m (x):

[0050] H JM-m (x) = DConv(Conv3(Concat(UpSample(x), H TY-n )))#(6)

[0051] where, H JM-m (x) is the output of the m-th layer of the decoder (m ≤ 4). When m = 1, x is the output after the output of the last layer of the encoder is processed by standard convolution and dilated convolution; when m > 1, x is the output of the previous layer of the decoder. Conv3 is a 3×3 convolution, DConv is a dilated convolution with a dilation rate of 2, Concat is the concatenation process, UpSample is upsampling, and HTY-n is the result after being processed by the RH Block on the n-th layer of the skip connection.

[0052] In addition, in the last layer of the decoder, after the feature map is processed by standard convolution and dilated convolution respectively, a standard convolution with a convolution kernel of 1×1 is adopted to reduce the number of feature channels to 2 layers, so as to obtain the segmentation result.

[0053] Step 4: Model training and hyperparameter optimization. The model is trained using the augmented dataset, and different combinations of hyperparameters are tried to train the DRH-UNet segmentation model with the optimal performance.

[0054] During the model training process, hyperparameter optimization is required, and the DSC coefficient and HD coefficient are used as evaluation metrics to evaluate the segmentation performance of the model. The DSC coefficient is used to measure the similarity between two sets and is used to measure the overlap degree between the prediction result and the ground truth label. The higher the DSC value, the more accurate the segmentation result. The HD coefficient is used to measure the accuracy of the segmentation boundary. The lower the HD value, the closer the boundary of the prediction result is to the ground truth label.

[0055] Such as Figure 2As shown, it shows the model training and validation flow chart in an embodiment of the present invention. The model training process includes: dividing the augmented dataset into a training set and a test set; training the constructed DRH-UNet segmentation model using the training set, and finding the optimal hyperparameter combination by parameter search, that is, trying different hyperparameter combinations during training, and fixing the weight file with the optimal performance of the DRH-UNet segmentation model. The hyperparameters at least include: the size of the batch, the number of epochs, the size of the learning rate, etc.

[0056] Step 5: Use the DRH-UNet segmentation model with the optimal performance obtained in Step 4 to perform segmentation testing on the test set to obtain the segmentation result.

[0057] As Figure 2 shown, after obtaining the weight file with the optimal performance, put it on the test set for segmentation testing. The segmentation results are as Figure 5 shown, which respectively show the comparison results of the DRH-UNet segmentation model and other models in terms of DSC coefficient and HD coefficient.

[0058] Experimental data table:

[0059]

[0060] The present invention performs image preprocessing and dataset augmentation on knee MRI, effectively preventing data imbalance, overfitting, etc., and enhancing the segmentation effect.

[0061] Based on the U-Net model, the present invention designs the RH Block to perform dual attention calculations on the channel dimension and spatial dimension of the image, and at the same time introduces dilated convolution to expand the model receptive field, enhancing the representation ability of the anterior cruciate ligament feature region and its structural relationship with other physiological tissues of the knee joint, and effectively improving the segmentation effect of the anterior cruciate ligament, providing preliminary pathological information for the diagnosis of the anterior cruciate ligament.

[0062] The above are only the preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. An anterior cruciate ligament image segmentation method that integrates dilated convolution and residual hybrid attention, characterized in that, It includes the following steps: Step 1: Dataset acquisition and image preprocessing; Step 2: Augment the preprocessed dataset using image enhancement methods; Step 3: Improve the U-Net model to establish the DRH-UNet segmentation model: Based on U-Net, design the RHBlock and introduce dilated convolution; Step 4: Model training and hyperparameter optimization: Use the augmented dataset to train the model, and at the same time try different combinations of hyperparameters to train the DRH-UNet segmentation model with the best performance; Step 5: Use the DRH-UNet segmentation model with the best performance obtained in Step 4 to perform segmentation tests on the test set to obtain the segmentation results.

2. The method according to claim 1, wherein The image preprocessing in Step 1 refers to screening out the images in which the anterior cruciate ligament is clearly and completely shown from the obtained sagittal knee MRI images, and using the image annotation tool Labelme to annotate them to obtain the mask labels, that is, the true labels for segmentation.

3. The method according to claim 1, wherein The augmentation of the preprocessed dataset using image enhancement methods in Step 2 means that for the dataset obtained after image preprocessing, data augmentation methods such as translation, small-angle rotation, and adding noise are used for augmentation to prevent overfitting caused by a small amount of data.

4. The method according to claim 1, wherein The improvement of the U-Net model in Step 3 to establish the DRH-UNet segmentation model mainly includes: Design the RHBlock on the skip connection between the encoder and decoder of the U-Net model. This module first calculates the channel attention and spatial attention for the input features in sequence to obtain the channel attention feature and spatial attention feature, and multiplies them with their input features respectively; then obtains the preliminarily processed features through a convolutional layer, and uses a residual connection to fuse the preliminarily processed features with the original input; finally, performs a non-linear transformation on the fused feature map through the parametric rectified linear unit (PReLU) activation function to obtain the final output; where the channel attention mechanism, spatial attention mechanism, and PReLU activation function are defined as follows: Mc(F c ) = σ(MLP(AvgPool(F c )) + MLP(MaxPool(F c ))) #(2) Ms(F s ) = σ(f 7×7 [AvgPool(F s );MaxPool(F s )])#(3) PReLU(x) = max{x, ax} #(4) Among them, in formulas (2) and (3), σ represents the Sigmoid activation function, MLP represents the multi-layer perceptron, AvgPool and MaxPool represent global average pooling and global max pooling respectively, f 7×7 represents a convolution operation with a convolution kernel size of 7×7, F c and F s respectively represent the feature inputs of the channel attention mechanism calculation function Mc and the spatial attention mechanism calculation function Ms; in formula (4), x represents the input of the non-linear activation function PReLU, and a represents the gradient of the negative value part; therefore, the output of the RH Block is H TY : H TY (x) = PReLU(Conv3(Ms(Mc(x))) + x) #(5) Where Conv3 is a 3×3 convolution, Mc and Ms are the channel and spatial attention calculations, PReLU is the activation function, and x is the original input.

5. The method according to claim 4, characterized in that The improvement of the U-Net model in Step 3 to establish the DRH-UNet segmentation model mainly includes: In each layer of the encoder and decoder of the U-Net model, replace the second ordinary convolution with a dilated convolution; Compared with the ordinary convolution, the dilated convolution introduces a hyperparameter of dilation rate, which changes the sampling method of the convolution kernel when processing data; When the dilation rate is 1, the dilated convolution degenerates into an ordinary convolution; When the dilation rate is greater than 1 and is n, there will be n - 1 data points between adjacent sampling points of the convolution kernel, sampling at non-consecutive positions in the input matrix, while keeping the physical size of the convolution kernel unchanged and not adding additional parameters.

6. The method according to claim 1, wherein The model training and hyperparameter optimization in step 4 refer to finding the optimal hyperparameter combination during the model training process by trying different hyperparameter combinations, which can be learning rate, batch size, regularization parameter, and training the DRH-UNet segmentation model with the optimal performance.

7. The method according to claim 1, wherein The segmentation test in step 5 refers to using the DRH-UNet segmentation model with the optimal performance obtained in step 4 to perform segmentation tests on the test set, and obtaining the segmentation results of relevant performance evaluation metrics: DSC coefficient and HD coefficient.

Images