An automatic bone imaging segmentation method based on improved U-Net and adaptive hybrid loss

Abstract

The application discloses a kind of automatic segmentation methods of bone imaging based on improved U-Net and adaptive hybrid loss, for bone imaging image signal-to-noise ratio low, small lesion is difficult to extract and artificial delineation lesion time-consuming and so on.A multi-scale dense connection convolution block is used as a basic convolution unit, and an automatic segmentation network of bone imaging based on improved U-Net is proposed.In the encoding part of U-Net, a multi-scale dense connection convolution block and a maximum pooling layer are used to extract features from bone imaging lesions.In the decoding part, transposed convolution is used for upsampling.In the network training, adaptive hybrid loss is used to optimize the segmentation result.The automatic segmentation method of bone imaging lesions disclosed in the application is better than the original segmentation method, effectively improving the segmentation accuracy of bone imaging lesions.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of deep learning, and particularly relates to a bone imaging automatic segmentation method based on improved U-Net and adaptive hybrid loss. BACKGROUND

[0002] With the development of medical technology, nuclear medical image recognition technology has become an important means of disease diagnosis. SPECT bone imaging has the characteristics of high sensitivity, strong practicability, low cost and large imaging field of view, and can show malignant tumor lesions in the early stage, and become the most important examination method for evaluating bone metastatic tumors. Doctors can judge the bone metastatic tumor lesions in the image by relying on rich reading experience and SPECT whole body bone imaging and other auxiliary information, but there are also problems of strong subjectivity, low repeatability and insufficient analysis. In order to solve these problems, in recent years, many researchers have carried out research on image processing technology and machine learning method. However, the recognition accuracy of the method based on digital image processing is not high, and the traditional machine learning algorithm has improved the recognition accuracy, but the calculation efficiency and specificity are low.

[0003] Deep learning, as a branch of machine learning, is widely used in speech recognition, computer vision, medical treatment and other fields. The literature “Nikolaos P, Elpiniki P, Athanasios A, et al. Bone metastasis classification using whole body images from prostate cancer patients based on convolutional neural networks application[J]. Plos One, 2020, 15(8): 1-29.” classifies whole-body bone imaging images into malignant and healthy categories by selecting and fine-tuning CNN hyperparameters, with classification accuracy and sensitivity reaching 97.38% and 95.80%, respectively. The literature “CHARIS N, DIMITRIOS E, NIKOLAOS P, et al. A lightweight Convolutional Neural Network architecture applied for bone metastasis classification in Nuclear Medicine: A Case Study on Prostate Cancer Patients[J]. Healthcare, 2020, 8(4): 1-13.” uses 778 bone imaging of prostate cancer men as a dataset, reduces the amount of calculation by using floating-point operation and improved CNN, and compares the improved CNN with VGG16, ResNet, Inception and other structures, proving that the algorithm is superior to the mainstream algorithm at that time in identifying bone metastasis. The literature “Su N, Ye HM, Wang XY. Research on multi-sensor docking pose fusion algorithm based on Adam optimization[J]. Modern Manufacturing Technology and Equipment, 2022, 58(2): 35-37.” optimizes the iteration of docking pose fusion solution by using Adam optimizer, improving the speed and accuracy of docking pose fusion solution.The literature "Zheng Z, Liu L X, Chen X Y, et al. Construction of bisection model of SPECT bone scan image based on VGGNet [C] / / IEEE of the International Conference on Artificial Intelligence and Industrial Design (AIID), May 28-30, 2021, Guangzhou, China. New York: IEEE, 2021: 150-154." expands the data by rotating, translating, mirroring and other processing, and compares the VGG network framework of different depths, and the accuracy, recall rate and F1 score of benign and malignant cases are all 99%. The accuracy, specificity and recall rate of using CNN for SPECT bone imaging classification are higher than traditional algorithms. At present, the method based on deep learning mainly focuses on the research of bone imaging image classification, and the research on bone imaging lesion segmentation method is very few. SUMMARY

[0004] The application discloses a kind of bone imaging automatic segmentation method based on improved U-Net and adaptive hybrid loss, comprising the following steps: (1) division training set and test set;(2) the combination of dense connection, multi-scale convolution and dense connection method is carried out, and multi-scale dense connection convolution block is proposed;(3) with multi-scale dense connection convolution block as basic unit, design an end-to-end U-Net model of coding and decoding;(4) the segmentation effect of bone imaging is optimized using weighted loss function;(5) training network model, and save the parameter of model after training is completed;(6) the bone imaging lesion segmentation graph in test set is output using the final model saved.This bone imaging automatic segmentation method based on improved U-Net and adaptive hybrid loss disclosed in the application can better extract bone imaging features and segment lesion areas by means of dense residual attention convolution block, U-Net model and weighted loss function containing Lovasz Softmax.

[0005] The technical scheme provided by the application is as follows: a bone imaging automatic segmentation method based on improved U-Net and adaptive hybrid loss, characterized by comprising the following steps:

[0006] To achieve the above technical purpose, the application adopts the following technical scheme:

[0007] A bone imaging automatic segmentation method based on improved U-Net and adaptive hybrid loss, characterized by comprising the following steps:

[0008] Step 1: The pre-processed bone image and the expert-labeled lesion mask image are used as the training set and the test set, and are divided into training set and test set, the specific steps are as follows:

[0009] (1) Normalize the bone image, convert the original bone image file to a visible gray image;

[0010] (2) Manually label the lesions in the bone image, and convert the labeled image to a mask image;

[0011] (3) Perform threshold denoising, median filtering, opening and closing operations, and cropping on the bone image to obtain the shoulder to pelvic region in the lesion set;

[0012] (4) Adjust the image to N×N pixel size, N is a positive integer, count the number of different class pixel points in the mask image, denoted as pix i , i is the class number;

[0013] (5) Divide the pre-processed bone image dataset into training set and test set according to the C:V ratio, C and V are positive integers less than 10;

[0014] Step 2: Construct a multi-scale dense connection convolution block, which is divided into multi-scale residual convolution and dense connection. The multi-scale residual convolution contains three branches, and the three branches use 1×1, 3×3, and two consecutive 3×3 convolutions to extract features of lesions of different scales. Then the results of the three branch convolutions are spliced, and a 1×1 convolution is used to compress the channel number and reduce the network operation amount. Finally, the features input by the 1×1 convolution are fused with the spliced and compressed features to restore the original part of the features. The calculation process from input to output in the module is as follows:

[0015] x1=Conv 1x1 (x in )

[0016] x2=Conv 3x3 (Conv 1x1 (x in ))

[0017] x3=Conv 3x3 (Conv 3x3 (Conv 1x1 (x in )))

[0018] x c =concat(x1,x2,x3)

[0019] x out =x in +xc

[0020] Conv 1×1 (·), Conv 3×3 (·) represent 1x1 convolution and 3x3 convolution operation respectively; x in is the input bone image of the multi-scale residual convolution block; x1, x2, x3 are the results after convolution calculation, that is, the bone image feature map; x out is the output feature map of the multi-scale residual convolution; concat represents feature concatenation, assuming that x1 is c1xhwx2 is c2xhwx3 is c3xhwxthen the calculation result of concat(x1, x2, x3) is (c1+c2+c3) xhwx+ represents fusion; x c is the feature map after feature concatenation of x1, x2, x3;

[0021] i multi-scale residual convolutions are adopted, and the dense connection method is used between each convolution, and the input of the i-th layer convolution is as follows:

[0022] x i = H i ([x0, x1, …, x i-1 ]), i∈[1, M]

[0023] In the formula: i is a positive integer greater than 1; x i represents the input of the i-th multi-scale residual convolution; [x0, x1, …, x i-1 ] represents the feature after concatenation of all feature maps before the i-th layer; H i represents a nonlinear mapping, that is, the combination of batch normalization and ReLU activation function operation;

[0024] Step 3: design an improved U-Net model, characterized in that the encoding process is composed of MxM, 5 multi-scale dense connection convolution blocks of different scales, M is a positive integer, and each convolution block is composed of 5 convolution layers and 1 pooling layer; the decoding process is symmetrical to the encoding process, composed of MxM, 4 groups of multi-scale dense connection convolution blocks of different scales, each residual block is composed of 1 deconvolution layer and 5 convolution layers, and the bone image lesion features extracted in the corresponding encoding stage are fused;

[0025] Step 4: use the training set obtained in step 1 to train the network model built in step 3, and design a self-adaptive hybrid loss function to output the training loss, and the Adam optimizer optimizes the network training process;

[0026] The adaptive hybrid loss function adopts the combination of cross-entropy loss and Lovasz Softmax loss, and its expression is as follows:

[0027] L H =w1(L ce +w2L ls )

[0028] In the formula, L ce represents cross-entropy loss, L ls represents Lovasz Softmax loss, w1 and w2 represent adjustment factors, w1 is used to adaptively adjust the weight of different categories of pixel points in each bone image, w2 is used to adjust the weight of Lovasz Softmax loss, and w2 is a constant.

[0029] The calculation formula of w1 is as follows:

[0030] i∈c

[0031] In the formula, c represents the number of categories, which is a positive integer; pix i represents the number of pixel points of the i-th category; represents the number of all pixel points; represents the total number of pixel points excluding the background; is a penalty term;

[0032] The cross-entropy calculation formula is as follows:

[0033]

[0034] In the formula, p represents the real label; q represents the predicted probability; and H(p, q) represents the difference between p and q.

[0035] The Lovasz Softmax loss calculation formula is as follows:

[0036]

[0037] In the formula, C is the number of categories; is the Jaccard operation;

[0038] The calculation formula of m(c) is as follows:

[0039]

[0040] In the formula, f i (c) represents the output probability distribution of the softmax function;

[0041] The average intersection-over-union calculation formula is as follows:

[0042]

[0043] In the formula: TP is true positive, indicating that the positive sample is predicted as a positive sample; FP is false positive, indicating that the positive sample is predicted as a negative sample; FN is true negative, indicating that the positive sample is predicted as a negative sample; TN is false negative, indicating that the negative sample is predicted as a negative sample, and TP, FP, FN and TN are all positive integers;

[0044] Step 5: The test set data pretreated in step 1 is sent into the network model built in steps 2 and 3, the loss value is calculated by using the adaptive hybrid loss function designed in step 4, and the segmentation effect of the model is evaluated by the precision rate and the average intersection over union, the higher the average intersection over union value, the better the segmentation effect of the model, and the final network model is saved, which is recorded as Improved U-Net model;

[0045] Step 6: The bone image in the test set is input into the Improved U-Net model, and the segmentation image is output.

[0046] The application discloses a kind of based on improved U-Net and adaptive hybrid loss's bone image automatic segmentation method, this method designs multi-scale dense connection convolution block instead of original convolution block in U-Net, and as basic convolution unit, it is proposed that a kind of based on improved U-Net's bone image automatic segmentation network, in the encoding part of U-Net, using multi-scale dense connection convolution block and maximum pooling layer are introduced to feature extraction to bone image focus, using transpose convolution for upsampling in decoding part, using adaptive hybrid loss optimization segmentation result when network training, adaptive hybrid loss uses Lovasz Softmax loss to constrain cross-entropy loss, and the designed loss can be adaptively weighted weight to hybrid loss function according to the pixel number of each bone image focus, the bone image focus automatic segmentation method disclosed in the application optimizes the training effect of model, improves the segmentation ability of small focus to model.

[0047] Beneficial effects:

[0048] Compared with the current mainstream medical influence denoising method, the application has the following beneficial effects:

[0049] (1) a kind of based on improved U-Net and adaptive hybrid loss's bone image automatic segmentation method, can effectively segment out bone image focus area;

[0050] (2) compared with conventional medical focus segmentation method, the segmentation effect of the bone image automatic segmentation method based on improved U-Net and adaptive hybrid loss is better than the current mainstream segmentation algorithm, has certain practical application value; DETAILED DESCRIPTION

[0051] Figure 1For the flowchart of the present application, the main process is preprocessing, establishing training set and test set, then training the model, saving the model after training, and finally testing the model;

[0052] Figure 2 For the bone imaging preprocessing flowchart;

[0053] Figure 3 For the cropping and denoising process chart;

[0054] Figure 4 For the multi-scale residual convolution chart; wherein, figure (a) is a 3x3 convolution used in the original U-Net model, and figure (b) is a multi-scale residual convolution constructed;

[0055] Figure 5 For the multi-scale dense connection chart; wherein, IR Conv is a multi-scale residual convolution;

[0056] Figure 6 For the improved U-Net model constructed by the present application; (a) is the pre-training part; (b) is the encoder; (c) is the decoder; (d) is the feature fusion output part;

[0057] Figure 7 For the partial segmentation result chart; Image is bone imaging, Label is the lesion mask chart labeled by experts; the remaining columns are the segmentation results of the mainstream segmentation model, and Improved U-Net is the model proposed by the present method. DETAILED DESCRIPTION

[0058] The present application will be described in detail below with reference to the accompanying drawings, Figure 1 For the overall flowchart of the present application, the specific implementation of the present application is as follows:

[0059] Step 1: The preprocessed bone imaging and the lesion mask chart labeled by experts are used as the training set and the test set, and are divided into training set and test set, and the specific steps are as follows:

[0060] (1) Convert to grayscale image by normalization, each example containing two pictures of human front and back;

[0061] (2) Manually label the lesions of bone imaging, and convert the labeled image into a mask image;

[0062] (3) Improve the signal-to-noise ratio of bone imaging, get the shoulder to pelvic region of bone imaging by cropping, and adjust the image to 288x288 pixels;

[0063] (4) Adjust the image to NxN pixel size, N is a positive integer, count the number of different class pixel points in the mask image, denoted as pix i , i is the class number;

[0064] (5) The original drawing and the lesion mask image obtained by marking are taken as a set of data, the preprocessed gray data is taken as a label, and the data is divided into a training set and a test set according to a ratio of 8:2;

[0065] Step 2: A multi-scale dense connection convolution block is constructed, the multi-scale dense connection convolution block is divided into a multi-scale residual convolution and a dense connection, the multi-scale residual convolution contains three branches, 1x1, 3x3 and two 3x3 convolutions in succession are used in the three branches to extract lesion features of different scales; then the results of the three branch convolutions are feature spliced, and 1x1 convolution is used to compress the channel number and reduce the network operation amount, finally the features input by the 1x1 convolution are fused with the spliced and compressed features to restore the original part of the features, and the calculation process from input to output in the module is as follows:

[0066] x1 = Conv 1x1 (x in )

[0067] x2 = Conv 3x3 (Conv 1x1 (x in ))

[0068] x3 = Conv 3x3 (Conv 3x3 (Conv 1x1 (x in )))

[0069] x c = concat(x1,x2,x3)

[0070] x out = x in +x c

[0071] In the formula, Conv 1×1 (·), Conv 3×3 (·) respectively represent 1x1 convolution and 3x3 convolution operation, x in is a bone image input into the multi-scale residual convolution block, x1, x2 and x3 are respectively results after convolution calculation, that is, bone image feature maps, x out is an output feature map of the multi-scale residual convolution; concat represents feature splicing, x1 is c1xhwx2 is c2xhwx3 is c3xhwxconcat(x1,x2,x3) has a calculation result of (c1+c2+c3)xhwx+ represents fusion; x c is a feature map after feature splicing of x1, x2 and x3;

[0072] i multi-scale residual convolutions are used, and each convolution is connected by dense connections. The input of the i-th convolution is shown below:

[0073] x i =H i ([x0,x1,…,x i-1 ]), i∈[1,4]

[0074] In the formula: i is a positive integer greater than 1, and in this example, i is 4; x i Represents the input of the i-th layer multi-scale residual convolution; [x0, x1, ..., x i-1 ] represents the features obtained by concatenating all feature maps up to the i-th layer; H i This represents a nonlinear mapping, which is a combination of batch normalization and ReLU activation function operations;

[0075] Step 3: Design an improved U-Net model, characterized by an encoding process consisting of M×M... It consists of 5 multi-scale densely connected convolutional blocks of different scales, where M is a positive integer. In this example, M is 288. Each convolutional block consists of 5 convolutional layers and 1 pooling layer. The decoding process is symmetrical to the encoding process, consisting of M×M... The residual block consists of four groups of multi-scale densely connected convolutional blocks of different scales. Each group of residual blocks consists of one deconvolutional layer and five convolutional layers, and is fused with bone imaging lesion features extracted in the corresponding coding stage.

[0076] Step 4: Use the training set obtained in Step 1 to train the network model built in Step 3, and design an adaptive hybrid loss function to output the training loss. Use the gradient descent algorithm with adaptive learning rate to optimize the network training process.

[0077] The adaptive hybrid loss function combines cross-entropy loss with Lovasz Softmax loss, and its expression is as follows:

[0078] L H =w1(L ce +w2L ls )

[0079] In the formula: L ce L represents the cross-entropy loss. ls w1 and w2 represent the Lovasz Softmax loss. w1 and w2 are adjustment factors designed in this method. w1 is used to adaptively adjust the weight of different categories of pixels in each bone image. w2 is used to adjust the weight of the Lovasz Softmax loss. w2 is a constant, which is 10 in this example.

[0080] The formula for calculating w1 is:

[0081] i∈c

[0082] wherein: c represents the number of categories, which is a positive integer; pix i represents the number of pixel points in the i-th category; represents the number of all pixel points; represents the total number of pixel points except the background; is a penalty term;

[0083] The cross-entropy calculation formula is:

[0084]

[0085] wherein: p represents the real label; q represents the predicted probability; H(p, q) represents the difference between p and q;

[0086] The Lovasz Softmax loss calculation formula is:

[0087] c∈C

[0088] wherein: C is the number of categories; is the Jaccard operation;

[0089] The calculation formula of m(c) is:

[0090]

[0091] wherein: f i (c) represents the softmax function output probability distribution;

[0092] The average intersection over union calculation formula is:

[0093]

[0094] wherein: TP is true positive, representing a positive sample predicted as a positive sample; FP is false positive, representing a positive sample predicted as a negative sample; FN is true negative, representing a positive sample predicted as a negative sample; TN is false negative, representing a negative sample predicted as a negative sample, and TP, FP, FN, and TN are all positive integers;

[0095] Step 5: The test set data preprocessed in step 1 is sent into the network model built in steps 2 and 3, the loss value is calculated by using the adaptive hybrid loss function designed in step 4, and the segmentation effect of the model is evaluated by the average intersection over union, the higher the average intersection over union value, the better the segmentation effect of the model, and the final network model is saved, which is recorded as the Improved U-Net model;

[0096] Step 6: input the bone imaging in the test set into the Improved U-Net model, and output the segmented image.

[0097] The following table presents the segmentation effect of bone imaging lesions of each model and the method of the present application, as shown in Table 1:

[0098] Table 1: Average intersection over union value of different algorithms

[0099]

[0100]

[0101] As can be seen from the above table, the average intersection over union value of the bone imaging automatic segmentation method based on the improved U-Net and the adaptive hybrid loss proposed by the present application is the highest, and the effect is the best.

Claims

1. An automatic bone segmentation method based on improved U-Net and adaptive hybrid loss, characterized in that... Includes the following steps: Step 1: Preprocess the bone scan image. The specific processing method is as follows: (1) Normalize the bone scintigraphy and convert the original bone scintigraphy file into a visible grayscale image; (2) Manually mark the lesions on the bone scintigraphy image and convert the marked image into a mask image; (3) Perform threshold denoising, median filtering, opening and closing operations and cropping on the bone scintigraphy, and select the shoulder to pelvic region where lesions are concentrated. (4) Adjust the image to an N×N pixel size, where N is a positive integer, and count the number of pixels of different categories in the mask image, denoted as pix. i where i is the number of categories; (5) Divide the preprocessed bone imaging dataset into training and test sets according to the C:V ratio, where C and V are both positive integers less than 10; Step 2: Construct a multi-scale densely connected convolutional block. The multi-scale densely connected convolutional block is divided into two parts: multi-scale residual convolution and dense connections. The multi-scale residual convolution contains three branches. The three branches use 1×1, 3×3 and two consecutive 3×3 convolutions to extract lesion features at different scales. The results of the three convolutional branches are then concatenated, and a 1×1 convolution is used to compress the number of channels and reduce the computational load of the network. Finally, a residual structure is used to fuse the features input by the 1×1 convolution with the concatenated and compressed features to recover some of the original features. The computational flow from input to output in the module is as follows: x1=Conv 1x1 (x in ) x2=Conv 3x3 (Conv 1x1 (x in )) x3=Conv 3x3 (Conv 3x3 (Conv 1x1 (x in ))) x c =concat(x1,x2,x3) x out =x in +x c In the formula: Conv 1×1 (·), Conv 3×3 (·) represents 1×1 convolution and 3×3 convolution operations respectively, x in The input is a bone scintillation image of a multi-scale residual convolution block. x1, x2, and x3 are the results after convolution, i.e., bone scintillation feature maps. out This is the output feature map of multi-scale residual convolution; `concat` indicates feature concatenation. Let x1 be c1×h×w, x2 be c2×h×w, and x3 be c3×h×w, then the result of `concat(x1,x2,x3)` is (c1+c2+c3)×h×w; `+` indicates fusion; x c The feature map after feature concatenation of x1, x2, and x3; We employ i multi-scale residual convolutions, each connected by dense connections. The input to the i-th convolutional layer is: x i =H i ([x0,x1,…,x i-1 ]), i∈[1,M] In the formula: x i Represents the input of the i-th layer multi-scale residual convolution, [x0, x1, ..., x...]. i-1 ] represents the features obtained by concatenating all feature maps up to the i-th layer; H i This represents a nonlinear mapping, which is a combination of batch normalization and ReLU activation function operations; Step 3: Design an improved U-Net model, characterized by an encoding process consisting of M×M... The system consists of five multi-scale densely connected convolutional blocks of different scales, where M is a positive integer. Each convolutional block comprises five convolutional layers and one pooling layer. The decoding process is symmetrical to the encoding process, consisting of M×M... The residual block consists of a group of multi-scale densely connected convolutional blocks of different scales. Each group of residual blocks consists of one deconvolutional layer and five convolutional layers, and is fused with bone imaging lesion features extracted from the corresponding coding stage. Step 4: Use the training set obtained in Step 1 to train the network model built in Step 3, and construct an adaptive hybrid loss function to calculate the training loss. Use the Adam optimizer to optimize the network training process. The adaptive hybrid loss function combines cross-entropy loss with Lovasz Softmax loss, and its expression is as follows: L H =w1(L ce +w2L ls ) In the formula: L ce Represents cross-entropy loss; L ls represents the Lovasz Softmax loss; w1 and w2 are the design adjustment factors, w1 is used to adaptively adjust the weight of different categories of pixels in each bone image, w2 is used to adjust the weight of the Lovasz Softmax loss, and w2 is a constant greater than 0; The formula for calculating w1 is: In the formula: c represents the number of categories, which is a positive integer; pix i Represents the number of pixels in the i-th category; Represents the total number of pixels; This represents the total number of pixels excluding the background. This is a penalty item; The formula for calculating cross-entropy is: In the formula: p represents the true label; q represents the predicted probability; H(p,q) represents the difference between p and q; The Lovasz Softmax loss calculation formula is as follows: In the formula: C is the number of categories; For Jaccard operations; The formula for calculating m(c) is: In the formula: f i (c) represents the output probability distribution of the softmax function; The formula for calculating the average intersection-union ratio is: In the formula: TP is a true positive, representing a positive sample predicted as a positive sample; FP is a false positive, representing a positive sample predicted as a negative sample; FN is a true negative, representing a positive sample predicted as a negative sample; TN is a false negative, representing a negative sample predicted as a negative sample; TP, FP, FN, and TN are all positive integers. Step 5: Input the preprocessed test set data from Step 1 into the network model built in Step 3, calculate the loss value using the adaptive hybrid loss function designed in Step 4, and evaluate the segmentation effect of the model by the mean intersection-over-union ratio (MIR). The higher the MIR value, the better the segmentation effect of the model. Save the final network model, denoted as the Improved U-Net model. Step 6: Input the bone scan images from the test set into the Improved U-Net model, and output the segmented images.

Images