A pancreatic tumor image segmentation method based on locally linear embedding interpolation neural network

Abstract

The present invention relates to the field of computer medical image processing, and proposes a pancreatic tumor image segmentation processing method based on a local linear embedded interpolation neural network. The method comprises the following steps: inputting a local interpolation network to interpolate the input 3D tumor slice and the corresponding tumor label slice; inputting a multi-scale cascade segmentation network to perform tumor segmentation, and visualizing the result. The present invention is easy to implement, improves the image quality and the number of tumor slices, reduces the tumor differences between adjacent slices, and avoids missed detection as much as possible. The present invention can effectively alleviate the fuzzy labeling problem of tumor boundaries, and process different tumor image features under limited computing conditions, significantly improving image resolution.

Description

Technical Field

[0001] The present invention relates to the field of computer medical image processing. Background Art

[0002] Pancreatic tumor is a disease that is difficult to diagnose and treat, with a one-year survival rate of less than 20% and a five-year survival rate of 9%.

[0003] CT / MR has higher soft tissue resolution and can identify contrast patterns, becoming one of the conventional methods for diagnosing pancreatic tumors. However, due to the limitation of scanner accuracy, the spacing between adjacent slices is large. Even for experienced radiologists, it takes a lot of time to detect tumors by carefully comparing adjacent slices, and for small tumors, missed detection is inevitable. In addition, the large spacing of thick CT / MR exacerbates the difficulty of automatic segmentation of pancreatic tumors. Therefore, using inter-slice interpolation, the automatic segmentation of pancreatic tumors on CT / MR images can not only help doctors determine the size, morphology, and location of the tumor, but also provide a reliable basis for subsequent diagnosis.

[0004] In recent years, deep learning has been widely used in medical image processing and has made breakthrough progress in pancreatic tumor segmentation. Y.Zhou et al. adopted a dual-path network in which parallel streams were interconnected with hyperconnections to perform dense information exchange between arterial and venous phase CT images. The study reported that 63.94% of 239 cases of pancreatic ductal adenocarcinoma (PDAC) were diagnosed as pancreatic cancer. Similarly, Y.Lang et al. adopted a square window-based convolutional neural network (CNN) architecture with three convolutional layer blocks. It fuses three phases of DCE-MRI, namely pre-contrast, arterial and venous images. However, multi-phase data is difficult to obtain, and the above model cannot be used for single-phase pancreatic tumor segmentation tasks. Multi-phase data also brings complexity to the model and requires a lot of computing resources. Due to the difficulty of tumor segmentation on medical images, especially pancreatic tumors, more and more studies use interpolation to enhance images. F.Isensee et al. used automatic interpolation as a preprocessing method to alleviate the anisotropy of voxels in segmentation tasks. In addition, more and more deep learning-based segmentation methods focus on the improvements brought by interpolation. J. Zhang et al. proposed a semi-automatic method that combines prior knowledge of Gaussian weighted Euclidean distance and nonlinear interpolation to enhance MR for leg segmentation. Summary of the invention

[0005] In view of this, the purpose of the present invention is to overcome the problems of low resolution of large-spacing thick CT / MR generated by the prior art, poor continuity between CT / MR image slices, and uncertainty caused by fuzzy boundaries of small tumors, and discloses a pancreatic tumor segmentation method based on a locally linear embedded interpolation neural network.

[0006] Technical solution of the present invention:

[0007] A pancreatic tumor segmentation method based on a local linear embedding interpolation neural network comprises the following steps:

[0008] Step 1: Dataset preparation:

[0009] The dataset is obtained from the hospital, including 3D MR slice images and annotated tumor segmentation labels as the dataset.

[0010] Step 2: Training and Optimization:

[0011] 2.1 Divide the original slice images and labels obtained in step 1 into groups of two, represented as x i-1 and x i+1 , and input into the inter-slice interpolation network to be trained (existing structure, referred to as "Network 1"). The structural similarity loss between the original slice image and the interpolated slice image is used as the evaluation index to train "Network 1". "Network 1" can obtain the inter-slice interpolation slice, represented as x i , making the 3D MR slice sequences more continuous.

[0012] 2.2 Next, merge the slice x obtained by interpolation in step 2.1 i and slice x i-1 、x i+1 A slice sequence is obtained, and the tumor label slices are input into a probability map generator to generate a probability map of the tumor label, which is provided to step 2.3;

[0013] 2.3 Input the MR slice sequence into the multi-scale cascade network to be trained (existing structure, referred to as "Network 2"), and output the tumor probability map according to the characteristics of the tumor. The mean square error between the tumor probability map output by "Network 2" and the tumor label probability map provided in step 2.2 is used as the evaluation index to train "Network 2".

[0014] Step 3 Predict pathology sections and visualization:

[0015] 3.1 Input the pathological slices into the trained "network 1" for interpolation between slices, and obtain the interpolated pathological slice sequence to provide step 3.2;

[0016] 3.2 Input the interpolated pathological slice sequence into the trained "Network 2" and output the tumor probability map. According to the set threshold (the probability boundary value for determining whether the location is a tumor), the tumor in the MR image is segmented and the results are visualized.

[0017] The beneficial effects of the present invention are:

[0018] The pancreatic tumor segmentation method based on the interpolation neural network with local linear embedding proposed in the present invention is easy to implement, improves the image quality and the number of tumor slices, reduces the tumor differences between adjacent slices, and avoids missed detection as much as possible. The interpolated image reduces the impact of tumor changes, such as changes in tumor size and position. The present invention can effectively alleviate the fuzzy labeling problem of tumor boundaries, and process different tumor features under limited computing conditions, significantly improving image resolution. As a medical detection technology solution, it can provide an accurate basis for doctors' subsequent treatment. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a general flow chart of implementing tumor segmentation on 3D pancreatic MR images according to the present invention.

[0020] Figure 2 It is a flow chart of implementing training optimization of the present invention.

[0021] Figure 3 It is a flow chart of the present invention for implementing the prediction of pathological sections and visualization.

[0022] Figure 4(a) is a workflow diagram for training “Network 1” and “Network 2”.

[0023] Figure 4(b) is the network structure of the inter-slice interpolation neural network.

[0024] Figure 4(c) is the network structure of a multi-scale cascade neural network.

[0025] FIG4( d ) is a workflow diagram for tumor segmentation prediction.

[0026] Figure 5 (a)(b) are schematic diagrams of four cases of pancreatic tumors.

[0027] Figure 6 (a) is a schematic diagram of a CT / MR slice containing a tumor.

[0028] Figure 6 (b) Schematic diagram of the tumor location (marked in white) marked by the surgeon.

[0029] Figure 6 (c) is a tumor location probability map generated by the present invention.

[0030] Figure 7 It is a schematic diagram of the inter-slice interpolation result of the present invention.

[0031] Figure 8 It is a schematic diagram of pancreatic tumor segmentation results of three cases achieved by the present invention. DETAILED DESCRIPTION

[0032] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is described in detail below in conjunction with the accompanying drawings and embodiments. It should be noted that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. Products that can achieve the same functions are equivalent replacements and improvements, and are all included in the protection scope of the present invention.

[0033] The present invention is further described below by way of embodiments in conjunction with the accompanying drawings. Figure 1 The overall flow chart of the present invention is given, which is divided into two steps: inputting the input MR tumor slices and the corresponding tumor label slices into the local interpolation network for interpolation and inputting the multi-scale cascade segmentation network for tumor segmentation, and visualizing the results.

[0034] Figure 1 It is a general flow chart of implementing tumor segmentation on 3D pancreatic MR images according to the present invention.

[0035] Figure 2 It is a flowchart of the present invention for implementing interpolation of input 3D tumor slices and corresponding tumor label slices input into a local interpolation network.

[0036] Figure 3 It is a flow chart of implementing tumor segmentation and visualizing the results in the present invention.

[0037] Figure 4(a) is a workflow diagram for training “Network 1” and “Network 2”. Figure 4(b) is the network structure of the inter-slice interpolation neural network. Figure 4(c) is the network structure of the multi-scale cascade neural network. Figure 4(d) is a workflow diagram for tumor segmentation prediction.

[0038] The workflow is shown in Figure 4(a):

[0039] Step 1: Dataset preparation. The specific implementation process is as follows:

[0040] The dataset is obtained from the hospital, including 3D MR slice images and annotated tumor segmentation labels as the dataset.

[0041] Step 2: training optimization. The specific implementation process is as follows:

[0042] 2.1 The original slice images and labels obtained in step 1 are divided into groups of two, and are simultaneously input into the inter-slice interpolation network to be trained (existing structure, referred to as "Network 1"). The structural similarity loss between the original slice image and the interpolated slice image is used as an evaluation index to train "Network 1". "Network 1" can obtain inter-slice interpolation slices, making the 3D MR slice sequence more continuous.

[0043] 2.1.1 “Network 1” consists of a dense connection module, a sub-pixel convolution module, a downsampling module, and a local linear embedding module, as shown in Figure 4(b), where the input slice is represented as x i ={x i1 ,x i2 ,…,x id}, the output interpolation slice is represented as

[0044] 2.1.2 The slices of tumor MR images and tumor labels are divided into groups of three in order. The middle slice is used as the label. The remaining two slices are input into "Network 1". After passing through the dense connection module and the sub-pixel convolution module, upsampling is performed and the results are further input into the dense connection module and the sub-pixel convolution module.

[0045] 2.1.3 Upsample the result of step 2.1.2, input the result into the dense connection module and downsample it through 2x2 convolution;

[0046] 2.1.4 Input the result of step 2.1.3 into the dense connection module and downsample it through 2x2 convolution. Input the sampling result into the dense connection module to obtain the input slice x i The K nearest neighbors of each dimension value of are Represents the K nearest neighbors of the jth dimension value of the i-th slice;

[0047] 2.1.5 Linearly combine the results of step 2.1.4 to obtain interpolated slices;

[0048] 2.1.5.1 First calculate the local covariance matrix

[0049]

[0050] in represents a system consisting of K identical x ij The vector composed of x ij represents the value of the jth dimension of the i-th slice, represents the K nearest neighbors of the jth dimension of the i-th slice obtained in step 1;

[0051] 2.1.5.2 Next, calculate the weights of the K nearest neighbors

[0052]

[0053] 1 of them K =[1,1,…,1]∈R K , S ij represents the covariance matrix obtained in step 1.2.1;

[0054] 2.1.5.3 Calculate interpolated slices by weighted linear sum of K nearest neighbors The values ​​of each dimension

[0055]

[0056] in represents the weight of the neighboring points obtained in step 1.2.2, represents the K nearest neighbors of the jth dimension of the i-th slice obtained in step 1;

[0057] 2.1.6 Compare the interpolated slice with the labeled slice and calculate the loss value. The calculation method is as follows:

[0058]

[0059] in, and Y represent the interpolated slice and label slice respectively. and μ Y They are and the pixel mean of Y, and σ Y They are and the pixel standard deviation of Y, express and the pixel covariance of Y, C1, C2, C3 are constants, |Ω| represents the sum of the pixel values ​​of all interpolated slices, f(x) represents the output value obtained by inputting the sample x into the inter-slice interpolation neural network, e1, e2, e3 represent the three basis vectors of the three-dimensional space respectively;

[0060] 2.1.7 Update the network parameters according to the loss value calculated in step 1.2.5 to complete the training;

[0061] 2.2 Next, the interpolated slices and the original slices are merged, and the tumor label slices are input into the probability map generator to generate a probability map of the tumor label;

[0062] 2.3 Input the MR slice sequence into the multi-scale cascade network to be trained (existing structure, referred to as "Network 2"), and output the probability map of the tumor according to the characteristics of the tumor. The mean square error between the output probability map and the tumor probability map of the data label is used as an evaluation index to train "Network 2".

[0063] 2.3.1 “Network 2” consists of convolutional layers, downsampling layers, and upsampling layers, as shown in Figure 4(c);

[0064] 2.3.2 The tumor slice sequence obtained in step 2.2 is composed of 64 3 The block is input into "Network 2". First, the size of 64 3Blocks are converted into four different scales: 64 3 , 32 3 , 16 3 and 8 3 ;

[0065] 2.3.3 Set the size obtained in step 2.3.2 to 32 3 The block input convolution layer, the output result is the same as the size of 64 3 The size of the block obtained by downsampling is 32 3 The blocks are merged to get a new size of 32 3 of blocks;

[0066] 2.3.4 Set the size obtained in step 2.3.2 to 16 3 The block input convolutional layer, the output result is the same as the size of 32 obtained in step 2.3.3 3 The size of the block obtained by downsampling is 16 3 The blocks are merged to get a new size of 16 3 of blocks.

[0067] 2.3.5 Set the size obtained in step 2.3.2 to 8 3 The block input convolutional layer, the output result is the same as the size of 16 obtained in step 2.3.4 3 The size of the block obtained by downsampling is 8 3 The blocks are merged to get a new size of 8 3 of blocks;

[0068] 2.3.6 Set the size obtained in step 2.3.3 to 32 3 The block is input to the convolutional layer, and the output result is downsampled to the size of 16 obtained in step 2.3.4. 3 The blocks are merged to get a new size of 16 3 of blocks;

[0069] 2.3.7 Set the size obtained in step 2.3.6 to 16 3 The block is downsampled to the size of 8 obtained in step 2.3.5 3 The blocks are merged to get a new size of 8 3 of blocks;

[0070] 2.3.8 Set the size obtained in step 2.3.6 to 16 3 The blocks are downsampled to the size of 8 obtained in step 2.3.7 3 The blocks are merged to get a new size of 8 3 of blocks;

[0071] 2.3.9 Set the size obtained in step 2.3.8 to 8 3The block is upsampled three times to get a new size of 64 3 The block generates a predicted probability map through the results;

[0072] 2.3.10 Input the trained tumor label into the probability map generator to generate the corresponding tumor probability map;

[0073] 2.3.11 Compare the labeled tumor probability map with the predicted tumor probability map and calculate the loss value;

[0074]

[0075] Among them, n is the number of samples, y is the label value, and y i The results obtained for the prediction

[0076] 2.3.12 Update the network parameters according to the loss value calculated in step 2.3.11 to complete the training

[0077] Step 3, predict pathological sections and visualization, specific implementation process:

[0078] 3.1 Input the pathological slices into "Network 1" for inter-slice interpolation as shown in Figure 4(d), providing step 3.2;

[0079] 3.2 The interpolated pathological slice sequence is input into "Network 2", and the output tumor probability map is shown in Figure 4(d). According to the set threshold (the probability boundary value for determining whether the location is a tumor), the tumor in the MR image is segmented and the result is visualized.

[0080] A set of implementation examples are given in accordance with the illustrated process and model structure.

[0081] Example

[0082] First, the 3D tumor slices and tumor label slices are input into the linear interpolation network, and the relationship between the interpolated slices and the corresponding adjacent slices is modeled by combining local linear embedding. The inter-slice interpolation network consists of a dense connection module, a sub-pixel convolution module, a downsampling module, and a local linear embedding module. First, the inter-slice interpolation neural network is trained, and real MR tumor slices and corresponding labeled slices are obtained from the hospital. The obtained slices are divided into groups of three, the middle slice is used as the label, and the remaining two slices are input into the inter-slice interpolation neural network. After the dense connection module and the sub-pixel convolution module, the upsampling is performed and the results are further input into the dense connection module and the sub-pixel convolution module. After upsampling again, the result is input into the dense connection module and downsampled through 2x2 convolution. The downsampled result is input into the dense connection module again and downsampled through 2x2 convolution. Finally, after the downsampled result is input into the dense connection layer, the K nearest points of each dimension value of the interpolated slice are obtained. By calculating the weights of the K adjacent points and linearly combining them according to the weights, the linear interpolation slice between the two slices is obtained. Using the defined loss function, calculate the loss value between the interpolated slice and the label, and update the parameters of the inter-slice interpolation neural network according to the loss value to complete the training. Next, input the samples that need to be segmented into the trained inter-slice interpolation neural network to obtain an interpolated slice sequence. Use the interpolated MR tumor slices and labels to train the multi-scale cascade neural network. The multi-scale cascade segmentation network consists of convolutional layers, downsampling layers, and upsampling layers. First, the interpolated slices are converted into probability maps as labels through a probability map generator, and the network also outputs prediction results in the form of probability maps. The size of each input three-dimensional slice is 64, which is aggregated into four different scales through the downsampling layer: 64, 32, 16, and 8. By convolutionally merging the feature results of tumors at different scales, the feature extraction network finally obtains a feature map of size 8. After upsampling and merging the feature maps, we get Figure 6 The tumor probability map in (c) is used as the output of the neural network. The error between the predicted value and the label is calculated by the mean square error method, the parameters of the multi-scale cascade neural network are updated, and the neural network is trained. Next, the sample that needs to be segmented is input into the trained multi-scale cascade neural network, and the probability map of the tumor is output. The tumor is segmented by the set probability threshold and the result is visualized.

Claims

1. A pancreatic tumor segmentation method based on interpolation neural network with local linear embedding, Features: Includes steps: Step 1: Dataset preparation: Obtain a data set from the hospital, including 3D MR slice images and annotated tumor segmentation labels as the data set; Step 2: Training and Optimization: 2.1 Divide the original slice images and labels obtained in step 1 into groups of two, represented as x i-1 and x i+1 , and are input into the inter-slice interpolation network to be trained, referred to as "Network 1"; the structural similarity loss between the original slice image and the interpolated slice image is used as the evaluation index to train "Network 1"; "Network 1" can obtain the inter-slice interpolation slice, represented as x i , making the 3D MR slice sequences more continuous; 2.2 Next, merge the slice x obtained by interpolation in step 2.1 i and slice x i-1 、x i+1 Obtain a slice sequence; input the tumor label slices therein into a probability map generator to generate a probability map of the tumor label for providing to step 2.3; 2.3 Input the MR slice sequence into the multi-scale cascade network to be trained, referred to as "Network 2", and output the tumor probability map according to the characteristics of the tumor; use the mean square error of the tumor probability map output by "Network 2" and the tumor label probability map provided in step 2.2 as the evaluation index to train "Network 2"; Step 3: predict pathological sections and visualize; Step 2.1: 2.1.1 "Network 1" consists of a dense connection module, a sub-pixel convolution module, a downsampling module, and a local linear embedding module, where the input slice is represented as x i ={x i1 ,x i2 ,…,x id }, the output interpolation slice is represented as The slices of tumor MR images and tumor labels are divided into groups of three in order. The middle slice is used as the label. The remaining two slices are input into "Network 1". After passing through the dense connection module and the sub-pixel convolution module, upsampling is performed and the results are further input into the dense connection module and the sub-pixel convolution module. 2.1.3 Upsample the result of step 2.1.2, input the result into the dense connection module and downsample it through 2x2 convolution; 2.1.4 Input the result of step 2.1.3 into the dense connection module and downsample it through 2x2 convolution. Input the sampling result into the dense connection module to obtain the input slice x i The K nearest neighbors of each dimension value of are Represents the K nearest neighbors of the jth dimension value of the i-th slice; 2.1.5 Linearly combine the results of step 2.1.4 to obtain interpolated slices; 2.1.5.1 First calculate the local covariance matrix in represents a system consisting of K identical x ij The vector composed of x ij represents the value of the jth dimension of the i-th slice, represents the K nearest neighbors of the jth dimension of the i-th slice obtained in step 1; 2.1.5.2 Next, calculate the weights of the K nearest neighbors 1 of them K =[1,1,…,1]∈R K , S ij represents the covariance matrix obtained in step 2.1.5.1; 2.1.5.3 Calculate interpolated slices by weighted linear sum of K nearest neighbors The values ​​of each dimension in represents the weight of the neighboring points obtained in step 2.1.5.2, Represents the K nearest neighbors of the jth dimension of the i-th slice; 2.1.6 Compare the interpolated slice with the labeled slice and calculate the loss value; 2.1.7 Update the network parameters according to the loss value calculated in step 2.1.6 to complete the training.

2. The pancreatic tumor segmentation method based on a locally linear embedding interpolation neural network as claimed in claim 1, characterized in that: Step 2.3: 2.3.1 "Network 2" consists of convolutional layers, downsampling layers and upsampling layers; 2.3.2 The tumor slice sequence obtained in step 2.2 is composed of 64 3 The block is input into "Network 2"; first, the size is 64 by downsampling 3 Blocks are converted into four different scales: 64 3 , 32 3 , 16 3 and 8 3 ; 2.3.3 Set the size obtained in step 2.3.2 to 32 3 The block input convolution layer, the output result is the same as the size of 64 3 The size of the block obtained by downsampling is 32 3 The blocks are merged to get a new size of 32 3 2.3.4 Set the size obtained in step 2.3.2 to 16 3 The block input convolutional layer, the output result is the same as the size of 32 obtained in step 2.3.3 3 The size of the block obtained by downsampling is 16 3 The blocks are merged to get a new size of 16 3 of blocks; 2.3.5 Set the size obtained in step 2.3.2 to 8 3 The block input convolutional layer, the output result is the same as the size of 16 obtained in step 2.3.4 3 The size of the block obtained by downsampling is 8 3 The blocks are merged to get a new size of 8 3 of blocks; 2.3.6 Set the size obtained in step 2.3.3 to 32 3 The block is input to the convolutional layer, and the output result is downsampled to the size of 16 obtained in step 2.3.

4. 3 The blocks are merged to get a new size of 16 3 of blocks; 2.3.7 Set the size obtained in step 2.3.6 to 16 3 The block is downsampled to the size of 8 obtained in step 2.3.5 3 The blocks are merged to get a new size of 8 3 of blocks; 2.3.8 Set the size obtained in step 2.3.6 to 16 3 The blocks are downsampled to the size of 8 obtained in step 2.3.7 3 The blocks are merged to get a new size of 8 3 of blocks; 2.3.9 Set the size obtained in step 2.3.8 to 8 3 The block is upsampled three times to get a new size of 64 3 The block generates a predicted probability map through the results; 2.3.10 Input the trained tumor label into the probability map generator to generate the corresponding tumor probability map; 2.3.11 Compare the labeled tumor probability map with the predicted tumor probability map and calculate the loss value; Among them, n is the number of samples, y is the label value, and y i To predict the result 2.3.12, based on the loss value calculated in step 2.3.11, update the network parameters and complete the training.

3. The pancreatic tumor segmentation method based on a locally linear embedding interpolation neural network as claimed in claim 1, Features: Step 3: 3.1 Input the pathological slices into the trained "network 1" for interpolation between slices, and obtain the interpolated pathological slice sequence to provide step 3.2; 3.2 Input the interpolated pathological slice sequence into the trained "Network 2" and output the tumor probability map. According to the boundary value set for determining whether the location is a tumor, the tumor in the MR image is segmented and the result is visualized.

4. The pancreatic tumor segmentation method based on a locally linear embedding interpolation neural network as claimed in claim 1, Features: The step 2.1.6 compares the interpolated slice with the label slice and calculates the loss value, which is calculated as follows: in, and Y represent the interpolated slice and label slice respectively. and μ Y They are and the pixel mean of Y, and σ Y They are and the pixel standard deviation of Y, express and the pixel covariance of Y, C 1 , C 2 , C 3 is a constant, |Ω| represents the sum of the pixel values ​​of all interpolated slices, f(x) represents the output value obtained by inputting sample x into the inter-slice interpolation neural network, and e 1 、e 2 、e 3 Represent the three basis vectors of the three-dimensional space respectively.

Images