Esophageal cancer ct image segmentation method based on deep learning

Abstract

The application relates to the technical field of image processing, in particular to an esophageal cancer CT image segmentation method based on deep learning, which comprises the steps of establishing and training a deep learning model and applying the deep learning model to process esophageal cancer CT image data, wherein the step of establishing and training the deep learning model comprises the following steps: acquiring esophageal cancer CT image dissection annotation data set, and pre-processing data; establishing and training a first deep learning model to coarsely segment the esophageal cancer CT image dissection annotation data set, wherein the first deep learning model is a U-Net model adjusted in parameters; establishing and training a second deep learning model to finely segment the data set output in the step 2, wherein the second deep learning model is a hybrid model of a U-Net-based CNN and a Vision Transformer; and predicting the esophageal cancer CT image dissection annotation prediction set, so that the CT image is converted from a large range to a small range, and the precision of model segmentation is improved, which is superior to the current most advanced method.

Description

Technical fields:

[0001] This invention relates to the field of image processing technology, specifically a segmentation method for esophageal cancer CT images based on deep learning. Background technology:

[0002] Esophageal cancer is a highly dangerous and fatal malignant tumor. According to a 2020 survey by the World Health Organization (WHO), esophageal cancer is one of the top ten most common cancers worldwide. Common treatment options for esophageal cancer include radiotherapy and chemotherapy, with CT images serving as the diagnostic basis. Accurate delineation of the tumor area is crucial for patient diagnosis during radiotherapy. Current methods involve manually delineating the tumor area by combining endoscopic and CT scan results. However, this method is time-consuming and resource-intensive, and the results vary between physicians, making the process very difficult and hindering the acquisition of effective data. Therefore, it is necessary to utilize computer-aided segmentation of esophageal tumor areas, which can not only save time but also maintain consistency in the delineation of complex tumor areas. However, the following challenges exist in esophageal tumor segmentation: First, the shape of the esophagus itself is not fixed, and the CT imaging angle and differences between different machines can significantly affect the final imaging results. Second, the esophagus has a large vertical range, and tumors can appear in any location within it. Therefore, unlike tumors in other organs, esophageal tumors do not have obvious relative positional characteristics. Finally, due to the indistinct boundary features between the tumor region and other areas, direct segmentation using CT images becomes extremely difficult. To overcome these challenges, in-depth research and improvement of the algorithm are necessary. Summary of the Invention:

[0003] This invention addresses the shortcomings and deficiencies of existing technologies by proposing a deep learning-based segmentation method for esophageal cancer CT images.

[0004] This invention achieves its purpose through the following measures:

[0005] A deep learning-based segmentation method for esophageal cancer CT images, characterized by comprising establishing and training a deep learning model, and applying the deep learning model to process esophageal cancer CT image data, wherein establishing and training the deep learning model includes the following steps:

[0006] Step 1: Obtain the anatomical annotation dataset of esophageal cancer CT images and preprocess the data; Step 2: Build and train the first deep learning model to perform coarse segmentation on the anatomical annotation dataset of esophageal cancer CT images, wherein the first deep learning model is a U-Net model with adjusted parameters;

[0007] Step 3: Build and train a second deep learning model to further segment the output dataset of Step 2. The second deep learning model is a hybrid model of CNN and Vision Transformer based on U-Net.

[0008] Step 4: Predict the esophageal cancer CT image anatomical annotation prediction set: The fine segmentation verification results are restored to the esophageal cancer lesion image region through matrix embedding. The matrix embedding embeds the fine segmentation results into a zero matrix with the same three-dimensional length as the original esophageal cancer region anatomical annotation image according to their relative coordinate positions in the original esophageal cancer region anatomical annotation image, so as to restore the fine segmentation results to the original three-dimensional length of the corresponding esophageal cancer region anatomical annotation image.

[0009] The first deep learning model in step 2 of this invention includes an encoder and a decoder. The encoder consists of five consecutive encoding modules, each of which is composed of a downsampling module. The decoder consists of four consecutive decoding modules, each of which is composed of an upsampling module and a skip connection module connected in series. Each downsampling module, upsampling module, and skip connection module consists of a convolutional layer, a 3D InstanceNorm layer, and a LeakyReLU layer. The convolutional layers of the downsampling module and the skip connection module are 3D Convolutional layers, and the convolutional layers of the upsampling module are 3D Transpose layers. In the Convolution layer, the esophageal cancer CT image anatomical annotation data passes through each downsampling module of the encoder to obtain an encoded vector. This encoded vector is then passed to the upsampling module of the corresponding decoder to obtain a decoded vector. Each skip connection module fuses the decoded vector from the current upsampling module and the encoded vector from the corresponding downsampling module to obtain a coarse segmentation result. The number of input channels in the convolutional layer of each skip connection module is twice the number of input channels in the convolutional layer of the corresponding downsampling or upsampling module. After each downsampling module (excluding the last one) of the encoder, the 3D length of the esophageal cancer CT image anatomical annotation data decreases to the 3D length calculated by the 3D Convolution layer, and the number of image channels increases to twice the number of channels before passing through this downsampling module. After passing through the last downsampling module of the encoder, the 3D length and number of channels remain unchanged. After each upsampling module and skip connection module of the decoder, the 3D length of the image increases to the 3D length calculated by the 3D Transpose Convolution layer, and the number of image channels decreases to half the number of channels before passing through this upsampling and skip connection modules.

[0010] In step 3 of this invention, the second deep learning model is a hybrid model of CNN and Vision Transformer based on U-Net, comprising five parts: encoder, pre-switcher, bridge, post-switcher, and decoder. The encoder and decoder have the same structure as those in the first deep learning model. The pre-switcher and post-switcher each consist of four consecutive pre-switch modules and post-switch modules, each consisting of one dual-branch fast ViT module. The bridge consists of four bridge modules, each consisting of three dual-branch fast ViT modules and one dual-branch axial ViT module. Each pre-switch module fuses the encoding vectors from two adjacent downsampling modules to obtain the neighboring... The neighbor-to-neighbor encoded vectors are passed to the bridge module to obtain neighbor-to-neighbor vectors. Each subsequent switching module fuses the neighbor-to-neighbor vectors from two adjacent bridge modules to obtain a neighbor-to-neighbor decoded vector, which is then passed to the upsampling module. The neighbor-to-neighbor encoded vectors output by the front switch have the same three-dimensional length and number of channels as the encoded vectors output by the shallow downsampling module. The neighbor-to-neighbor decoded vectors output by the subsequent switch have the same three-dimensional length and number of channels as the neighbor-to-neighbor vectors output by the shallow bridge module. The three-dimensional length and number of channels of the neighbor-to-neighbor encoded vectors remain unchanged after passing through the bridge.

[0011] The vector transmission directions of the front and rear switches are opposite to those of the encoder; the vector transmission direction of the arch bridge is the same as that of the encoder.

[0012] The dual-branch fast ViT module is specifically as follows:

[0013] The dual-branch fast ViT module has a dual-input vector and single-output vector structure in the front-switching module and the rear-switching module, and a dual-input vector and single or dual-output vector structure in the arch-bridge module.

[0014] The two input vectors of the dual-branch fast ViT module are the encoding vector s from the deep downsampling module and the encoding vector p from the shallow downsampling module, respectively. They are calculated as follows:

[0015] The encoded vector p from the shallow downsampling module passes through the BatchNorm layer BN in its branch. p The calculation result p′ is obtained:

[0016] p′=BN p (p)

[0017] The encoded vector s from the deep downsampling module passes through the first BatchNorm layer BN in its branch. s and the first depthwise separable convolutional layer DWConvs The calculated result is then joined with the original vector using a residual concatenation operation to obtain the calculated result s′:

[0018] s′=DWConv s (BN s (s))+s

[0019] s′ passes through the second BatchNorm layer BN in its branch. s′ The calculated result after downsampling is sent to p′. p′ The calculation results are then concatenated along the channel to obtain the calculation result s″:

[0020] s″=concat(BN s′ (s′), send p′ (p′))

[0021] s″ through the second depth-separable convolutional layer DWConv in its branch s″ The calculation result is obtained, and a residual concatenation is performed between this result and s′. This result is then passed through a feedforward module consisting of convolutional layers in its branch. Obtain the final deep calculation results

[0022]

[0023] The calculation result obtained after upsampling is concatenated with p′ along the channel to obtain p″:

[0024]

[0025] p″ can be separable from depth in its branches by convolution DWConv p″ The subsequent calculation result is connected to p′ using a residual concatenation, and this result is then passed through a feedforward module consisting of convolutional layers in its branch. The final calculation results of the shallow layer are obtained.

[0026]

[0027] For the dual-branch fast ViT module in the pre-switching and post-switching modules, only the shallow final calculation result is returned. As the final output; for the two-branch fast ViT module in the arch bridge, the deep computation results are also returned. Shallow calculation results As the final output.

[0028] Furthermore, the dual-branch axial ViT module specifically comprises:

[0029] The dual-branch axial ViT module has a dual-input vector and dual-output vector structure.

[0030] The two input vectors of the dual-branch axial ViT module are derived from the encoding vectors m and n of two adjacent downsampling modules, respectively, and are calculated as follows:

[0031] After passing through the CrossAxial self-attention mechanism, m and n are used to obtain the calculation results m′ and n′:

[0032] m′, n′=Cross Axial(m, n)

[0033] m′ passes through the feedforward module composed of convolutional layers in its branch. Obtain the final calculation result of the current branch.

[0034]

[0035] n′ passes through the feedforward module consisting of convolutional layers in its branch. Obtain the final calculation result of the current branch.

[0036]

[0037] The cross-axial self-attention mechanism consists of two parallel axial self-attention mechanisms, Axial, with m and n respectively connected to the learnable feature matrix W in their respective branches. Q W K W V This yields the query vector Q, key vector K, and value vector V for each branch; Q from the two branches is then swapped and input into the corresponding branch's Axial.

[0038]

[0039] Q m K m V m Let Q represent the query vector, key vector, and value vector of the branch containing vector m, respectively. n K n V n These represent the query vector, key vector, and value vector of the branch containing vector n, respectively. These represent the learnable feature matrices of the query vector, the key vector, and the value vector in the branch containing vector m, respectively. These represent the learnable feature matrices of the query vector, the key vector, and the value vector, respectively, for the branch containing the n vector.

[0040]

[0041] The axial self-attention mechanism calculates the self-attention result (SelfAttention) separately along each of the three dimensions of the vector. i Then, the results of the three dimensions are summed, where i represents the dimension in which the self-attention is being calculated, i∈{x, y, z}, and x, y, and z represent the x-axis component, y-axis component, and z-axis component of the vector, respectively.

[0042] Axial(Q,K,V)=∑Self Attention i (Q, K, V)

[0043] The self-attention calculation method is as follows: multiply the transposes of Q and K along dimension i to obtain the attention weights, then perform softmax normalization on them, and finally multiply the attention weights with V along dimension i to obtain the weighted representation.

[0044]

[0045] Q i This indicates the component of the query matrix along dimension i. V represents the transpose of the components of the bond matrix along dimension i. i This represents the components of the value matrix along dimension i. This represents the component lengths of Q and K along dimension i.

[0046] The preprocessing in step 1 of this invention specifically includes:

[0047] Chest CT images of patients diagnosed with esophageal cancer and corresponding anatomically annotated images of the esophageal cancer region were obtained as samples, forming a sample set. The sample set underwent sample screening, format conversion, segmentation, and enhancement to obtain an esophageal cancer CT image anatomically annotated dataset. Sample screening removed images from the sample set that clearly did not show esophageal cancer symptoms or whose annotated locations were clearly unrelated to the actual esophageal cancer lesion. Format conversion converted the chest CT images in the sample set from IBEX format to NIFTI format. Segmentation further refined the anatomically annotated images. This set is divided into a training sample set and a validation sample set according to a five-fold cross-validation method, with the ratio of the training sample set to the validation sample set in each fold being 8:2. The sample enhancement for the training sample set involves voxel intensity limiting, voxel intensity normalization, application of Gaussian noise, Gaussian blurring, brightness adjustment, contrast adjustment, low-resolution simulation, gamma transformation, and mirror transformation on its chest CT images. Spatial rotation is also performed on the chest CT images and esophageal cancer region anatomical annotation images in the sample set. The esophageal cancer CT image anatomical annotation dataset consists of a training dataset and a validation dataset, and the data is divided in the same way according to the sample division process.

[0048] Based on the above scheme, the present invention has the following advantages: (1) The present invention establishes and trains a first deep learning model to perform coarse segmentation on the esophageal cancer CT image anatomical annotation dataset to obtain the region of interest, and then inputs the region of interest into the second deep learning model to obtain the esophageal cancer lesion region, thereby transforming the CT image from a large area to a small area, thereby improving the accuracy of model segmentation; (2) The present invention constructs a novel model based on the CNN-Transformer hybrid U-Net structure as the second deep learning model for fine segmentation of the esophageal cancer CT image anatomical annotation dataset, further improving the segmentation accuracy; (3) The present invention proposes two plug-and-play deep learning modules, which input vectors into two different branches and enhance cross-layer interaction by exchanging neighbor vectors to reduce early feature loss of input vectors and further improve the model segmentation accuracy; (4) Extensive experiments on large-scale datasets have demonstrated that the method proposed in the present invention is superior to the current state-of-the-art methods. Attached image description:

[0049] Appendix Figure 1 This is a flowchart of a deep learning-based CT image segmentation method for esophageal cancer provided in an embodiment of the present invention.

[0050] Appendix Figure 2 This is a schematic diagram of the second deep learning model structure provided in an embodiment of the present invention; in the figure, D0~D4 are downsampling modules, U0~U3 are upsampling modules, S0~S3 are skip connection modules, and C F0 ~CF3 For the front-end switching module, C B0 ~C B3 For the rear switching module, A0 to A3 are the arch bridge modules; C F0 ~C F3 C B0 ~C B3 A0 to A2 are composed of dual-branch fast ViT modules; A3 is composed of dual-branch axial ViT modules.

[0051] Appendix Figure 3 This is a schematic diagram of the dual-branch fast ViT module structure provided in an embodiment of the present invention.

[0052] Appendix Figure 4 This is a schematic diagram of the dual-branch axial ViT module structure provided in an embodiment of the present invention. (See attached diagram.) Figure 5 This is a schematic diagram of the ConvFFN structure, which is composed of convolutional layers, provided in an embodiment of the present invention.

[0053] Appendix Figure 6 This is a schematic diagram of the effects of various models under the data of the fine segmentation stage provided in the embodiment of the present invention. Each two rows in the figure are a single slice planar view of the anatomical annotation data of esophageal cancer CT image and a three-dimensional rendering of the corresponding data. The columns from left to right are: the real annotation area of ​​the dataset, the effect of the second deep learning model in the present invention, and the effect of other methods. Detailed implementation method:

[0054] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0055] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. It should be noted that the descriptions of various concepts in the following embodiments are only to make the content of the present invention easier to understand and do not imply a limitation on the scope of protection of the present invention. Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0056] Example 1:

[0057] This example provides a deep learning-based CT image segmentation method for esophageal cancer, including the following steps:

[0058] S1 obtains the anatomical annotation dataset of esophageal cancer CT images;

[0059] S2 performs coarse segmentation of the esophageal cancer CT image anatomical annotation dataset using the first deep learning model;

[0060] S3 inputs the region of interest into the second deep learning model for fine segmentation;

[0061] S4 predicts the anatomical annotations of CT images for esophageal cancer.

[0062] Furthermore, S1 specifically refers to:

[0063] Chest CT images of patients diagnosed with esophageal cancer and corresponding anatomically annotated images of the esophageal cancer region were obtained as samples, and a sample set was formed from several samples. The sample set was then subjected to sample screening, sample format conversion, sample segmentation, and sample enhancement to obtain an esophageal cancer CT image anatomically annotated dataset.

[0064] The sample screening process removes samples from the sample set that do not show obvious signs of esophageal cancer in the anatomical annotation images of esophageal cancer regions or whose annotation locations are obviously unrelated to the actual esophageal cancer lesion areas.

[0065] The sample format conversion converts the chest CT images in the sample set from IBEX format to NIFTI format;

[0066] The sample partitioning divides the sample set into a training sample set and a validation sample set according to the five-fold cross-validation method, with the ratio of the training sample set to the validation sample set in each fold being 8:2;

[0067] The sample augmentation for the training sample set includes voxel intensity limiting, voxel intensity normalization, application of Gaussian noise, Gaussian blurring, brightness adjustment, contrast adjustment, low-resolution simulation, gamma transformation, and mirror transformation of the chest CT images; and simultaneous spatial rotation of the chest CT images and anatomically annotated images of the esophageal cancer region in the sample set.

[0068] The esophageal cancer CT image anatomical annotation dataset consists of a training dataset and a validation dataset, and the data is divided in the same way according to the sample division process.

[0069] In this embodiment, the process of a deep learning-based esophageal cancer CT image segmentation method is as follows: Figure 1 As shown.

[0070] In this embodiment, electronic medical records from a hospital from 2005 to 2021 with a diagnosis of esophageal cancer were used to create a sample set for training the deep learning model. The number of samples obtained was 1665.

[0071] In this embodiment, the sample enhancement parameters are shown in Table 1.

[0072] Table 1: Sample Augmentation Parameters

[0073]

[0074] Furthermore, S2 specifically refers to:

[0075] A first deep learning model is constructed; the anatomical annotation dataset of esophageal cancer CT images is input into the constructed first deep learning model; the first deep learning model is trained, validated, and the maximum connected component of the validation result of the first deep learning model is calculated to obtain the trained first deep learning model and the coarse segmentation validation result; the region of interest is extracted from the coarse segmentation validation result to obtain the region of interest.

[0076] Furthermore, the first deep learning model is specifically as follows:

[0077] The first deep learning model is a U-Net model with adjusted parameters, which includes an encoder and a decoder.

[0078] The encoder consists of 5 consecutive encoding modules, and each encoding module consists of 1 downsampling module;

[0079] The decoder consists of four consecutive decoding modules, each of which is composed of an upsampling module and a skip connection module connected in series.

[0080] Each downsampling module, upsampling module, and skip connection module consists of one convolutional layer, one 3DInstanceNorm layer, and one LeakyReLU layer. The convolutional layers of the downsampling module and skip connection module are 3D Convolutional layers, while the convolutional layers of the upsampling module are 3D Transpose Convolutional layers.

[0081] The anatomical annotation data of the esophageal cancer CT image is processed by each downsampling module of the encoder to obtain the encoding vector, and the encoding vector is passed to the upsampling module of the corresponding decoder to obtain the decoding vector.

[0082] Each skip connection module fuses the decoded vector from the current upsampling module and the encoded vector from the corresponding downsampling module to obtain a coarse segmentation result. The number of input channels of the convolutional layer of each skip connection module is twice the number of input channels of the convolutional layer of the corresponding downsampling module or the corresponding upsampling module.

[0083] The anatomical annotation data of the esophageal cancer CT image, after passing through each downsampling module of the encoder (excluding the last downsampling module), has its three-dimensional length reduced to the three-dimensional length calculated by the 3D Convolution layer, and its number of image channels increased to twice the number of channels before passing through this downsampling module; after passing through the last downsampling module of the encoder, the three-dimensional length and number of channels of the data remain unchanged; after passing through each upsampling module and skip connection module of the decoder, the three-dimensional length of the data increases to the three-dimensional length calculated by the 3D Transpose Convolution layer, and its number of image channels decreases to half the number of channels before passing through this upsampling module and skip connection module.

[0084] In this embodiment, the parameters of the first deep learning model are shown in Table 2. Here, k, s, and p represent the kernel size, stride, and padding, respectively.

[0085] Table 2: Parameters of the First Deep Learning Model

[0086]

[0087] In this embodiment, the model is deployed on a server computer with an RTX 2080Ti graphics processor and uses the PyTorch 2.0.1 deep learning framework.

[0088] In this embodiment, the coarse segmentation training parameters are shown in Table 3.

[0089] Table 3: Coarse Segmentation Training Parameters

[0090]

[0091] In this embodiment, the following evaluation metrics are used:

[0092]

[0093]

[0094]

[0095] Where X and Y represent the voxel set of the predicted segmentation result and the voxel set of the ground truth labeled region, respectively. |X| and |Y| represent the number of voxels in sets X and Y, respectively. |X∩Y| represents the number of voxels correctly classified by both the predicted segmentation result and the ground truth labeled region. |X∪Y| represents the total number of voxels in sets X and Y. |XY| represents the sum of the difference voxels in sets X and Y. |XY| may have negative values. A negative value means that the voxels in the predicted segmentation result are fewer than those in the ground truth labeled region, also known as underestimation. A positive value means that the voxels in the predicted segmentation result are more than those in the ground truth labeled region, also known as overestimation. Therefore, the absolute value is used to calculate the VS average.

[0096]

[0097] Where TP represents the number of voxels correctly classified as positive by the model, and FP represents the number of voxels incorrectly classified as positive by the model.

[0098] In this embodiment, the coarse segmentation verification results are shown in Table 4.

[0099] Table 4: Coarse Segmentation Validation Results

[0100]

[0101] In this embodiment, the training set extracts the region of interest (ROI) using a scaling factor of twice the radius of the ground truth labeled region, while the validation and test sets extract the ROI using a scaling factor of twice the radius of the coarse segmentation result. Table 5 shows the ROI coverage of the coarse segmentation validation result under different radius scaling factors in the validation set.

[0102] Table 5: Region of Interest and Coverage of Coarse Segmentation Validation Results under Different Radius Scaling Factors on the Validation Set

[0103]

[0104] Furthermore, S3 specifically refers to:

[0105] Construct a second deep learning model; input the region of interest into the constructed second deep learning model; train and validate the second deep learning model, calculate the maximum connected component of the second deep learning model validation result, and obtain the trained second deep learning model and the fine segmentation validation result; restore the esophageal cancer lesion region by matrix embedding of the fine segmentation validation result;

[0106] The matrix embedding embeds the fine segmentation results into a zero matrix of the same three-dimensional length as the original esophageal cancer region anatomical annotation image, according to their relative coordinate positions in the original esophageal cancer region anatomical annotation image, so as to restore the fine segmentation results to the original three-dimensional length of the corresponding esophageal cancer region anatomical annotation image.

[0107] Furthermore, the second deep learning model is specifically as follows:

[0108] The second deep learning model is a hybrid model of CNN and Vision Transformer based on U-Net, consisting of five parts: encoder, front switcher, bridge, back switcher, and decoder.

[0109] The encoder and decoder have the same structure as the encoder and decoder in the first deep learning model;

[0110] The front and rear switches each consist of four consecutive front and rear switching modules, and each front and rear switching module consists of one dual-branch fast ViT module; the arch bridge consists of four arch bridge modules, and each arch bridge module consists of three dual-branch fast ViT modules and one dual-branch axial ViT module.

[0111] Each front-switching module fuses the encoded vectors from two adjacent downsampling modules to obtain a neighbor-to-neighbor encoded vector, and then passes the neighbor-to-neighbor encoded vector to the bridge module, where a neighbor-to-neighbor vector is obtained. Each back-switching module fuses the neighbor-to-neighbor vectors from two adjacent bridge modules to obtain a neighbor-to-neighbor decoded vector, and then passes the neighbor-to-neighbor decoded vector to the upsampling module. The neighbor-to-neighbor encoded vector output by the front-switching module has the same three-dimensional length and number of channels as the encoded vector output by the shallow downsampling module. The neighbor-to-neighbor decoded vector output by the back-switching module has the same three-dimensional length and number of channels as the neighbor-to-neighbor vector output by the shallow bridge module. The three-dimensional length and number of channels of the neighbor-to-neighbor encoded vector remain unchanged after passing through the bridge.

[0112] The vector transmission directions of the front and rear switches are opposite to those of the encoder; the vector transmission direction of the arch bridge is the same as that of the encoder. In this embodiment, the structure of the second deep learning model is as follows: Figure 2 As shown.

[0113] Furthermore, the dual-branch fast ViT module specifically comprises:

[0114] The dual-branch fast ViT module has a dual-input vector and single-output vector structure in the front-switching module and the rear-switching module, and a dual-input vector and single or dual-output vector structure in the arch-bridge module.

[0115] The two input vectors of the dual-branch fast ViT module are the encoding vector s from the deep downsampling module and the encoding vector p from the shallow downsampling module, respectively. They are calculated as follows:

[0116] The encoded vector p from the shallow downsampling module passes through the BatchNorm layer BN in its branch.p The calculation result p′ is obtained:

[0117] p′=BN p (p)

[0118] The encoded vector s from the deep downsampling module passes through the first BatchNorm layer BN in its branch. s and the first depthwise separable convolutional layer DWConv s The calculated result is then joined with the original vector using a residual concatenation operation to obtain the calculated result s′:

[0119] s′=DWConv s (BN s (s))+s

[0120] After passing through the second BatchNorm layer BN in its branch s′ The calculated result after downsampling is sent to p′. p′ The calculation results are then concatenated along the channel to obtain the calculation result s″:

[0121] s″=concat(BN s′ (s′), send p′ (p′))

[0122] s″ through the second depth-separable convolutional layer DWConv in its branch s″ The calculation result is obtained, and a residual concatenation is performed between this result and s′. This result is then passed through a feedforward module consisting of convolutional layers in its branch. Obtain the final deep calculation results

[0123]

[0124] The calculation result obtained after upsampling is concatenated with p′ along the channel to obtain p″:

[0125]

[0126] p″ can be separable from depth in its branches by convolution DWConv p″ The subsequent calculation result is connected to p′ using a residual concatenation, and this result is then passed through a feedforward module consisting of convolutional layers in its branch. The final calculation results of the shallow layer are obtained.

[0127]

[0128] For the dual-branch fast ViT module in the pre-switching and post-switching modules, only the shallow final calculation result is returned. As the final output; for the two-branch fast ViT module in the arch bridge, the deep computation results are also returned. Shallow calculation results As the final output.

[0129] In this embodiment, the dual-branch fast ViT module structure is as follows: Figure 3 As shown.

[0130] In this embodiment, the feedforward module structure composed of convolutional layers is as follows: Figure 5 As shown.

[0131] Furthermore, the dual-branch axial ViT module specifically comprises:

[0132] The dual-branch axial ViT module has a dual-input vector and dual-output vector structure.

[0133] The two input vectors of the dual-branch axial ViT module are derived from the encoding vectors m and n of two adjacent downsampling modules, respectively, and are calculated as follows:

[0134] After passing through the CrossAxial self-attention mechanism, m and n are used to obtain the calculation results m′ and n′:

[0135] m′,n′=CrossAcial(m,n)

[0136] m′ passes through the feedforward module composed of convolutional layers in its branch. Obtain the final calculation result of the current branch.

[0137]

[0138] n′ passes through the feedforward module consisting of convolutional layers in its branch. Obtain the final calculation result of the current branch.

[0139]

[0140] The cross-axial self-attention mechanism consists of two parallel axial self-attention mechanisms, Axial, with m and n respectively connected to the learnable feature matrix W in their respective branches. Q W K W v This yields the query vector Q, key vector K, and value vector V for each branch; Q from the two branches is then swapped and input into the corresponding branch's Axial.

[0141]

[0142] Q m K m V m Let Q represent the query vector, key vector, and value vector of the branch containing vector m, respectively. n K n V n These represent the query vector, key vector, and value vector of the branch containing vector n, respectively. These represent the learnable feature matrices of the query vector, the key vector, and the value vector in the branch containing vector m, respectively. These represent the learnable feature matrices of the query vector, the key vector, and the value vector, respectively, for the branch containing the n vector.

[0143]

[0144] The axial self-attention mechanism calculates the self-attention result (SelfAttention) separately along each of the three dimensions of the vector. i Then, the results of the three dimensions are summed, where i represents the dimension in which the self-attention is being calculated, i∈{x, y, z}, and x, y, and z represent the x-axis component, y-axis component, and z-axis component of the vector, respectively.

[0145] Axial(Q,K,V)=∑SelfAttention i (Q, K, V)

[0146] The self-attention calculation method is as follows: multiply the transposes of Q and K along dimension i to obtain the attention weights, then perform softmax normalization on them, and finally multiply the attention weights with V along dimension i to obtain the weighted representation.

[0147]

[0148] Q i This indicates the component of the query matrix along dimension i. V represents the transpose of the components of the bond matrix along dimension i. i This represents the components of the value matrix along dimension i. This represents the component lengths of Q and K along dimension i.

[0149] In this embodiment, the dual-branch axial ViT module structure is as follows: Figure 4 As shown.

[0150] In this embodiment, the sample augmentation parameters used in S3 are the same as those in S1. The training parameters used in S3 are the same as those in S1, except that the patch size is [32,80,96].

[0151] In this embodiment, in addition to using the model disclosed in this invention as the second deep learning model, other publicly available deep learning models are also used as the second deep learning model as a control group.

[0152] Other publicly published deep learning models include:

[0153] (1)UNETR(Hatamizadeh A,Tang Y,Nath V,et al.Unetr:Transformers for 3dmedical image segmentation[C] / / Proceedings of the IEEE / CVF winter conferenceon applications of computer vision.2022:574-584.)

[0154] (2)TransUNet(Chen J,Lu Y,Yu Q,et al.Transunet:Transformers make strong encoders for medical imagesegmentation[J].arXiv preprint arXiv:2102.04306,2021.)

[0155] (3)Swin-UNETR (Hatamizadeh A, Nath V, Tang Y, et al. Swinunetr: Swintransformers for semantic segmentation of braintumors in MRI images [C] / / International MICCAI BrainlesionWorkshop.Cham: Springer InternationalPublishing, 2021: 272-284.)

[0156] (4)nnFormer (Zhou HY, Guo J, Zhang Y, et al. nnFormer: volumetric medical image segmentation via a 3Dtransformer[J]. IEEE Transactions on ImageProcessing, 2023.)

[0157] (5)UNETR++(Shaker A, Maaz M, Rasheed H, et al.UNETR++: delving into efficient and accurate 3D medical imagesegmentation[J].arXiv preprint arXiv:2212.04497,2022.)

[0158] (6)KAIST's(Luu HM,Park S H.Extending nn-UNet forbrain tumorsegmentation[C] / / International MICCAIBrainlesion Workshop.Cham:SpringerInternationalPublishing,2021:173-186.)

[0159] (7)nnU-Net (Isensee F, Jaeger PF, Kohl SAA, et al. nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation [J]. Nature methods, 2021, 18(2): 203-211.)

[0160] In this example, the fine segmentation verification results of the model disclosed in this invention are compared with the fine segmentation verification performance indicators of other models, as shown in Table 6.

[0161] Table 6: Comparison of fine segmentation validation results of the model disclosed in this invention with fine segmentation validation performance indicators of other models.

[0162]

[0163] In this embodiment, the fine segmentation verification results of the model disclosed in this invention are visualized and compared with the fine segmentation verification results of other models, for example. Figure 6 As shown.

[0164] Furthermore, S4 specifically includes:

[0165] The process involves obtaining chest CT images of patients diagnosed with esophageal cancer and corresponding anatomical annotations of the esophageal cancer region as new samples, forming a new sample set. This new sample set undergoes format conversion and enhancement to obtain a predicted set of esophageal cancer CT image anatomical annotations. The weights of a first deep learning model are then locked. This predicted set is input into the trained first deep learning model for prediction, and the maximum connected component of the first deep learning model's prediction is calculated to obtain a coarse segmentation prediction result. The coarse segmentation prediction result is then used to extract a new region of interest (ROI). The weights of a second deep learning model are then locked. The new ROI is input into the trained second deep learning model for prediction, and the maximum connected component of the second deep learning model's prediction is calculated to obtain a fine segmentation prediction result. Finally, the fine segmentation prediction result is used to reconstruct the final prediction result of the esophageal cancer lesion region through matrix embedding.

[0166] Example 2:

[0167] This embodiment includes a computer-readable storage medium storing a data processing program, which is executed by a processor according to the deep learning-based esophageal cancer CT image segmentation method of Embodiment 1.

[0168] Based on the above scheme, the present invention has the following advantages: (1) The present invention establishes and trains a first deep learning model to perform coarse segmentation on the esophageal cancer CT image anatomical annotation dataset to obtain the region of interest, and then inputs the region of interest into the second deep learning model to obtain the esophageal cancer lesion region, thereby transforming the CT image from a large area to a small area, thereby improving the accuracy of model segmentation; (2) The present invention constructs a novel model based on the CNN-Transformer hybrid U-Net structure as the second deep learning model for fine segmentation of the esophageal cancer CT image anatomical annotation dataset, further improving the segmentation accuracy; (3) The present invention proposes two plug-and-play deep learning modules, which input vectors into two different branches and enhance cross-layer interaction by exchanging neighbor vectors to reduce early feature loss of input vectors and further improve the model segmentation accuracy; (4) Extensive experiments on large-scale datasets have demonstrated that the method proposed in the present invention is superior to the current state-of-the-art methods.

Claims

1. A segmentation method for esophageal cancer CT images based on deep learning, characterized in that, This includes building and training a deep learning model and applying the deep learning model to process esophageal cancer CT image data. The building and training of the deep learning model includes the following steps: Step 1: Obtain the anatomical annotation dataset of esophageal cancer CT images and preprocess the data; Step 2: Build and train the first deep learning model to perform coarse segmentation on the esophageal cancer CT image anatomical annotation dataset, wherein the first deep learning model is a U-Net model with adjusted parameters; Step 3: Build and train a second deep learning model to further segment the output dataset from Step 2. The second deep learning model is a hybrid model of CNN and Vision Transformer based on U-Net. This model comprises five parts: encoder, pre-switch, bridge, post-switch, and decoder. The encoder and decoder have the same structure as those in the first deep learning model. Each pre-switch and post-switch consists of four consecutive pre-switch and post-switch modules, each composed of one two-branch fast ViT module. The bridge consists of four bridge modules, each composed of three two-branch fast ViT modules and one two-branch axial ViT module. Each pre-switch module fuses the encoding vectors from two adjacent downsampling modules to obtain a neighbor encoding vector, and then passes this neighbor encoding vector to the bridge module. Each block obtains a neighbor vector; each post-switching module fuses the neighbor vectors from two adjacent bridge modules to obtain a neighbor decoding vector, and passes the neighbor decoding vector to the upsampling module. The neighbor encoding vector output by the pre-switching module has the same three-dimensional length and number of channels as the encoding vector output by the shallow downsampling module; the neighbor decoding vector output by the post-switching module has the same three-dimensional length and number of channels as the neighbor vector output by the shallow bridge module; the neighbor encoding vector does not change its three-dimensional length and number of channels after passing through the bridge; the vector transmission direction of the pre-switching and post-switching modules is opposite to the encoder vector transmission direction; the vector transmission direction of the bridge is the same as the encoder vector transmission direction. Step 4: Predict the esophageal cancer CT image anatomical annotation prediction set: The fine segmentation verification results are restored to the esophageal cancer lesion image region through matrix embedding. The matrix embedding embeds the fine segmentation results into a zero matrix with the same three-dimensional length as the original esophageal cancer region anatomical annotation image according to their relative coordinate positions in the original esophageal cancer region anatomical annotation image, so as to restore the fine segmentation results to the original three-dimensional length of the corresponding esophageal cancer region anatomical annotation image.

2. The method for segmenting esophageal cancer CT images based on deep learning according to claim 1, characterized in that, In step 2, the first deep learning model includes an encoder and a decoder. The encoder consists of five consecutive encoding modules, each of which is a downsampling module. The decoder consists of four consecutive decoding modules, each of which is a series-connected upsampling module and a skip connection module. Each downsampling module, upsampling module, and skip connection module consists of one convolutional layer, one 3D InstanceNorm layer, and one LeakyReLU layer. The convolutional layers of the downsampling module and skip connection module are 3D Convolutional layers, while the convolutional layers of the upsampling module are 3D Transpose layers. In the Convolution layer, the esophageal cancer CT image anatomical annotation data passes through each downsampling module of the encoder to obtain an encoded vector. This encoded vector is then passed to the upsampling module of the corresponding decoder to obtain a decoded vector. Each skip connection module fuses the decoded vector from the current upsampling module and the encoded vector from the corresponding downsampling module to obtain a coarse segmentation result. The number of input channels in the convolutional layer of each skip connection module is twice the number of input channels in the convolutional layer of the corresponding downsampling or upsampling module. After each downsampling module (excluding the last one) of the encoder, the 3D length of the esophageal cancer CT image anatomical annotation data decreases to the 3D length calculated by the 3D Convolution layer, and the number of image channels increases to twice the number of channels before passing through this downsampling module. After passing through the last downsampling module of the encoder, the 3D length and number of channels remain unchanged. After each upsampling module and skip connection module of the decoder, the 3D length increases to the 3D length calculated by the 3D TransposeConvolution layer, and the number of image channels decreases to half the number of channels before passing through this upsampling and skip connection modules.

3. The method for segmenting esophageal cancer CT images based on deep learning according to claim 1, characterized in that, The dual-branch fast ViT module is specifically configured as follows: in the pre-switching and post-switching modules, it has a dual-input vector, single-output vector structure; in the bridging module, it has a dual-input vector, single or dual-output vector structure. The two input vectors of the dual-branch fast ViT module are respectively derived from the encoding vectors of the deep downsampling module. and the encoded vector from the shallow downsampling module The two are calculated as follows: the encoded vector from the shallow downsampling module After passing through the BatchNorm layer in its branch Calculation results : ， Encoded vectors from the deep downsampling module After passing through the first BatchNorm layer in its branch and the first depth-separable convolutional layer The calculated result is then joined with the original vector using a residual concatenation operation to obtain the final result. : ， After passing through the second BatchNorm layer in its branch The calculation results after that and After downsampling The calculation results are then spliced ​​along the channel. The calculation results are obtained. : ， The second depth-separable convolutional layer in its branch Obtain the calculation result and compare it with... Perform a residual connection, and then pass the calculation result through the feedforward module consisting of convolutional layers in its branch. The final deep calculation results are obtained. ; ， The calculation results obtained after upsampling are... splicing along the channel ,get : ， Depth-separable convolutions through their branches The subsequent calculation results, and Perform a residual connection, and then pass the result through a feedforward module consisting of convolutional layers in its branch. The final calculation results for the shallow layer are obtained. ; ， For the dual-branch fast ViT module in the pre-switching and post-switching modules, only the shallow final calculation result is returned. As the final output; for the two-branch fast ViT module in the arch bridge, the deep computation results are also returned. Shallow calculation results As the final output.

4. The method for segmenting esophageal cancer CT images based on deep learning according to claim 3, characterized in that, The dual-branch axial ViT module is specifically configured as follows: the dual-branch axial ViT module has a dual-input vector and dual-output vector structure; the two input vectors of the dual-branch axial ViT module are respectively derived from the encoding vectors of two adjacent downsampling modules. and The calculation methods for both are as follows: and After cross-axis self-attention mechanism The calculation results are obtained. and : ， The feedforward module, composed of convolutional layers, passes through its branch. To obtain the final calculation result of the current branch. : ， The feedforward module, composed of convolutional layers, passes through its branch. To obtain the final calculation result of the current branch. : ， The cross-axial self-attention mechanism consists of two parallel axial self-attention mechanisms. Composition, consisting of and Each of them is compared with the learnable feature matrices in its respective branch. , , To obtain the query vector in each branch Key vector Value vector ; In the two branches They exchanged and the input was fed into the corresponding branch. middle: ， in , , They represent The query vector, key vector, and value vector of the branch containing the vector. , , They represent The query vector, key vector, and value vector of the branch containing the vector. , , They represent The learnable feature matrix of the query vector, the learnable feature matrix of the key vector, and the learnable feature matrix of the value vector in the branch containing the vector. , , They represent The learnable feature matrix of the query vector, the learnable feature matrix of the key vector, and the learnable feature matrix of the value vector in the branch where the vector is located. ， The axial self-attention mechanism calculates the self-attention result separately along the three dimensions of the vector, and then sums the results of the three dimensions. This indicates the dimension used to compute self-attention. , , , Representing vectors respectively Axial components, Axial components, Axial components: , The self-attention calculation method is as follows: and transpose along dimension The attention weights are obtained by multiplying them, then subjected to softmax normalization, and finally multiplied by... Along the dimension Multiplying yields the weighted representation: ， in Indicates the query matrix along the dimension The amount, The key matrix represents the matrix along the dimension. The transpose of the components. Represents the value matrix along the dimension The amount, express and Along the dimension The length of the component.

5. The method for segmenting esophageal cancer CT images based on deep learning according to claim 1, characterized in that, The preprocessing in step 1 specifically includes: Chest CT images of patients diagnosed with esophageal cancer and corresponding anatomically annotated images of the esophageal cancer region were obtained as samples, forming a sample set. The sample set underwent sample screening, format conversion, segmentation, and enhancement to obtain an esophageal cancer CT image anatomically annotated dataset. Sample screening removed samples from the sample set that clearly did not show esophageal cancer symptoms or whose annotated locations were clearly unrelated to the actual esophageal cancer lesion area. Format conversion converted the chest CT images in the sample set from IBEX format to NIFTI format. Segmentation divided the sample set into a training sample set and a validation sample set using a five-fold cross-validation method, with a ratio of 8:2 for each fold. Enhancement of the training sample set involved applying voxel intensity limiting, voxel intensity normalization, Gaussian noise application, Gaussian blurring, brightness adjustment, contrast adjustment, low-resolution simulation, gamma transformation, and mirror transformation to the chest CT images. The chest CT images and esophageal cancer region anatomically annotated images in the sample set are simultaneously spatially rotated; the esophageal cancer CT image anatomically annotated dataset consists of a training dataset and a validation dataset, and the data is divided in the same way according to the sample division process.

Images