A pulmonary nodule image detection method and system based on CT images

Abstract

The present invention discloses a pulmonary nodule image detection method and system based on CT images, and the detection method includes: S1, image serialization: reshaping the slices of the input pulmonary CT image into a set of patch sequences to perform tokenization; S2, using patch embedding, using a trainable linear mapping to map the vectorized patch sequence to a potential two-dimensional embedding space; S3, establishing a hybrid encoder of CNN and Transformer: encoding the tokenized image blocks from the CNN feature map into an input sequence for extracting the global context through the Transformer; S4, cascade decoder: firstly upsampling the encoded features obtained in step S3 through the decoder, and then combining the upsampled encoded features with the high-resolution CNN feature map to achieve precise positioning, and finally using U-Net to enhance more accurate detail detection information by restoring local spatial information. The present invention can effectively improve the accuracy of pulmonary nodule detection.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method and system for detecting lung nodules based on CT images. Background Art

[0002] Lung cancer is the leading cause of death in the world. Lung nodules are an early manifestation of lung cancer and can be observed as circular lung shadows with a diameter of no more than 3 cm on CT images. Accurate detection of their contours can help doctors diagnose benign and malignant lung nodules. Due to the small size of lung nodules, their morphology, brightness and other characteristics are similar to those of blood vessels and other tissues in the lung parenchyma. It is difficult to separate the two by naked eye observation alone, which can easily cause serious interference to doctors' judgment. In order to reduce doctors' workload and improve the efficiency of nodule diagnosis, computer-aided diagnosis technology has begun to be used in clinical work.

[0003] Deep learning has achieved outstanding results in the field of computer vision. In various medical image segmentation tasks, the U-Net architecture has become the de facto standard and has achieved great success. However, due to the inherent locality of convolution operations, U-Net often shows limitations in explicitly modeling long-range dependencies. The Transformer, designed for sequence-to-sequence prediction, has emerged as an alternative architecture with an innate global self-attention mechanism, but its lack of low-level feature details may lead to limited localization capabilities. Summary of the invention

[0004] The present invention aims at the problem that the existing method of only using Transformer to encode the labeled image blocks and then directly upsampling the hidden feature representation to a dense output of full resolution cannot produce satisfactory results. Such architectures usually show large differences in texture, shape and size between patients. In order to overcome this limitation, the present invention provides a method and system for lung nodule image detection based on CT images. The detection method uses the TransformerUnet combined architecture. On the basis of existing research, it is proposed to establish a self-attention mechanism based on CNN features (i.e., self-attention mechanism). Different from the existing CNN-based methods, TransformerUnet establishes a self-attention mechanism from the perspective of sequence-to-sequence prediction. In order to compensate for the feature resolution loss caused by Transformer, the network adopts a hybrid architecture of CNN and Transformer to utilize detailed high-resolution spatial information from CNN features and global context information encoded by Transformer. Inspired by the U-shaped architecture design, the self-attention features encoded by Transformer are then upsampled to be combined with different high-resolution CNN features that are skipped from the encoding path to achieve precise positioning.

[0005] The present invention discloses a method and system for detecting lung nodules based on CT images, which adopts a TransformerUnet combined architecture. The TransformerUnet combined architecture is a neural network architecture (Transformer) based on attention coding technology in deep learning and a biomedical semantic segmentation network architecture (U-Net) based on full convolutional network technology. The Transformer designed for sequence-to-sequence prediction has an alternative architecture with an innate global self-attention mechanism. By adding a medical image segmentation model, the problem of low positioning accuracy due to insufficient low-level features can be solved. Different from the existing manually designed lung nodule detection model, the detection framework proposed by the present invention consists of two parts: a Transformer part and a U-Net part.

[0006] Terminology explanation:

[0007] 1. Transformer: A neural network architecture with attention encoding technology.

[0008] 2. CNN: Convolutional Neural Networks.

[0009] 3. U-Net: Biomedical semantic segmentation network architecture, a type of fully convolutional neural network.

[0010] 4. Patch: The feature detector in the convolutional neural network divides the input image into multiple small blocks, each of which is called a patch.

[0011] 5. CUP: Cascaded Upsampler, cascaded decoder, using a decoder with low computational complexity to upsample larger images can increase decoding speed.

[0012] 6. MSA: Multi-Head Self Attention, a method that understands the input sequence from different angles at the same time and calculates multiple Attentions.

[0013] 7. MLP: Multilayer Perceptron, also known as artificial neural network, is an architecture that contains multiple hidden layers in addition to the input and output layers.

[0014] The technical solution adopted by the present invention to overcome the technical problems is:

[0015] The present invention discloses a lung nodule image detection method based on CT images, which uses a TransformerUnet combined architecture to perform lung nodule image detection. The TransformerUnet combined architecture includes a Transformer part and a U-Net part. The detection method includes the following steps:

[0016] S1. Image serialization: reshape the slices of the input lung CT image into a set of patch sequences to perform tokenization;

[0017] S2, using patch embedding, using a trainable linear mapping to map the vectorized patch sequence to a latent two-dimensional embedding space;

[0018] S3. Build a hybrid encoder of CNN and Transformer: Encode the tokenized image blocks from the CNN feature map into an input sequence for extracting global context through Transformer;

[0019] S4, cascade decoder: First, the encoded features obtained in step S3 are upsampled through the decoder, and then the upsampled encoded features are combined with the high-resolution CNN feature map to achieve accurate positioning. Finally, U-Net is used to enhance more accurate detail detection information by restoring local spatial information.

[0020] Further, in step S1, the lung CT image is assumed to be H×W is the spatial resolution, and C is the number of channels.

[0021] Furthermore, step S1 specifically includes:

[0022] Tokenization is performed by reshaping the input lung CT image x into a sequence of patches Where p is the sequence size, so the size of each patch is p×p, and the number of patches per image is That is, the length of the input sequence.

[0023] Furthermore, step S2 specifically includes:

[0024] S21. In order to encode the spatial information of the patch sequence, a specific position encoding added to the patch sequence embedding is learned to preserve the position information, as shown in the following formula:

[0025]

[0026] in, is the patch embedding map, Represents the position embedding information, and D is the dimension of the input patch;

[0027] S22. In order to restore the spatial order of the patch after embedding, the size of the encoded feature is first become A 1×1 convolution is used to reduce the channel size of the feature to the number of feature classes, and then the feature map is directly upsampled to the full resolution H×W for predicting the final segmentation result.

[0028] Furthermore, step S3 specifically includes:

[0029] The hybrid encoder of CNN and Transformer consists of l layers of multi-head self-attention and multi-layer perceptron as shown in equations (2) and (3), so the output of the lth layer can be written as follows:

[0030]

[0031]

[0032] Among them, MSA represents multi-head self-attention, MLP represents multi-layer perceptron, LN(·) represents the normalization operator of the image, represents the multi-head attention output of the lth layer, z l Represents the description of the coded image at layer l.

[0033] Furthermore, it also includes compensation for the information loss of the hybrid encoder of CNN and Transformer, including:

[0034] Similar to U-Net, skip connections are used to fuse multi-scale features from the hybrid encoder with upsampled features, and CNN is used as a feature extractor to generate feature maps instead of 1×1 patches extracted from the original image as input, thereby retaining more deep and shallow features to compensate for information loss.

[0035] Furthermore, step S4 specifically includes:

[0036] Multiple upsampling steps are used to decode the hidden features to output the final segmentation mask map, specifically:

[0037] In the hidden features The sequence is reconstructed into After that, the cascaded decoder is realized by cascading multiple upsampling blocks to achieve To H×W full resolution, the cascaded multiple upsampling blocks include two upsampling, a 3×3 convolutional layer, and a ReLU layer in sequence;

[0038] Finally, the cascaded decoder and the hybrid encoder together form a U-shaped architecture, which achieves feature fusion by upsampling feature maps of different levels of resolution through skip connections.

[0039] The present invention also discloses a pulmonary nodule image detection system based on CT images, which adopts a TransformerUnet combined architecture and specifically includes:

[0040] An image serialization module is used to reshape the slices of the input lung CT image into a set of patch sequences to perform tokenization;

[0041] A patch embedding module for mapping vectorized patch sequences to a latent 2D embedding space using a trainable linear mapping;

[0042] A hybrid encoder module of CNN and Transformer, which encodes tokenized image patches from CNN feature maps into input sequences for extracting global context through Transformer;

[0043] The cascade decoder module is used to first upsample the encoded features obtained by the hybrid encoder module of CNN and Transformer through the decoder, then combine the upsampled encoded features with the high-resolution CNN feature map to achieve precise positioning, and finally use U-Net to enhance more accurate detail detection information by restoring local spatial information.

[0044] The beneficial effects of the present invention are:

[0045] 1. Existing algorithms for lung nodule detection take a lot of time to extract features. Traditional feature extraction algorithms require a lot of manual annotation, and these features require a lot of prior knowledge. The deep learning method for lung nodule detection and classification can effectively avoid the subjective uncertainty of doctors' judgments, effectively relieve doctors' workload and improve the accuracy of lung nodule detection, because the deep learning model can automatically learn to extract features that are suitable for the current task.

[0046] 2. The effect of using Transformer directly for lung nodule detection is not as good as the performance of U-Net or Attention. Since Transformer can extract high-level semantic features well, which is beneficial for classification tasks, it lacks low-level features to segment lung nodule images. Therefore, by combining Transformer jump connection with U-Net structure to form TransformerUnet network, it has strong learning ability of high-level semantic features and underlying detail features, which can effectively improve the accuracy of lung nodule detection and assist doctors in making judgments. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a serialized slice of a lung CT image according to an embodiment of the present invention.

[0048] Figure 2 It is a schematic diagram of the principle of the TransformerUnet combined architecture described in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to facilitate those skilled in the art to better understand the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The following is only exemplary and does not limit the protection scope of the present invention.

[0050] Embodiment 1,

[0051] like Figure 1 and Figure 2 As shown, this embodiment discloses a lung nodule image detection method based on CT images, which uses a TransformerUnet combined architecture to perform lung nodule image detection. The TransformerUnet combined architecture includes a Transformer part and a U-Net part.

[0052] The lung nodule image detection method based on CT images described in this embodiment includes the following steps:

[0053] Step S1, image preprocessing. The image preprocessing described in this embodiment adopts image serialization, that is, the slices of the input lung CT image are reshaped into a group of patch sequences to perform labeling.

[0054] Assume that the given lung CT image is H×W is the spatial resolution, and C is the number of channels. The goal is to predict a pixel label map of the corresponding size H×W. Unlike the existing method of training CNN (such as U-Net), encoding the image into a high-level feature representation and then decoding it into the full spatial resolution, the self-attention mechanism is introduced into the encoder design by using Transformer, first encoding the image into a high-level feature representation and then decoding it into the original resolution size.

[0055] Since the pixel size and granularity of different scanning surfaces are different, this is not conducive to the model training task. The use of image serialization can effectively avoid this situation. The image serialization described in this embodiment specifically includes:

[0056] Tokenization is performed by reshaping the input lung CT image x into a sequence of patches Where p is the sequence size, and the unit of p is pixels, so the size of each patch is p×p, and the number of patches per image is That is, the length of the input sequence.

[0057] Step S2, patch embedding: Using patch embedding, a trainable linear mapping is used to map the vectorized patch sequence to a latent two-dimensional embedding space.

[0058] In this embodiment, step S2 specifically includes:

[0059] S21. In order to encode the spatial information of the patch sequence, a specific position encoding added to the patch sequence embedding is learned to preserve the position information, as shown in the following formula:

[0060]

[0061] in, is the patch embedding map, Represents the position embedding information, and D is the dimension of the input patch;

[0062] S22. In order to restore the spatial order of the patch after embedding, the size of the encoded feature is first become A 1×1 convolution is used to reduce the channel size of the feature to the number of feature classes, and then the feature map is directly upsampled to the full resolution H×W for predicting the final segmentation result.

[0063] Step S3: Establish a hybrid encoder of CNN and Transformer: Encode the tokenized image blocks from the CNN feature map into an input sequence for extracting global context through Transformer.

[0064] In the hybrid encoder of CNN and Transformer, different sets of suspected pulmonary nodule candidates are obtained according to the patch embedding. Due to the internal limitations of convolution operations (there are still limitations in modeling long-range relationships), these architectures usually produce weaker performance, especially for patients who show large differences in structural texture, shape and size. To overcome this limitation, a self-attention mechanism based on CNN features is adopted. This method encodes the tokenized image patches from the CNN feature map into an input sequence that extracts the global context. Secondly, unlike previous CNN-based methods, Transformer is not only powerful in global feature extraction, but also shows excellent transferability to downstream tasks under large-scale pre-training. As an alternative architecture, it fully adopts the distributed convolution operation and relies only on the attention mechanism.

[0065] Specifically, step S3 includes:

[0066] The hybrid encoder of CNN and Transformer consists of l layers of multi-head self-attention and multi-layer perceptron as shown in equations (2) and (3), so the output of the lth layer can be written as follows:

[0067]

[0068]

[0069] Among them, MSA represents multi-head self-attention, MLP represents multi-layer perceptron, LN(·) represents the normalization operator of the image, represents the multi-head attention output of the lth layer, z l Represents the description of the coded image at layer l.

[0070] Due to the information loss of the hybrid encoder of CNN and Transformer, this embodiment also includes compensation for the information loss of the hybrid encoder of CNN and Transformer, using a hybrid CNN-Transformer architecture as an encoder and cascade upsampling to achieve accurate positioning. Specifically including:

[0071] Similar to U-Net, skip connections are used to fuse multi-scale features from the hybrid encoder with upsampled features, and CNN is used as a feature extractor to generate feature maps instead of 1×1 patches extracted from the original image as input, thereby retaining more deep and shallow features to compensate for information loss.

[0072] Here, shallow features and deep features are concatenated together to reduce the spatial information loss caused by downsampling. Then there is a linear layer, and the concatenated feature size remains the same as the upsampled feature size.

[0073] Step S4, cascade decoder: First, the coded features obtained in step S3 are upsampled through the decoder, and then the upsampled coded features are combined with the high-resolution CNN feature map to achieve precise positioning. Finally, U-Net is used to restore local spatial information to enhance more accurate detail detection information, effectively reduce false positives in lung nodule detection, and provide more accurate images for the auxiliary diagnosis system.

[0074] In this embodiment, step S4 specifically includes:

[0075] Multiple upsampling steps are used to decode the hidden features to output the final segmentation mask map, specifically:

[0076] In the hidden features The sequence is reconstructed into After that, the cascaded decoder is realized by cascading multiple upsampling blocks to achieve To H×W full resolution, the cascaded multiple upsampling blocks include two upsampling, a 3×3 convolutional layer, and a ReLU layer in sequence;

[0077] Finally, the cascaded decoder and the hybrid encoder together form a U-shaped architecture, which achieves feature fusion by upsampling feature maps of different levels of resolution through skip connections.

[0078] The TransformerUnet combined architecture proposed in this invention is as follows Figure 2 As shown in the figure, it establishes a self-attention mechanism from the perspective of sequence-to-sequence prediction. In order to compensate for the loss of feature resolution brought by Transformer, TransformerUnet adopts a CNN-Transformer hybrid structure to utilize the high-resolution spatial information from CNN features and the global context information encoded by Transformer. Inspired by the design of U-Net, the self-attention features encoded by Transformer are then upsampled and combined with different high-resolution CNN features that are skipped from the encoding path to achieve accurate positioning. This design enables the overall network framework to retain the advantages of Transformer and is also beneficial for lung nodule image detection. Figure 1 It is a slice of the acquired lung CT image.

[0079] The network was built under the Ubuntu16 operating system, based on the Nvidia RTX2080Ti GPU hardware platform in the Python environment to build a deep learning framework, and trained using the LUNA16 and LIDC datasets. A large number of experiments have proved the feasibility of TransformerUnet model training and testing.

[0080] All experiments used data augmentation, such as random rotation and flipping. For the Transformer encoder, only ViT with 12 Transformer layers was used. For the design of the hybrid encoder, combining ResNet-50 and ViT, all Transformer architectures (i.e., ViT) and ResNet-50 were pre-trained on ImageNet, and the resolution and patch size of the input image were set to 224×224 and 16, respectively. Four cascaded upsampling blocks were required in CUP to reach the resolution of the original image. The model was trained using SGD optimizer with a learning rate of 0.01, a momentum of 0.9, and a weight decay of 1e -4 The default batch size is 24, the default number of training iterations is 20k for the LUNA16 dataset, and the default number of training iterations is 14k for the LIDC dataset.

[0081] The present invention is characterized in that, on the one hand, the CNN architecture (U-Net) is used to provide a way to extract low-level feature clues, which can well supplement such fine spatial details. On the other hand, the Transformer network is used to encode the tokenized image blocks from the convolutional neural network (CNN) feature map under the U-Net framework into an input sequence for extracting the global context. Finally, the decoder upsamples the encoded features and combines them with the high-resolution CNN feature map to achieve accurate positioning. With the combination of U-Net, Transformers can be used as a powerful encoder for the task of pulmonary nodule detection by restoring local spatial information.

[0082] Embodiment 2,

[0083] This embodiment discloses a system for detecting lung nodules in CT images based on the method described in Embodiment 1, which adopts a TransformerUnet combined architecture and specifically includes:

[0084] An image serialization module is used to reshape the slices of the input lung CT image into a set of patch sequences to perform tokenization;

[0085] A patch embedding module for mapping vectorized patch sequences to a latent 2D embedding space using a trainable linear mapping;

[0086] A hybrid encoder module of CNN and Transformer, which encodes tokenized image patches from CNN feature maps into input sequences for extracting global context through Transformer;

[0087] The cascade decoder module is used to first upsample the encoded features obtained by the hybrid encoder module of CNN and Transformer through the decoder, then combine the upsampled encoded features with the high-resolution CNN feature map to achieve precise positioning, and finally use U-Net to enhance more accurate detail detection information by restoring local spatial information.

[0088] The functions of the above modules correspond to those in Example 1 and will not be described in detail here.

[0089] The above only describes the basic principles and preferred embodiments of the present invention. Those skilled in the art may make many changes and improvements based on the above description, and these changes and improvements should fall within the protection scope of the present invention.

Claims

1. A lung nodule detection method based on CT images, characterized in that: The TransformerUnet combined architecture is used to detect lung nodules. The detection method includes the following steps: S1. Image serialization: reshape the slices of the input lung CT image into a set of patch sequences to perform tokenization; S2, using patch embedding, using a trainable linear mapping to map the vectorized patch sequence to a potential two-dimensional embedding space; step S2 specifically includes: S21. In order to encode the spatial information of the patch sequence, a specific position encoding added to the patch sequence embedding is learned to preserve the position information, as shown in the following formula: in, is the patch embedding map, Represents the position embedding information, and D is the dimension of the input patch; S22. In order to restore the spatial order of the patch after embedding, the size of the encoded feature is first become Use 1×1 convolution to reduce the channel size of the feature to the number of feature classes, and then directly upsample the feature map to the full resolution H×W for predicting the final segmentation result; S3, establishing a hybrid encoder of CNN and Transformer: encoding the tokenized image blocks from the CNN feature map into an input sequence for extracting the global context through the Transformer; step S3 specifically includes: The hybrid encoder of CNN and Transformer consists of l layers of multi-head self-attention and multi-layer perceptron as shown in equations (2) and (3), so the output of the lth layer can be written as follows: Among them, MSA represents multi-head self-attention, MLP represents multi-layer perceptron, LN(·) represents the normalization operator of the image, represents the multi-head attention output of the lth layer, z l Represents the description of the coded image at layer l; S4, cascade decoder: first, the coded features obtained in step S3 are upsampled by the decoder, and then the upsampled coded features are combined with the high-resolution CNN feature map to achieve accurate positioning, and finally, U-Net is used to enhance more accurate detail detection information by restoring local spatial information; step S4 specifically includes: Multiple upsampling steps are used to decode the hidden features to output the final segmentation mask map, specifically: In the hidden features The sequence is reconstructed into After that, the cascaded decoder is realized by cascading multiple upsampling blocks to achieve To H×W full resolution, the cascaded multiple upsampling blocks include two upsampling, a 3×3 convolutional layer, and a ReLU layer in sequence; Finally, the cascaded decoder and the hybrid encoder together form a U-shaped architecture, which achieves feature fusion by upsampling feature maps of different levels of resolution through skip connections.

2. The method according to claim 1, characterized in that In step S1, the lung CT image is assumed to be H×W is the spatial resolution, and C is the number of channels.

3. The method according to claim 2, characterized in that Step S1 specifically includes: Tokenization is performed by reshaping the input lung CT image x into a sequence of patches Where p is the sequence size, so the size of each patch is p×p, and the number of patches per image is That is, the length of the input sequence.

4. The method according to any one of claims 1 to 3, characterized in that: It also includes compensation for the information loss of the hybrid encoder of CNN and Transformer, including: Similar to U-Net, skip connections are used to fuse multi-scale features from the hybrid encoder with upsampled features, and CNN is used as a feature extractor to generate feature maps instead of 1×1 patches extracted from the original image as input, thereby retaining more deep and shallow features to compensate for information loss.

5. A pulmonary nodule image detection system based on CT images, characterized in that: The method for detecting lung nodules based on CT images according to any one of claims 1 to 4 is implemented, using a TransformerUnet combined architecture, specifically comprising: An image serialization module is used to reshape the slices of the input lung CT image into a set of patch sequences to perform tokenization; A patch embedding module for mapping vectorized patch sequences to a latent 2D embedding space using a trainable linear mapping; A hybrid encoder module of CNN and Transformer, which encodes tokenized image patches from CNN feature maps into input sequences for extracting global context through Transformer; The cascade decoder module is used to first upsample the encoded features obtained by the hybrid encoder module of CNN and Transformer through the decoder, then combine the upsampled encoded features with the high-resolution CNN feature map to achieve precise positioning, and finally use U-Net to enhance more accurate detail detection information by restoring local spatial information.

Images