A method, device and equipment for segmenting CT images of pneumonia

Abstract

The present invention relates to a method for segmenting pneumonia CT images, which includes inputting the preprocessed pneumonia CT image to be detected into a pre-trained U-Former model, extracting local features through a Stem module to generate a feature image; inputting the feature image into four encoding layers connected in series in the forward propagation direction of the encoder for downsampling. The processing of each encoding layer for the input image includes: sending the input image through a convolutional block to N cascaded hybrid modules; using the parallel efficient multi-head self-attention module EMSHA and convolutional module in the hybrid module to process the convolved image respectively; EMSHA divides the input image with a size of H×W×C into #imgabs0# patches, each patch having a size of p h ×p w , and consists of p h ×p w tokens; after division, self-attention calculation is performed. The tokens in each patch only perform self-attention calculation with the tokens at the corresponding positions in other patches, and the output feature image and the output image after passing through the convolutional module are sent to the residual module for output; then, through upsampling by a decoder connected to the encoder through a skip connection, the segmentation result of the pneumonia CT image lesion is output.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and particularly to a method, device and equipment for segmenting CT images of pneumonia. Background Art

[0002] Deep learning is one of the machine learning methods, which can perform efficient feature extraction and data modeling by establishing a multi-layer neural network. Compared with traditional machine learning methods, deep learning can automatically learn more complex patterns and rules, can process a large amount of high-dimensional data including images, speech, natural language, etc., and has broad application prospects. With the rapid development and application of deep learning technology, its role in the field of medical image processing and analysis has become increasingly significant. Compared with traditional medical image analysis methods, deep learning can automatically learn and extract features from a large amount of data, thus providing a more reliable and accurate basis for the diagnosis of related diseases. In addition, using deep learning technology can realize the automatic analysis and recognition of medical images, which can greatly improve the work efficiency and accuracy of doctors. With the continuous development and application of deep learning technology, it has become a promising development direction in the modern medical field and is expected to make greater contributions to the development of human health.

[0003] CT (Computed Tomography), as a non-invasive imaging method, can depict the characteristic manifestations of the lungs and help doctors diagnose diseases in a timely manner. However, with the increase in the number of infected patients, the CT data of patients' lungs has gradually increased. Due to the differences in artificial experience, the accuracy cannot be guaranteed, and it is very easy to cause misjudgment, increasing the risk of misdiagnosis and bringing a greater burden to doctors. Therefore, developing a computer-aided system to automatically analyze CT images and output pneumonia disease type information is beneficial for radiologists to make decisions and also helps in the early detection and treatment of lung diseases.

[0004] Currently, the research on CT image analysis based on computer-aided systems includes: (1) Wang Guangyu from Donghua University conducted a relatively systematic and in-depth study on the segmentation and classification methods of pneumonia CT images based on deep learning in his master's thesis "Research on Pneumonia CT Image Detection Method Based on Deep Learning". This thesis constructed a Lung Seg-Net model to pre-segment CT image data and remove the background regions in the dataset images except for the lung regions; then, on the segmented dataset, segmentation experiments were carried out using U-Net, U-Net++, U-Net+Res Net101, and Deep Lab V3+ models respectively; and a convolutional attention module and a depthwise separable convolutional module were introduced into the Deep Lab V3+ model to improve the model performance. (2) In the paper "Improved U-Net Method for Pneumonia Image Segmentation" published by Song Yao in "Computer Engineering and Applications", an automatic segmentation method based on an improved U-Net model was proposed for the problem of pneumonia lesion segmentation. The pre-trained EfficientNet-B0 network was used in the encoder to extract features of effective information; a new upsampling operation structure was used in the decoder to fully obtain the detailed feature information of the lesion edges, and finally, the accuracy of segmentation was improved through the integration of model snapshots; the experimental results on the public dataset showed that this method could effectively segment the pneumonia lesion regions and had good segmentation performance. (3) Chinese Patent CN111932559B provides a deep learning-based pneumonia lung lesion region segmentation system, which can accurately segment the lung lesion regions from the lung regions, quickly and automatically outline the contours of COVID-19 infection regions, facilitate the qualitative evaluation of diseases in on-site applications, save a large amount of manpower and time, and at the same time improve the diagnostic accuracy of COVID-19 suspected patients and reduce the relatively high false negative probability. However, the neural networks adopted by the above technologies in the field of image segmentation are mainly fully convolutional networks. Due to the limitations of convolutional operations, there are certain difficulties in extracting global features. Even though some technologies have introduced self-attention mechanisms, their high computational complexity and the limitation of requiring more computing resources also pose great challenges to their applications in the field of medical image segmentation.

[0005] In summary, with the development of deep learning technology and the continuous optimization of neural network structures, image segmentation algorithms based on neural networks have achieved extensive applications and important results in many fields such as medical imaging, natural images, and remote sensing images. However, in the field of medical image segmentation, due to the complexity and variability of medical images, existing fully convolutional neural networks still have some problems. On the one hand, these algorithms still cannot completely and accurately segment different types of lesions or tissues; on the other hand, due to the limitations of convolutional operations, these algorithms also cannot well capture long-range dependencies, so they may not be able to comprehensively analyze the entire image. Currently, although some researchers have introduced self-attention mechanisms into convolutional neural networks to improve the problem of global feature extraction, these network models often face high computational complexity and limitations of computing resources, resulting in very difficult applications in the field of medical image segmentation. This situation is mainly because the self-attention mechanism needs to process the relationships between all pixels in the image, causing the computational amount to increase exponentially with the increase of the image size and depth. Summary of the Invention

[0006] To this end, the technical problem to be solved by the present invention is to overcome the problem of high computational complexity of the image segmentation model in the prior art.

[0007] To solve the above technical problem, the present invention provides a method for segmenting pneumonia CT images, including:

[0008] Obtain the pneumonia CT image to be detected and perform preprocessing; input the preprocessed pneumonia CT image to be detected into a pre-trained U-Former model;

[0009] The U-Former model has a U-shaped structure, and uses convolution to extract local features from the input image to generate a feature image;

[0010] Input the feature image into the encoder for downsampling. The encoder includes a plurality of encoding layers connected in series along the forward propagation direction. The processing of the input feature image by each encoding layer includes:

[0011] After the input feature image is convolved by a convolutional block, it is sent to a plurality of cascaded hybrid modules, and the image output by the last hybrid module is taken as the output feature image of this encoding layer;

[0012] Each hybrid module uses a parallel efficient multi-head self-attention module and a convolutional module to process the convolved image respectively; the efficient multi-head self-attention module divides the input feature image with a size of into Patchs, where is the size of each Patch, and each Patch consists of It consists of

[0013] The decoding layer of the decoder is skip-connected to the corresponding encoding layer. Upsampling is performed based on the output feature image of the corresponding encoding layer and the output image of the previous decoding layer. The result of upsampling in the last decoding layer passes through a convolutional module to output the lesion segmentation result of the pneumonia CT image.

[0014] In one embodiment of the present invention, the encoding end of the U-Former model includes:

[0015] The Stem module is used to extract local features from the input image using convolution to generate a feature image.

[0016] The encoder is used to downsample the feature image, and it includes four encoding layers connected in series along the forward propagation direction. The encoding layer includes, along the forward propagation direction, in sequence:

[0017] A convolutional block;

[0018] Multiple hybrid modules.

[0019] In one embodiment of the present invention, the decoding end of the U-Former model includes a decoder for upsampling the input feature image, which includes, along the forward propagation direction, in sequence:

[0020] The first decoding layer, whose input is the output image of the third encoding layer and the output image of the fourth encoding layer after upsampling;

[0021] The second decoding layer, whose input is the output image of the second encoding layer and the output image of the first decoding layer after upsampling;

[0022] The third decoding layer, whose input is the output image of the first encoding layer and the output image of the second decoding layer after upsampling.

[0023] In one embodiment of the present invention, the Stem module includes a 3×3 convolutional block with a stride of 2 and two 3×3 convolutional blocks with a stride of 1 connected in series along the forward propagation direction.

[0024] In one embodiment of the present invention, the hybrid module sends the input image through a normalization module and then into a parallel efficient multi-head self-attention module and a convolutional module respectively. The outputs of the two modules pass through a 1×1 convolution, are skip-connected to the normalized input image, and then output through a residual MLP module.

[0025] In one embodiment of the present invention, the preprocessing includes data augmentation for the pneumonia CT image to be detected.

[0026] In one embodiment of the present invention, on the path where the decoding layer of the decoder is skip-connected to the corresponding encoding layer, there is a multi-scale feature fusion module MSFF. The MSFF is a parallel multi-branch structure. After convolving the input image of the MSFF, it is concatenated with the output images of multiple branches and then convolved to output. The parallel multi-branch structure includes:

[0027] The first branch, which includes a 3×3 convolution block with a stride of 1;

[0028] The second branch, which includes a 3×3 convolution block with a stride of 1 and a 3×3 convolution block with a stride of 2 connected in series along the forward propagation direction;

[0029] The third branch, which includes a 3×3 convolution block with a stride of 1, a 3×3 convolution block with a stride of 2, and a 3×3 convolution block with a stride of 3 connected in series along the forward propagation direction;

[0030] In the first branch, the second branch, and the third branch, after each convolution block, a batch normalization layer and an activation function layer are connected in series.

[0031] In one embodiment of the present invention, on the path where the decoding layer of the decoder is skip-connected to the corresponding encoding layer, there is a channel attention mechanism CA, which is used to encode each channel of the input image along the horizontal coordinate direction and the vertical coordinate direction using pooling kernels of sizes (H,1) and (1,W) respectively to obtain a pair of direction-aware attention maps. After concatenate, a 1×1 convolution block, and an activation function, a joint feature map is generated. Along the spatial dimension, a split operation is performed on the joint feature map to obtain a width feature map and a height feature map. After passing through a 1×1 convolution and an activation function respectively, a width attention vector and a height attention vector are obtained, and the output feature image is redistributed.

[0032] The embodiment of the present invention also provides another pneumonia CT image segmentation device, including:

[0033] An image input module, which is used to obtain the pneumonia CT image to be detected and perform preprocessing; input the preprocessed pneumonia CT image to be detected into a pre-trained U-Former model;

[0034] Coding module. The U-Former model has a U-shaped structure, extracts local features from the input image using convolution, and generates a feature image. The feature image is input into the encoder for downsampling. The encoder includes a plurality of coding layers connected in series along the forward propagation direction. The processing of the input feature image by each coding layer includes: after convolving the input feature image through a convolution block, sending it to a plurality of cascaded hybrid modules, and obtaining the image output by the last hybrid module as the output feature image of this coding layer. Each hybrid module uses a parallel efficient multi-head self-attention module and a convolution module to process the convolved image respectively. The efficient multi-head self-attention module divides the input feature image with a size of into Patches, is the size of each Patch, and each Patch consists of Tokens; after division, self-attention calculation is performed. The Tokens in each Patch only perform self-attention calculation with the Tokens at the corresponding positions in other Patches. According to the self-attention of each Token, a feature image with global information and the output image after passing through the convolution module are obtained and sent to the residual module for output;

[0035] Decoding module. The decoding layer of the decoder is skip-connected to the corresponding coding layer, and upsampling is performed according to the output feature image of the corresponding coding layer and the output image of the previous decoding layer. The result of upsampling by the last decoding layer passes through the convolution module to output the lesion segmentation result of the pneumonia CT image.

[0036] The embodiment of the present invention also provides a pneumonia CT image segmentation device, including:

[0037] A pneumonia CT image acquisition device for acquiring pneumonia CT images;

[0038] A processor, communicatively connected to the pneumonia CT image acquisition device, for executing a computer program to implement the steps of the pneumonia CT image segmentation method as described above, and obtaining the lesion segmentation result of the pneumonia CT image;

[0039] A display device, communicatively connected to the processor, for obtaining and displaying the lesion segmentation result of the pneumonia CT image.

[0040] The above technical solution of the present invention has the following advantages compared with the prior art:

[0041] The pneumonia CT image segmentation method described in the present invention uses a hybrid module in the encoder that combines an efficient multi-head self-attention module and a convolutional module in parallel to extract features from the input image. The efficient multi-head self-attention module and convolutional module based on Transformer not only leverage the advantage of Transformer in extracting global context information but also retain the ability of convolutional neural networks to extract local detailed features. Moreover, by using the efficient multi-head self-attention module, when performing self-attention calculations, the tokens in each Patch only perform self-attention calculations with the tokens at the corresponding positions in other Patches, thereby reducing the computational cost.

[0042] The U-Former model provided by the present invention adds a multi-scale feature fusion module and a channel attention module to the path of the decoder-encoder skip connection. The channel attention module takes into account coordinate information and can not only obtain long-range dependencies in one direction but also retain coordinate information in the other direction. The multi-scale feature fusion module uses dilated convolution to make full use of both local and global features. The present invention enhances the extraction of detailed features from the input image, improves the sensitivity to pneumonia lesions, and is conducive to improving the segmentation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] To make the content of the present invention easier to understand clearly, the following further details the present invention according to specific embodiments of the present invention in combination with the accompanying drawings, where

[0044] Figure 1 is the flowchart of the steps of the pneumonia CT image segmentation method provided by the present invention;

[0045] Figure 2 is the structural schematic diagram of the U-Former model provided by the present invention;

[0046] Figure 3 is the structural schematic diagram of the hybrid module provided by the present invention;

[0047] Figure 4 is the calculation schematic diagram of the standard multi-head self-attention mechanism in the prior art;

[0048] Figure 5 is the feature image input to the efficient multi-head self-attention mechanism provided by the present invention;

[0049] Figure 6 is the schematic diagram of the image division of the efficient multi-head self-attention mechanism provided by the present invention;

[0050] Figure 7 is the calculation schematic diagram of the efficient multi-head self-attention mechanism provided by the present invention;

[0051] Figure 8It is another computational schematic diagram of the efficient multi-head self-attention mechanism provided by the present invention;

[0052] Figure 9 It is a structural schematic diagram of the multi-scale feature fusion module provided by the present invention;

[0053] Figure 10 It is a structural schematic diagram of the channel attention mechanism provided by the present invention. Specific Embodiments

[0054] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, so that those skilled in the art can better understand the present invention and be able to implement it, but the embodiments given are not intended to limit the present invention.

[0055] Embodiment 1: Refer to Figure 1 As shown, it is a method for segmenting pneumonia CT images provided by the present invention. The U-Former model is used to segment the collected pneumonia CT images to be detected. The specific steps include:

[0056] Obtain the pneumonia CT image to be detected and perform preprocessing; input the preprocessed pneumonia CT image to be detected into a pre-trained U-Former model;

[0057] The U-Former model has a U-shaped structure, and local features are extracted from the input image using convolution to generate a feature image;

[0058] Input the feature image into the encoder for downsampling. The encoder includes a plurality of encoding layers connected in series along the forward propagation direction. The processing of the input feature image by each encoding layer includes:

[0059] After the input feature image is convolved by a convolution block, it is sent to a plurality of cascaded hybrid modules, and the image output by the last hybrid module is taken as the output feature image of this encoding layer;

[0060] Each hybrid module uses a parallel efficient multi-head self-attention module and a convolution module to process the convolved image respectively; the efficient multi-head self-attention module divides the input feature image of size into Patches, where is the size of each Patch, and each Patch consists of

[0061] The decoding layer of the decoder is skip-connected to the corresponding encoding layer. It performs upsampling based on the output feature image of the corresponding encoding layer and the output image of the previous decoding layer. The result of upsampling in the last decoding layer passes through a convolution module to output the lesion segmentation result of the pneumonia CT image.

[0062] Specifically, referring to Figure 2 As shown, it is a schematic structural diagram of the U-Former model provided by the present invention, specifically including:

[0063] Encoding end:

[0064] The Stem module is used to extract local features from the input image using convolution to generate a feature image. It includes a 3×3 convolution block with a stride of 2 and two 3×3 convolution blocks with a stride of 1 connected in series along the forward propagation direction.

[0065] The encoder is used to perform downsampling on the feature image. It includes four encoding layers connected in series along the forward propagation direction, and each layer outputs feature images of different sizes. The encoding layer includes a 3×3 convolution block with a stride of 2 and multiple hybrid modules connected in series along the forward propagation direction.

[0066] Decoding end:

[0067] The decoder is used to perform upsampling on the input feature image. Along the forward propagation direction, it includes: the first decoding layer, whose input is the output image of the third encoding layer and the output image of the fourth encoding layer after upsampling; the second decoding layer, whose input is the output image of the second encoding layer and the output image of the first decoding layer after upsampling; the third decoding layer, whose input is the output image of the first encoding layer and the output image of the second decoding layer after upsampling.

[0068] The output convolution is used to perform convolution on the output image of the third decoding layer and output.

[0069] Specifically, referring to Figure 3 As shown, it is a schematic structural diagram of the hybrid module provided by the present invention, including a convolution module and an efficient multi-head self-attention mechanism combined through a parallel architecture, enabling the U-Former model to capture global information while also paying attention to local detailed features, thereby enhancing the model's ability to segment lesions in pneumonia CT images. The hybrid module sends the input image through a normalization module and then into the parallel efficient multi-head self-attention module and convolution module respectively. The outputs of the two modules pass through a 1×1 convolution, are skip-connected to the normalized input image, and then output through a residual MLP module.

[0070] Specifically, in the embodiments of the present invention, for the multi-head self-attention mechanism with high computational complexity in the Transformer, inspired by MobileViT, the efficient multi-head self-attention mechanism EMHSA (Efficient Multi-head Self-attention) is proposed. Refer to Figure 4 As shown, taking a 3×3 feature image as an example, in the traditional multi-head self-attention mechanism, the channel dimension is ignored, and self-attention calculations are performed between each Token and all Tokens at other positions; the Token represents the vector of the image block after being split and flattened. To reduce the computational cost of the self-attention mechanism, refer to Figure 4 As shown, the embodiments of the present invention propose EMHSA. Refer to Figure 5 As shown, for a given feature image of size , refer to Figure 6 As shown, first, the feature image is divided into Patches, each of size , and there are Patches in total. Each Patch consists of Tokens; refer to Figure 7 As shown, when performing self-attention calculations based on the EMHSA, the Tokens in each Patch only perform self-attention calculations with the Tokens at the corresponding positions in other Patches, and no longer calculate self-attention with all Tokens, thereby reducing the computational cost.

[0071] Specifically, refer to Figure 8 As shown, it is a schematic diagram for performing self-attention calculations based on the efficient multi-head self-attention mechanism provided by the embodiments of the present invention. Among them, the size of the feature image is 6×6, and are both set to 2. Each small square represents a Token, and 4 Tokens form a Patch.

[0072] The embodiments of the present invention improve the traditional multi-head self-attention mechanism, and use the improved efficient multi-head self-attention mechanism in parallel with the convolution, which not only gives full play to the advantage of the Transformer in extracting global context information, but also retains the extraction of local detail features by the convolutional neural network, and greatly reduces the computational amount of the Transformer.

[0073] The embodiments of the present invention also provide another pneumonia CT image segmentation device, including:

[0074] An image input module, configured to obtain a pneumonia CT image to be detected, perform preprocessing; and input the preprocessed pneumonia CT image to be detected into a pre-trained U-Former model;

[0075] An encoding module, where the U-Former model has a U-shaped structure. It extracts local features from the input image using convolution to generate a feature image. The feature image is input into the encoder for downsampling. The encoder includes a plurality of encoding layers connected in series along the forward propagation direction. The processing of the input feature image by each encoding layer includes: after convolving the input feature image through a convolution block, sending it to a plurality of cascaded hybrid modules, and obtaining the image output by the last hybrid module as the output feature image of this encoding layer. Each hybrid module uses a parallel efficient multi-head self-attention module and a convolution module to process the convolved image respectively. The efficient multi-head self-attention module divides the input feature image of size into patches, where is the size of each patch, and each patch consists of tokens. After division, self-attention calculation is performed. The tokens in each patch only perform self-attention calculation with the tokens at the corresponding positions in other patches. Based on the self-attention of each token, a feature image with global information and the output image after passing through the convolution module are sent to the residual module for output.

[0076] A decoding module. The decoding layer of the decoder is skip-connected to the corresponding encoding layer. It performs upsampling based on the output feature image of the corresponding encoding layer and the output image of the previous decoding layer. The result of upsampling by the last decoding layer passes through a convolution module to output the lesion segmentation result of the pneumonia CT image.

[0077] An embodiment of the present invention also provides a pneumonia CT image segmentation device, including:

[0078] A pneumonia CT image acquisition device for acquiring pneumonia CT images;

[0079] A processor, communicatively connected to the pneumonia CT image acquisition device, for executing a computer program to implement the steps of the pneumonia CT image segmentation method as described above, and obtaining the lesion segmentation result of the pneumonia CT image;

[0080] A display device, communicatively connected to the processor, for obtaining and displaying the lesion segmentation result of the pneumonia CT image.

[0081] Embodiment 2: Based on the above embodiment, in this embodiment, in order to enhance the representation ability of the feature image, during the skip connection between the decoder and the encoder, a multi-scale feature fusion module MSDD is constructed using dilated convolution, and a channel attention mechanism CA (Coordinate Attention) is introduced. The multi-scale feature fusion module enables the full utilization of both local and global features by fusing multi-scale features, thereby improving the CT image lesion segmentation ability of the U-Former model. The channel attention mechanism CA is a lightweight and efficient attention mechanism that enables the U-Former model to extract regions of interest from the input feature image and resist the interference of confounding information, thus focusing on the key information of effective targets; it adds coordinate information to the channel attention, different from the traditional channel attention mechanism that converts the tensor-form features into vector-form features and uses pooling operations for processing. The channel attention mechanism CA splits the channel attention into two parts, encodes the features of each part in different directions, and finally combines these two parts; using the channel attention mechanism provided by the embodiments of the present invention, long-range dependencies can be obtained in one direction, and coordinate positions can be retained in the other direction.

[0082] Referring to Figure 9 As shown, the multi-scale feature fusion module provided by the present invention is arranged on the path of the skip connection between the decoding layer and the corresponding encoding layer. The MSFF is a parallel multi-branch structure, and the input image of the MSFF after convolution is concatenated with the output images of multiple branches and then convolved and output. The parallel multi-branch structure includes:

[0083] The first branch, which includes a 3×3 convolution block with a stride of 1;

[0084] The second branch, which includes a 3×3 convolution block with a stride of 1 and a 3×3 convolution block with a stride of 2 connected in series along the forward propagation direction;

[0085] The third branch, which includes a 3×3 convolution block with a stride of 1, a 3×3 convolution block with a stride of 2, and a 3×3 convolution block with a stride of 3 connected in series along the forward propagation direction;

[0086] In the first branch, the second branch, and the third branch, a batch normalization layer and an activation function layer are connected in series after each convolution block.

[0087] Referring to Figure 10As shown, the channel attention mechanism module provided by the present invention is arranged on the path of the skip connection between the decoding layer and the corresponding encoding layer, and is used to encode each channel of the input image along the horizontal coordinate direction and the vertical coordinate direction by using pooling kernels of sizes (H, 1) and (1, W) respectively to obtain a pair of direction-aware attention maps, and generate a joint feature map through concatenate, a 1×1 convolution block and an activation function; perform a split operation on the joint feature map along the spatial dimension to obtain a width feature map and a height feature map, and respectively pass through a 1×1 convolution and an activation function to obtain a width attention vector and a height attention vector, and perform reallocation to output a feature image.

[0088] In the embodiment of the present invention, a multi-scale feature fusion module and a channel attention module are introduced on the path of the skip connection between the encoding layer and the corresponding decoding layer, which strengthens the extraction of detailed features; the multi-scale feature fusion module enables full utilization of both local features and global features; the channel attention module enables the U-Former model to extract regions of interest from the input and resist the interference of confounding information, so as to concentrate on the key information of effective targets and improve the model segmentation accuracy.

[0089] Based on the above embodiment, in this embodiment, data augmentation is performed on the existing pneumonia CT image dataset, and the augmented dataset is divided into a training set, a validation set and a test set according to a ratio of 8:1:1.

[0090] Specifically, one of the existing pneumonia CT image datasets used in this embodiment is from the "2019 Novel Coronavirus Information Database" of the National Center for Biotechnology Information; this dataset contains CT scan images of 150 patients and mask annotations from professional doctors; among them, the sizes of the CT images and the masks are both 512×512, and each pixel in the mask corresponds to a value from 0 to 3, representing background, lung field, ground-glass opacity and lung consolidation respectively. Compared with other similar databases, this dataset is more comprehensive in terms of the number of samples collected and feature performance, and has the same image size, balanced category numbers, high image quality and has been studied and certified by national professionals. Another existing pneumonia CT image dataset used in this embodiment is a dataset from the National Center for Biotechnology Information that has been finely annotated by professionals; the original CT images in this dataset are from the CT images of the lungs of COVID-19 infected patients in the classification dataset; the annotation of this dataset is the same as that of the first dataset, both having four categories, and is a detailed mask annotation made by professional doctors based on their own experience and cognition with reference to the annotation of the first dataset.

[0091] In the original lung CT dataset, although various CT samples contain different characteristic differences, they are relatively fixed in terms of shape, angle, quantity, etc. In terms of complexity, it does not meet the actual application in the real scenario. Therefore, in this embodiment, the two datasets adopted in this embodiment are merged, and random image enhancement processing is performed on each CT image sample participating in the training in the merged dataset; the image enhancement processing mainly includes flipping in the horizontal and vertical directions, rotating at random angles, changing the contrast, etc.; finally, after sorting and screening, the dataset is expanded.

[0092] Build the convolutional neural network U-Former constructed by the present invention through the Pytorch deep learning framework, use the divided training set as the input of U-Former, and train the neural network model parameters; set the initial learning rate to 0.001, iterate 200 epochs, and the adopted learning rate decay strategy is cosine annealing decay, use CrossEntropyLoss as the loss function, and optimize the model through the SGD optimizer.

[0093] Based on the trained U-Former network model, use the Dice coefficient, sensitivity Sensitivity (Sen), and intersection over union (IoU) as evaluation indicators, and verify the performance of the neural network model on the test set.

[0094] Dice coefficient: ;

[0095] Sensitivity: ;

[0096] Intersection over union: ;

[0097] Among them, TP represents True Positive, which refers to the number of samples where positive samples are simultaneously predicted as positive; FN represents False Negative, which refers to the number of samples where positive samples are predicted as negative; FP represents False Positive, which refers to the number of samples where negative samples are predicted as positive; TN represents True Negative, which refers to the number of samples where negative samples are predicted as negative.

[0098] In this embodiment, the Dice coefficient, sensitivity, and intersection over union of the pneumonia CT image segmentation method provided by the present invention in the pneumonia classification task are 82.54%, 85.70%, and 80.01% respectively.

[0099] According to the evaluation metrics of the training set, adjust the hyperparameters of the neural network model to optimize the model. Use the optimized U-Former model to conduct tests on the test set, and continuously adjust the model hyperparameters according to the test results to optimize the model. Based on the optimized U-Former model, validate the validation set to obtain the lesion segmentation results of pneumonia CT images, so as to provide accurate disease data for doctors and facilitate the subsequent diagnosis and treatment of patients.

[0100] The method for segmenting lesions in pneumonia CT images provided by the present invention improves the multi-head self-attention in Transfoemer to obtain efficient multi-head self-attention EMSHA, and applies the hybrid module parallel with convolution of EMSHA to the encoder of the U-Former model and introduces it into the field of medical image segmentation. It not only gives full play to the advantages of Transformer in extracting global context information but also retains the extraction of local detail features by the convolutional neural network. The present invention strengthens the extraction of detail features by introducing a multi-scale feature fusion module and a channel attention module between the encoder and the decoder, extracts the key information required for segmentation from rich features, and improves the segmentation accuracy. The U-foemer model has good robustness and strong adaptability in the segmentation task of CT images. The present invention promotes the application of the combination of Transformer, convolutional network, multi-scale feature fusion module and channel self-attention mechanism in the field of medical image analysis, and has important research value and practical significance for the diagnostic analysis of medical images.

[0101] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0102] The present application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram can be implemented by computer program instructions, and the combination of the processes and / or blocks in the flowchart and / or block diagram can also be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for realizing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0103] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer-readable memory produce a manufacture including an instruction device that implements the functions specified in one or more of the processes and / or blocks Figure 1 of the process or processes and / or blocks Figure 1 specified in the block or blocks.

[0104] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable apparatus provide steps for implementing the functions specified in one or more of the processes and / or blocks Figure 1 of the process or processes and / or blocks Figure 1 specified in the block or blocks.

[0105] Obviously, the above embodiments are merely examples for clear illustration and are not limitations on the implementation. For those of ordinary skill in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to exhaustively list all the implementation manners here. And the obvious changes or modifications derived therefrom are still within the protection scope of the present invention.

Claims

1. A method for segmenting CT images of pneumonia, characterized in that, Including: Obtain the CT image of pneumonia to be detected and perform preprocessing; Input the preprocessed CT image of pneumonia to be detected into the pre-trained U-Former model; The U-Former model has a U-shaped structure, including an encoding end and a decoding end; the encoding end includes a Stem module and an encoder, and the encoder includes four encoding layers connected in series along the forward propagation direction; each encoding layer includes a convolutional block and multiple hybrid modules along the forward propagation direction in sequence; the decoding end includes a decoder for upsampling the input feature image, which includes, along the forward propagation direction: the first decoding layer, whose input is the output image of the third encoding layer and the output image of the fourth encoding layer after upsampling; the second decoding layer, whose input is the output image of the second encoding layer and the output image of the first decoding layer after upsampling; the third decoding layer, whose input is the output image of the first encoding layer and the output image of the second decoding layer after upsampling; The Stem module extracts local features from the input image using convolution to generate a feature image; Input the feature image into the encoder for downsampling. In each encoding layer, after the input feature image is convolved by the convolutional block, it is sent to multiple cascaded hybrid modules, and the image output by the last hybrid module is taken as the output feature image of this encoding layer; Each hybrid module uses parallel efficient multi-head self-attention modules and convolutional modules to process the convolved images respectively; the efficient multi-head self-attention module processes the input feature image of size and divides it into Patches, where is the size of each Patch, and each Patch consists of tokens; after the division, self-attention calculation is performed, and the tokens in each Patch only perform self-attention calculation with the tokens at the corresponding positions in other Patches; based on the self-attention of each token, a feature image with global information and the output image after passing through the convolutional module are obtained and sent to the residual module for output; The decoding layers of the decoder are skip-connected to the corresponding encoding layers. According to the output feature image of the corresponding encoding layer and the output image of the previous decoding layer, upsampling is performed. The result of upsampling by the last decoding layer passes through a convolutional module to output the lesion segmentation result of the CT image of pneumonia.

2. The pneumonia CT image segmentation method according to claim 1, wherein, The Stem module includes a 3×3 convolutional block with a stride of 2, two 3×3 convolutional blocks with a stride of 1 connected in series along the forward propagation direction in sequence.

3. The pneumonia CT image segmentation method according to claim 1, wherein The hybrid module normalizes the input image and then sends it into a parallel efficient multi-head self-attention module and a convolutional module respectively. The outputs of the two modules pass through a 1×1 convolution, are skip-connected to the normalized input image, and then output through a residual MLP module.

4. The pneumonia CT image segmentation method according to claim 1, wherein, The preprocessing includes data augmentation for the CT image of pneumonia to be detected.

5. The pneumonia CT image segmentation method according to claim 1, wherein, A multi-scale feature fusion module MSFF is included on the path where the decoding layers of the decoder are skip-connected to the corresponding encoding layers. The MSFF is a parallel multi-branch structure. After convolving the input image of the MSFF, it is spliced and convolved with the output images of multiple branches and then output. The parallel multi-branch structure includes: The first branch, which includes a 3×3 convolutional block with a stride of 1; The second branch, which includes a 3×3 convolutional block with a stride of 1 and a 3×3 convolutional block with a stride of 2 connected in series along the forward propagation direction in sequence; The third branch, which includes a 3×3 convolutional block with a stride of 1, a 3×3 convolutional block with a stride of 2, and a 3×3 convolutional block with a stride of 3 connected in series along the forward propagation direction in sequence; In the first branch, the second branch, and the third branch, a batch normalization layer and an activation function layer are connected in series after each convolutional block.

6. The pneumonia CT image segmentation method according to claim 1, wherein On the path of the skip connection between the decoding layer and the corresponding encoding layer of the decoder, there is a channel attention mechanism CA, which is used to encode each channel of the input image along the horizontal coordinate direction and the vertical coordinate direction using pooling kernels of sizes (H,1) and (1,W) respectively to obtain a pair of direction-aware attention maps. After concatenation, a 1×1 convolutional block, and an activation function, a joint feature map is generated. The joint feature map is split along the spatial dimension to obtain a width feature map and a height feature map. After passing through a 1×1 convolution and an activation function respectively, a width attention vector and a height attention vector are obtained, and the output feature image is redistributed.

7. A pneumonia CT image segmentation device, characterized in that, Comprising: An image input module, which is used to obtain the pneumonia CT image to be detected and perform preprocessing; Input the preprocessed pneumonia CT image to be detected into a pre-trained U-Former model; Encoding module, the U-Former model has a U-shaped structure, including an encoding end and a decoding end; the encoding end includes a Stem module and an encoder, and the encoder includes four encoding layers connected in series along the forward propagation direction; each encoding layer includes a convolutional block and multiple hybrid modules in sequence along the forward propagation direction; the decoding end includes a decoder for upsampling the input feature image, which includes, in sequence along the forward propagation direction: the first decoding layer, whose input is the output image of the third encoding layer and the output image of the fourth encoding layer after upsampling; the second decoding layer, whose input is the output image of the second encoding layer and the output image of the first decoding layer after upsampling; the third decoding layer, whose input is the output image of the first encoding layer and the output image of the second decoding layer after upsampling; the Stem module extracts local features from the input image using convolution to generate a feature image; the feature image is input into the encoder for downsampling. In each encoding layer, after the input feature image is convolved by the convolutional block, it is sent to multiple serially connected hybrid modules, and the image output by the last hybrid module is taken as the output feature image of this encoding layer; each hybrid module uses a parallel efficient multi-head self-attention module and a convolutional module to process the convolved image respectively; the efficient multi-head self-attention module divides the input feature image with a size of into patches, is the size of each patch, and each patch consists of tokens; after division, self-attention calculation is performed. The tokens in each patch only perform self-attention calculation with the tokens at the corresponding positions in other patches; based on the self-attention of each token, a feature image with global information and the output image after passing through the convolutional module are sent to the residual module for output; A decoding module, where the decoding layer of the decoder is skip-connected to the corresponding encoding layer, and upsampling is performed according to the output feature image of the corresponding encoding layer and the output image of the previous decoding layer. The result of upsampling of the last decoding layer passes through a convolutional module to output the lesion segmentation result of the pneumonia CT image.

8. A pneumonia CT image segmentation device, characterized in that, Comprising: A pneumonia CT image acquisition device, which is used to acquire pneumonia CT images; A processor, which is communicatively connected to the pneumonia CT image acquisition device and is used to execute a computer program to implement the steps of the pneumonia CT image segmentation method according to any one of claims 1 to 6, and obtain the lesion segmentation result of the pneumonia CT image; A display device, which is communicatively connected to the processor and is used to obtain and display the lesion segmentation result of the pneumonia CT image.

Images