A COVID-19 Patient Identification Method Based on Capsule Networks and Attention Mechanisms

Abstract

This paper proposes a COVID-19 patient detection method based on capsule networks and attention mechanisms. The main steps are as follows: First, the classic U-net network structure is trained to segment lung regions in CT scans. After successful training, this network is used for lung region segmentation. Next, a slice-level feature extraction network based on capsule networks is built to extract primary feature maps from patient CT slices. Based on the extracted feature maps, an attention mechanism is used to determine the criticality of the slices, enhancing attention to critical slices and suppressing attention to non-critical information to extract more effective final feature information. After obtaining the final feature information, dimensionality-reduced feature information is obtained by combining max pooling and average pooling sampling. Finally, a neural network structure is used to determine the category of the reduced feature information. Experiments were conducted on CT images of 305 patients. The results show that, compared with some state-of-the-art methods, the proposed method has high performance, achieving an accuracy of 96.3%.

Description

Technical Field

[0001] This invention relates to the field of medical image digital processing technology, specifically to a CT image recognition method based on capsule networks and attention mechanisms. Background Technology

[0002] The novel coronavirus COVID-19, caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), has become a global pandemic virus in the past two years.

[0003] Reverse transcription polymerase chain reaction (RTPCR) has become the fastest method for diagnosing COVID-19. However, its drawback is a high false negative rate, which is not conducive to the early diagnosis of COVID-19. In recent years, CT (computed tomography) images have played a crucial role in the diagnosis of COVID-19, as studies have reported cases where chest CT scans showed typical COVID-19 symptoms, but RTPCR results were negative. However, it is difficult to accurately extract COVID-19 from a large number of CT slices. Because many other lung diseases also have similar pathological features, the diagnosis of COVID-19 requires experienced specialists to spend a significant amount of time performing scans for confirmation. Therefore, there is an urgent need to find an effective, accurate, and rapid diagnostic tool.

[0004] Currently, AI-based models have achieved significant success in the field of medical image analysis, particularly in COVID-19 detection. Numerous studies indicate that automated diagnosis of COVID-19 from medical scan images holds great potential. Recently, Heidarian proposed a CT-caps network based on 3D CNNs, which extracts features from 3D CT scans using a capsule network and obtains the final feature map using max-pooling sampling. The feature map is then fed into a fully connected layer for case classification. However, several challenges remain in utilizing CT-caps networks for COVID-19 image diagnosis.

[0005] (1) Current capsule networks only isolate each slice. When a COVID-19 patient has multiple mild lesions scattered across different CT scans, the capsule network cannot identify the most critical and representative slices from all the CT scans, thus incorrectly extracting features from low-value slices. This will ultimately lead to patients with pathological changes being missed in COVID-19 testing.

[0006] (2) Current feature map sampling methods use max pooling, which focuses only on the most active local information and discards other information. For patients with multiple lesion areas, the activity information only represents a portion of the lesion area. Discarding a large amount of information leads to misclassification to some extent. Summary of the Invention

[0007] The purpose of this invention is to propose a method for identifying COVID-19 patients based on capsule networks and attention mechanisms, in order to solve the problem that existing convolutional neural networks have limited ability to identify patients with multiple lesion areas.

[0008] This invention provides a method for identifying COVID-19 patients based on capsule networks and attention mechanisms, the method comprising the following:

[0009] S1: Acquire CT images of the patient to be identified and segment the lung region using a pre-trained U-net model;

[0010] S2: Extract key lung feature information from the medical image to be extracted using a pre-trained feature extraction network based on capsule network and attention mechanism;

[0011] S3: Based on the identified key lung features, the feature information is combined and pooled to obtain a dimensionality-reduced feature map. The dimensionality-reduced feature map is then analyzed and identified to finally obtain the lung CT image recognition result.

[0012] Furthermore, step S1, which preprocesses the optical musical score image to be identified, includes the following specific steps:

[0013] Acquire the 3D image data of the medical image to be identified;

[0014] Based on a pre-trained U-net model, unimportant regions and afterimage regions in the image are removed, and then normalization and resizing are performed.

[0015] Furthermore, the specific content of step S2 is as follows:

[0016] The model for identifying COVID-19 patients based on capsule networks and attention mechanisms consists of a capsule network and an SE attention network;

[0017] The preprocessed CT medical images are first input into a capsule network, which is initialized with a stack consisting of four convolutional layers, a batch normalization layer, and a max-pooling layer. Then, the last convolutional layer provides information for subsequent capsule layers. This facilitates more efficient feature extraction by the subsequent capsule structure; specifically, it allows for the extraction of deeper and smaller feature maps through a routing protocol defined within the capsule structure. Specifically, the model chemically adds two capsule layers with identical structures, but using two layers allows for the extraction of more critical patient-level features. The features are then input into the SE attention network of this invention, which consists of a compression layer, an activation layer, and a weighted layer. The network learns the weight distribution coefficients of each feature map through the compression and activation layers, and then assigns weights to all slices through the weighted layer, strengthening the focus on key regions and suppressing the focus on non-key regions, further capturing key lung features.

[0018] Furthermore, the routing protocol in the capsule network described above is a novel forward propagation method, and its main steps are as follows:

[0019]

[0020] b ij =b ij +a ij

[0021]

[0022] In a capsule network, each capsule consists of a set of neurons, and a set of capsules forms a capsule layer. Together, they create the instantiation vector u of the associated instance. i The underlying capsule attempts to use a trainable weight matrix W. ij To predict the output of the high-level capsule, in the formula This is the predicted output of the bottom capsule i for the next capsule j. j Indicates the predicted output The correlation between a and the actual output of capsule j, where a ij b is the coefficient of consistency between the predicted and actual output. ij Let a be the coupling coefficient. ij The logarithmic prior, the actual contribution of capsule i to the output of the next capsule j is determined by the coupling coefficient c. ij Decisions j This represents the capsule output before the flattening function is applied.

[0023] Furthermore, the SE attention network described above consists of three parts: a compression layer, an activation layer, and a weighted layer. The network learns the weight distribution coefficients for each channel through the first two layers, and then assigns weights to all slices through the weighted layer. The purpose is to strengthen the focus on key areas and suppress the focus on non-key areas, thereby further capturing key lung features. The specific formula is as follows:

[0024]

[0025] s2=F ex (s1)=Sigmoid(Relu(s1W1+b1)W2+b2)

[0026] x c =F mul (model1 c s2 c )=s2 c *model1 C

[0027] Attention_m1=[x1,x2,...,x c ]

[0028] Where model1 (i，j) This represents the individual feature map extracted by the capsule network. W1 and b1 are parameters that can be obtained through training in the fully connected layer. model1 c This represents a two-dimensional feature map of size H×W, which in practice is the feature map of a single slice. Attention_m1 is the final feature map obtained after weighting by the SE attention mechanism.

[0029] Furthermore, step S3 specifically includes the following steps:

[0030] After obtaining the final feature map extracted by the feature extraction part of the pre-trained model based on capsule network and attention mechanism, combined pooling is used to perform dimensionality reduction, as shown in the following formula:

[0031] Maximum (H，W) =Max(Attention_m1) (H，W)

[0032]

[0033]

[0034] Last_feature map = F com (Max_m1, Mean_m1)=(Max_m1+Mean_m1) / 2

[0035] Attention_m1 in the first formula (H，W) This represents the values ​​of all slices at a location in the attention-weighted feature map, with a size of R. 1×n1 n1 represents the total number of slices, Maximum (H，W () represents the value at a location in the reduced-dimensional feature map. Attention_m1 (i，H，W) Average represents the value at a specific location in the feature map. (H，W) This represents the value at a location in the final reduced-dimensionality feature map.

[0036] Furthermore, the obtained dimensionality-reduced feature map is input into a fully connected classification network, and the final classification result can be obtained, as shown in the following formula:

[0037] predicate=Sigmoid(Relu(last_feature map×W1+b1)W2+b2)W3+b3

[0038] In the formula, W and b are parameters that can be obtained through training in the fully connected layer, used for parameter compression calculation. The formula ultimately yields the prediction result.

[0039] This invention proposes a COVID-19 identification network based on capsule networks and attention mechanisms, abbreviated as ACSN, ​​to rapidly, accurately, and interpretably screen for COVID-19. First, we utilize a proposed key slice enhancement method to improve the network's ability to discover key slices in large-scale image processing. Then, we employ a key pooling sampling method to obtain as much key information as possible. ACSN can effectively classify COVID-19 from CT images, addressing the problem of capsule networks' poor ability to identify patients with multiple lesion regions. Attached Figure Description

[0040] To more clearly illustrate the technical solution in this specification, the accompanying drawings used in the technical solution will be briefly introduced below.

[0041] Figure 1 This is a schematic diagram of a COVID-19 identification network based on capsule networks and attention mechanisms provided by the present invention.

[0042] Figure 2 The image obtained from lung region segmentation provided by the present invention.

[0043] Figure 3 This is a schematic diagram of a COVID-19 identification network based on capsule networks and attention mechanisms provided by the present invention.

[0044] Figure 4 This is a schematic diagram of the capsule network structure provided by the present invention.

[0045] Figure 5 This is a schematic diagram of the SE attention mechanism provided by the present invention.

[0046] Figure 6 This is a schematic diagram of a combined pooling sampling and fully connected layer classification network provided by the present invention. Detailed Implementation

[0047] like Figure 1 The diagram shows the framework of the COVID-19 recognition network based on capsule network and attention mechanism proposed in this invention. By capturing rich features of medical images through capsule network and attention mechanism, and then using combined pooling to obtain global and local information of weighted feature maps, the accuracy of COVID-19 medical image recognition is higher and the model training time is reduced.

[0048] The present invention proposes a COVID-19 identification method based on capsule networks and attention mechanisms, which specifically includes the following steps:

[0049] Step 1: Acquire the CT image of the patient to be identified and perform lung region segmentation using a pre-trained U-net model; This invention first reads the CT medical image to be identified. More limited image information is obtained by using a pre-trained U-net network. Specifically, this involves removing unimportant and afterimage regions from the image, followed by standardization and resizing. Finally, slices of invisible lung tissue are removed. Figure 2 As shown, (a) represents the input image, and (b) represents the lung region image obtained after segmentation. To reduce complexity and memory requirements, information loss is negligible.

[0050] Step 2: Extract lesion region features from the CT medical images to be identified using a pre-trained COVID-19 identification method based on capsule networks and attention mechanisms;

[0051] like Figure 3 As shown, the COVID-19 identification method based on capsule networks and attention mechanisms consists of a capsule network, an SE attention layer, and a combined pooling sampling layer.

[0052] The preprocessed CT medical images are first input into a capsule network, which is initialized with a stack consisting of four convolutional layers, a batch normalization layer, and a max-pooling layer. Then, the last convolutional layer provides information for subsequent capsule layers. This allows the subsequent capsule structure to extract features more efficiently; specifically, the routing protocol defined within the capsule structure enables the extraction of deeper and smaller feature maps. The specific operation involves adding two capsule layers to the model. These two capsule layers have the same structure, but using two layers allows for the extraction of more critical patient-level features, ultimately resulting in a lung feature map. The specific structure is as follows... Figure 4 .

[0053] Furthermore, the routing protocol in the capsule network described above is a novel forward propagation method, and its main steps are as follows:

[0054]

[0055] b ij =b ij +a ij

[0056]

[0057]

[0058] In a capsule network, each capsule consists of a set of neurons, and a set of capsules forms a capsule layer. Together, they create the instantiation vector u of the associated instance. i The underlying capsule attempts to use a trainable weight matrix W. ij To predict the output of the high-level capsule, in the formula This is the predicted output of the bottom capsule i for the next capsule j. j Indicates the predicted output The correlation between a and the actual output of capsule j, where a ij b is the coefficient of consistency between the predicted and actual output. ij Let a be the coupling coefficient. ij The logarithmic prior, the actual contribution of capsule i to the output of the next capsule j is determined by the coupling coefficient c. ij Decisions j This represents the capsule output before the flattening function is applied.

[0059] Furthermore, after obtaining the lung feature map, it is input into the SE attention network, which consists of three parts: a compression layer, an activation layer, and a weighted layer. The specific structure is as follows: Figure 5As shown, the network learns the weight distribution coefficients for each channel through the first two layers, and then assigns weights to all slices through a weighted layer. The purpose is to strengthen the focus on key areas and suppress the focus on non-key areas, thereby further capturing key lung features. The specific formula is as follows:

[0060]

[0061] s2=F ex (s1)=Sigmoid(Relu(s1W1+b1)W2+b2)

[0062] x c =F mul (model1 c s2 c )=s2 c *model1 C

[0063] Attention_m1=[x1,x2,......x c ]

[0064] Where model1 (i，j) This represents the individual feature map extracted by the capsule network. W1 and b1 are parameters that can be obtained through training in the fully connected layer. model1 c This represents a two-dimensional feature map of size H×W, which in practice is the feature map of a single slice. Attention_m1 is the final feature map obtained after weighting by the SE attention mechanism.

[0065] Furthermore, step S3 specifically includes the following steps:

[0066] After obtaining the final feature map extracted by the feature extraction part of the pre-trained model based on capsule network and attention mechanism, combined pooling is used for dimensionality reduction, as shown in the specific structure. Figure 6 As shown, the formula is as follows:

[0067] Maximum (H，W) =Max(Attention_m1) (H，W) )

[0068]

[0069] last_feature map = F com (Max_m1, Mean_m1)=(Max_m1+Mean_m1) / 2

[0070] Attention_m1 in the first formula (H，W)This represents the values ​​of all slices at a location in the attention-weighted feature map, with a size of R. 1×n1 n1 represents the total number of slices, Maximum (H，W) This represents the value at a location in the reduced-dimensional feature map. Attention_m1 (i，H，W) Average represents the value at a specific location in the feature map. (H，W) This represents the value at a location in the final reduced-dimensionality feature map.

[0071] Furthermore, the obtained dimensionality-reduced feature map is input into a fully connected classification network, and the final classification result can be obtained, as shown in the following formula:

[0072] predicate=Sigmoid(Relu(last_feature map×W1+b1)W2+b2)W3+b3

[0073] In the formula, W and b are parameters that can be obtained through training in the fully connected layer, used for parameter compression calculation. The formula ultimately yields the prediction result.

[0074] Finally, the method proposed in this invention was benchmarked on a public dataset, and the test results showed that the method has a high accuracy rate in identifying COVID-19 patients.

[0075] Dataset: The dataset used in this paper is from 2021 and is called COVID-19-CT-MD. It includes 169 COVID-19 infected individuals, 60 CAP patients, and 76 healthy individuals. In COVID-19-CT-MD, an experienced radiologist analyzed 25 CAP cases and 55 COVID-19 patients to identify and label slides with significant infection. More specifically, patient-level labeling was performed by three radiologists, and a vote was used to determine the final results. However, for slide-level labeling, due to time and complexity limitations, only one radiologist provided the slide-level labeling. The labeling process aimed to identify slides with significant COVID-19 lesions. The labeled subset contains 4993 slides showing infection and 18416 slides showing no signs of infection. 90% of the training set was used for model training, and the remaining 10% was used to optimize parameters and validate model performance.

[0076] Parameter Settings and Training: The method proposed in this invention is based on the Tensorflow framework and implemented in Python. Computation was performed on an NVIDIA GeForce RTX 2060, CUDA 11.3, and 24GB of memory. Each sample image was preprocessed and scaled to a height of 256 pixels before being input into the model for recognition. To ensure the objective function converges to its optimal value at an appropriate time, the batch size was set to 16 and the learning rate to 0.0001. The Adam optimizer was used, which is computationally efficient, requires little memory, and its parameter updates are unaffected by gradient scaling, typically requiring no adjustment or only minimal fine-tuning. A stopping strategy was implemented, stopping training when the model's performance did not decrease within 100 epochs. Results show that the above parameter settings can optimize the recognition model and effectively improve the recognition accuracy of note sequences.

[0077] Evaluation Metrics: This invention employs three commonly used evaluation metrics in medical image recognition: accuracy, sensitivity, and specificity. Accuracy refers to the proportion of patients correctly identified out of the total number of patients. Sensitivity compares accurately identified COVID-19 patients with all COVID-19 patients, and specificity compares accurately identified healthy patients with all healthy patients. The specific formulas are as follows:

[0078]

[0079] The characteristics in the above formulas have different meanings: True Positive (TP): Predicted to be COVID-19, but the patient actually has COVID-19. True Negative (TN): Predicted not to be COVID-19, but the patient is actually healthy. False Positive (FP): Predicted to be COVID-19, but the patient is actually healthy. False Negative (FN): Predicted to be healthy, but actually has COVID-19.

[0080] The COVID-19 identification method based on capsule networks and attention mechanisms proposed in this invention was applied to the COVID-19-CT-MD dataset, using CT medical images as input and patient classification results as output. To further evaluate the effectiveness of the method, the following experiments were designed: (1) a COVID-19 identification method based on the ResNet network structure (referred to as Resne); (2) a COVID-19 identification method based on capsule networks (referred to as CT-Caps); (3) a COVID-19 identification method based on fully automated capsule networks (referred to as Covid-Fact); and (4) the COVID-19 identification method based on capsule networks and attention mechanisms proposed in this invention. Table 1 lists the evaluation results of the above four methods on the test set.

[0081] Table 1: Comparison of experimental results for different models in the same database

[0082]

[0083] As can be seen from Table 1, the method proposed in this invention can achieve higher accuracy, higher specificity, and higher sensitivity in the identification of COVID-19 patients.

Claims

1. A method for identifying COVID-19 patients based on capsule network and attention mechanism, characterized in that, The method comprises the following steps: S1: obtaining a CT image of a patient to be identified and segmenting a lung region using a pre-trained U-net model; The step S1 of pre-processing the music score image to be identified specifically comprises the following steps: S1-1: obtaining three-dimensional image data of the medical image to be identified; S1-2: removing unimportant regions and residual shadow regions in the image according to a pre-trained U-net model, and then performing standardization and size adjustment; S2: extracting key lung feature information in the medical image to be extracted through a feature extraction network of a pre-trained model based on a capsule network and an attention mechanism; The step S2 comprises the following steps: S2-1: the model for identifying COVID-19 patients based on a capsule network and an attention mechanism is composed of a capsule network and an SE attention network; S2-2: the pre-processed CT medical image is first input into the capsule network, which is initialized by a stack composed of 4 convolutional layers, a batch normalization layer and a maximum pooling layer; Then, the last convolutional layer is used to provide information for the subsequent capsule layer; in this way, the subsequent capsule structure can more efficiently extract features, specifically by using the routing protocol defined in the capsule structure to extract deeper and smaller feature maps; specifically, two layers of capsule layers are added to the model, and the two capsule layers have the same structure, but using two layers can extract more critical patient-level features, and finally obtain a lung feature map; S2-3: further, after obtaining the lung feature map, it is input into the SE attention network, which comprises three parts, namely a compression layer, an excitation layer and a weighting layer, the network learns the weight distribution coefficient of each channel through the first two layers, and then weights all slices through the weighting layer, the purpose is to strengthen the attention of key areas and suppress the attention of non-key areas, and further capture key lung features; S3: according to the identified key lung feature information, the feature information is combined and pooled to obtain a reduced dimension feature map, and the reduced dimension feature map is analyzed and identified, and finally the result of lung CT image recognition is obtained; The step S3 comprises the following steps: S3-1: after obtaining the final feature map extracted by the feature extraction part of the pre-trained model based on the capsule network and the attention mechanism, the feature map is reduced in dimension by using combined pooling; S3-2: the reduced dimension feature map is input into a fully connected classification network, and finally the specific classification result can be obtained.

2. The method for identifying COVID-19 patients based on capsule network and attention mechanism as claimed in claim 1, wherein, The forward propagation method of the capsule network structure in the step S2-2, a routing protocol, can be respectively expressed by a formula as follows: b ij = b ij + a ij Each capsule in a capsule network consists of a set of neurons, and a set of capsules forms a capsule layer, which collectively creates an instantiation vector u for an associated instance i The lower-level capsules try to predict the output of the higher-level capsules using a trainable weight matrix W ij where is the predicted output of a lower-level capsule i for a next-level capsule j; v j represents the agreement between the predicted output and the actual output of capsule j, where a ij is the coefficient of agreement between the predicted and actual outputs, and b ij is the log prior of a ij , the actual contribution of capsule i to the output of the next-level capsule j is determined by the coupling coefficient c ij s j denotes the capsule output before the squashing function is applied; The method for identifying COVID-19 patients based on a capsule network and an attention mechanism according to claim 1, wherein the SE attention structure in step S2-3 is represented by the following formula: s2 = F ox (s1) = Sigmoid(Relu(s1W1 + b1)W2 + b2) x c = F mul (model1 c , s2 c ) = s2 c *model1 C Attention_m1 = [x1, x2,.......x c ] where model1 (i，j) represents the individual feature map extracted by the capsule network, W1 and b1 are parameters that can be obtained by training in the fully connected layer, model1 c represents a two-dimensional feature map with a size of HxW, and the actual meaning is the feature map of a single slice. Attention_m1 is the final feature map obtained after weighting by the sE attention mechanism.

3. The method for identifying COVID-19 patients based on capsule network and attention mechanism as claimed in claim 1, wherein, The formula of the combined pooling sampling method in step s3-1 is: Maximum (H，W) = Max(Attention_m1 (H，W) ) Last_feature map=F com (Max_m1, Mean_m1)=(Max_m1+Mean_m1) / 2 Attention_m1 in the first equation (H，W) representing the value of all slices of a position in the feature map weighted by the attention mechanism, size R 1×n1 , n1 represents the total number of slices, Maximum (H，W) representing the value of a position in the reduced dimension feature map; Attention_m1 (i，H，W) representing the value of a specific position in the feature map, Average (H，W) representing the value of a position in the finally obtained reduced dimension feature map.

4. The method for identifying COVID-19 patients based on capsule network and attention mechanism as claimed in claim 1, wherein, The formula of the fully connected neural network in step S3-2 is: predicate = Sigmoid(Relu(Relu(last_feature map × W1 + b1) W2 + b2) W3 + b3 In the formula, w and b are parameters that can be obtained through training in the full connection layer, which are used for parameter compression calculation; the final formula can obtain the prediction result.

Images