Pulmonary nodule interpretable joint classification and segmentation system based on class activation mapping

Abstract

The invention discloses a pulmonary nodule interpretable joint classification and segmentation system based on class activation mapping, and the system comprises a data importing module which is used for loading and preprocessing pulmonary nodule CT data; the pulmonary nodule classification module is used for outputting a classification result of the semantic features of the pulmonary nodule, providing interpretability for a classification result of the malignancy degree of the pulmonary nodule from the perspective of model reasoning, generating a class activation graph of each semantic feature based on a class activation mapping method, and providing interpretability for the classification result of the semantic features of the pulmonary nodule from the perspective of visualization; and the pulmonary nodule segmentation module is used for generating anatomical attention by using the class activation diagram output by the pulmonary nodule classification module and outputting a pulmonary nodule three-dimensional segmentation result. The low-level semantic feature classification task and the pulmonary nodule malignancy degree classification task are introduced to construct the pulmonary nodule classification module, the semantic feature classification information is utilized to guide the pulmonary nodule segmentation module, multi-aspect analysis results of classification and segmentation can be output for the pulmonary nodule, and the method has high interpretability and reliability.

Description

Technical Field

[0001] The present invention relates to the technical field of computer-aided medicine, and in particular to an interpretable joint classification and segmentation system for pulmonary nodules based on class activation mapping. Background Art

[0002] Early screening of lung cancer is crucial for reducing its incidence and mortality. One of the early lesions of lung cancer usually presents as pulmonary nodules. Analyzing the CT images of pulmonary nodules is an important means for early screening of lung cancer. However, manual reading, analysis, discrimination, and annotation are time-consuming and laborious. Artificial intelligence technology plays a key role in improving the quality and efficiency of medical services, reducing misdiagnosis and missed diagnosis. Therefore, developing an automated computer-aided diagnosis system has become a key development direction for lung cancer automatic screening technology. In medical diagnosis, doctors often judge the malignancy degree of pulmonary nodules based on the morphological characteristics of pulmonary nodules on CT images of pulmonary nodules. However, existing auxiliary diagnosis models often ignore the anatomical characteristics of some semantic features of pulmonary nodules, resulting in a loss of segmentation accuracy, and failing to provide doctors with interpretive information for diagnosis, reducing the credibility of the model output results and restricting the practical application of the model in clinics. Summary of the Invention

[0003] The purpose of the present invention is to overcome the problems of the loss of anatomical features of some semantic features of pulmonary nodules during multiple downsampling in the prior art, resulting in a decrease in segmentation accuracy, and the poor credibility caused by the black box nature of the deep learning model. The present invention provides an interpretable joint classification and segmentation system for pulmonary nodules based on class activation mapping, adopts a multi-task strategy, introduces a low-level semantic feature classification task and a pulmonary nodule malignancy degree classification task, provides interpretable results for the pulmonary nodule malignancy degree classification task in the form of a class activation map, and uses the class activation map to assist the pulmonary nodule segmentation task to improve the segmentation accuracy of pulmonary nodules.

[0004] To achieve the above purpose, the technical solution provided by the present invention is: an interpretable joint classification and segmentation system for pulmonary nodules based on class activation mapping, including:

[0005] A data import module, configured to load pulmonary nodule CT data according to an annotation file, preprocess the pulmonary nodule CT data, and output images, masks, and binary labels of semantic features of a unified size; wherein, the pulmonary nodule CT data includes CT images, masks, and grade labels of semantic features of pulmonary nodules, the semantic features include low-level semantic features and high-level semantic features, there are four low-level semantic features, namely lobulation sign, spiculation sign, margin, and texture, and the high-level semantic feature is malignancy degree;

[0006] The pulmonary nodule classification module is used to extract features from the CT images of pulmonary nodules through convolution and downsampling operations, output the results of semantic feature classification of pulmonary nodules, provide interpretability for the results of pulmonary nodule malignancy classification from the perspective of model inference, generate class activation maps for each semantic feature based on the class activation mapping method, provide interpretability for the results of pulmonary nodule semantic feature classification from the perspective of visualization, and use the class activation maps of each semantic feature to output the final class activation map;

[0007] The pulmonary nodule segmentation module is used to extract features from the CT images of pulmonary nodules and the final class activation map output by the pulmonary nodule classification module through convolution and downsampling operations respectively, use the feature map obtained by class activation map feature extraction to provide anatomical attention guidance for the feature map obtained by CT image feature extraction of pulmonary nodules, and output the three-dimensional segmentation result of pulmonary nodules.

[0008] Preferably, the data import module includes a data loading module and a data preprocessing module, where:

[0009] The data loading module loads the CT data of pulmonary nodules according to the annotation file, and the CT data of pulmonary nodules includes the CT images of pulmonary nodules, masks and grade labels of semantic features;

[0010] The data preprocessing module is used to resample, window and normalize the CT images and masks of the CT data of pulmonary nodules, output the CT images and masks of pulmonary nodules with a fixed size, and perform label mapping on the grade labels of semantic features to output binary labels of semantic features.

[0011] Preferably, the pulmonary nodule classification module includes an image feature extraction module, a semantic feature classification module and a class activation mapping module. The pulmonary nodule classification module is trained using the CT images of pulmonary nodules and the binary labels of semantic features output by the data import module, where:

[0012] The image feature extraction module adopts the encoder part of ResUNet and uses convolution and downsampling operations to extract features from the CT images of pulmonary nodules to generate feature maps;

[0013] The semantic feature classification module introduces a low-level semantic feature classification task and a pulmonary nodule malignancy classification task. After the feature map output by the last convolutional layer of the image feature extraction module undergoes global average pooling operation, it passes through two fully connected layers in sequence. The first fully connected layer predicts four low-level semantic features of the pulmonary nodule, and the second fully connected layer predicts the malignancy of the pulmonary nodule. Each semantic feature corresponds to two categories, so the number of neurons in the first layer is 8, and the number of neurons in the second layer is 2. The semantic feature classification module outputs the binary classification result of each semantic feature according to the scores of the two neurons corresponding to each semantic feature, and at the same time outputs the score of the neuron with the higher score of each semantic feature. The low-level semantic feature classification loss function and the malignancy classification loss function of the pulmonary nodule both adopt the cross-entropy loss CE Loss, and the total loss function of the classification task is:

[0014]

[0015] In the formula, is the malignancy classification loss function, is the lobulation sign classification loss function, is the spiculation sign classification loss function, is the edge classification loss function, is the texture classification loss function;

[0016] The class activation mapping module uses the output of the semantic feature classification module, that is, the score of the specific category c of the i-th semantic feature to calculate the gradient with respect to the feature map A j of the j-th convolutional layer of the image feature extraction module. Then for the k-th channel j of the feature map A its weight is:

[0017]

[0018] For the specific category c of the i-th semantic feature, the element-wise product with the feature map A j of the j-th convolutional layer is its corresponding class activation map

[0019]

[0020] In the formula, represents the Hadamard product;

[0021] The class activation mapping module obtains the final class activation map M by using the class activation maps of each semantic feature. For the spatial element with spatial coordinates (x, y, z) in M, its value M(x, y, z) is the maximum value of the class activation maps of each semantic feature at the corresponding position:

[0022]

[0023] In the formula, the total number n = 5, corresponding to five semantic features: lobulation sign, spiculation sign, margin, texture, and malignancy degree; for the class activation map corresponding to the specific category c of the i-th semantic feature is the value of the spatial element with spatial coordinates (x, y, z).

[0024] Preferably, the pulmonary nodule segmentation module uses ResUNet as the backbone network, the loss function of the segmentation task is the Dice loss, and the encoder part of the pulmonary nodule segmentation module has two parallel branches, namely the pulmonary nodule CT image feature extraction branch and the class activation map feature extraction branch; the pulmonary nodule segmentation module is trained using the CT image, mask of the pulmonary nodule output by the data import module, and the final class activation map output by the pulmonary nodule classification module. The anatomical attention module uses the class activation map to provide anatomical attention guidance for the feature extraction of the pulmonary nodule CT image, where:

[0025] Both the pulmonary nodule CT image feature extraction branch and the class activation map feature extraction branch adopt the encoder part of ResUNet. The CT image of the pulmonary nodule is input into the pulmonary nodule CT image feature extraction branch, and the corresponding class activation map output by the pulmonary nodule classification module is input into the class activation map feature extraction branch. Convolution and downsampling operations are used to extract features from the input image to generate feature maps;

[0026] The anatomical attention module takes the e-th layer feature map A e of the pulmonary nodule CT image feature extraction branch and the e-th layer feature map M e of the class activation map feature extraction branch as inputs. After the feature maps of the two branches are concatenated, two parallel 1×1×1 convolutions are performed to obtain the weight tensor of the feature map of the corresponding pulmonary nodule CT image and the weight tensor of the feature map of the corresponding class activation map

[0027]

[0028] In the formula, A e and M e are concatenated in the channel direction to obtain [A e , M e , and They are the parameters of two convolutional layers respectively, * represents the convolution operation, and σ is the Sigmoid activation function;

[0029] The two sets of obtained weight tensors are multiplied element-wise with the two sets of input feature maps respectively. Finally, the weighted feature maps of the CT images of the pulmonary nodules and the weighted feature maps of the class activation maps are added element-wise to output the weighted sum feature map F e :

[0030]

[0031] In the formula, · is the element-wise multiplication.

[0032] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0033] 1. The semantic feature classification module introduces the low-level semantic feature classification task and the pulmonary nodule malignancy classification task, providing an interpretation for the pulmonary nodule malignancy classification result from the perspective of model inference.

[0034] 2. The class activation mapping module generates class activation maps for each semantic feature with respect to the shallow feature maps, reflecting the regions of interest of the model, providing an interpretation for the pulmonary nodule semantic feature classification result from the perspective of result visualization, and at the same time providing auxiliary information for the pulmonary nodule segmentation module.

[0035] 3. The anatomical attention module uses the class activation maps of multiple semantic features to provide anatomical attention for the pulmonary nodule segmentation task, solving the problem that the anatomical features of some semantic features are lost during the downsampling process in the current pulmonary nodule segmentation research work, resulting in a decrease in segmentation accuracy.

[0036] 4. The system of the present invention completes the joint classification and segmentation tasks, and simultaneously outputs the classification results of the lobulation sign, spiculation sign, edge, texture, malignancy degree of the pulmonary nodules, as well as the three-dimensional segmentation result of the pulmonary nodules, giving a comprehensive and accurate description of the pulmonary nodules, and having high clinical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a schematic diagram of the relationship between the modules of the system of the present invention.

[0038] Figure 2 It is a schematic diagram of the structure of the pulmonary nodule classification module.

[0039] Figure 3 It is a schematic diagram of the structure of the class activation mapping module.

[0040] Figure 4Schematic diagram of the structure of the pulmonary nodule segmentation module; in the figure, A1, A2, and A3 respectively correspond to the feature maps of the first, second, and third convolutional layers of the pulmonary nodule CT image feature extraction branch, and M1, M2, and M3 respectively correspond to the feature maps of the first, second, and third convolutional layers of the class activation map feature extraction branch.

[0041] Figure 5 Schematic diagram of the structure of the anatomical attention module. Specific implementation manner

[0042] The present invention will be further described in detail below in conjunction with the embodiments and the accompanying drawings, but the implementation manners of the present invention are not limited thereto.

[0043] This embodiment discloses a pulmonary nodule interpretability joint classification and segmentation system based on class activation mapping, which is a system developed in Python language and can run on Windows devices. The relationship between the modules of the system is as Figure 1 shown. It includes:

[0044] A data import module, which is used to load pulmonary nodule CT data according to the annotation file, preprocess the pulmonary nodule CT data, and output images, masks, and binary labels of semantic features of a unified size; wherein, the pulmonary nodule CT data includes the CT image, mask, and grade label of semantic features of the pulmonary nodule, and the semantic features include low-level semantic features and high-level semantic features. There are four low-level semantic features, namely lobulation sign, spiculation sign, margin, and texture, and the high-level semantic feature is malignancy degree;

[0045] A pulmonary nodule classification module, which is used to extract features from the CT image of the pulmonary nodule through convolution and downsampling operations, output the result of the pulmonary nodule semantic feature classification, provide interpretability for the result of the pulmonary nodule malignancy degree classification from the perspective of model inference, generate class activation maps of each semantic feature based on the class activation mapping method, provide interpretability for the result of the pulmonary nodule semantic feature classification from the perspective of visualization, and output the final class activation map by using the class activation maps of each semantic feature;

[0046] A pulmonary nodule segmentation module, which is used to extract features from the CT image of the pulmonary nodule and the final class activation map output by the pulmonary nodule classification module through convolution and downsampling operations respectively, use the feature maps obtained by class activation map feature extraction to provide anatomical attention guidance for the feature maps obtained by pulmonary nodule CT image feature extraction, and output the three-dimensional segmentation result of the pulmonary nodule.

[0047] Specifically, the data import module includes a data loading module and a data preprocessing module, wherein:

[0048] The data loading module loads the pulmonary nodule CT data according to the annotation file, and the pulmonary nodule CT data includes the CT image, mask, and grade label of semantic features of the pulmonary nodule;

[0049] The data preprocessing module is used to resample, window, and normalize the CT images and masks of the pulmonary nodule CT data, and perform label mapping on the grade labels of the semantic features to output the binary labels of the semantic features. For the 1018 CT images in the LIDC-IDRI pulmonary nodule dataset, after excluding the CT images with a slice thickness greater than 2.5 mm, the slice spacing of all CT images is resampled to 1×1×1, and the CT value is limited to -1000~600 HU according to the CT characteristics of the lung tissue, and the CT value is normalized to 0~1. Subsequently, pulmonary nodules marked by at least three doctors are selected, the center of the pulmonary nodule is calculated according to the marked pulmonary nodule area by the doctors, the CT cut data of the pulmonary nodule with a size of 32×32×32 is extracted, and a pulmonary nodule mask is generated. The average values of the lobulation sign, spiculation sign, margin, texture, and malignancy grade labels marked by each doctor are taken. If the average grade is 3, it is considered that the marking is ambiguous and the pulmonary nodule is discarded. If the average grade is less than 3, the corresponding semantic feature label is set to 0, otherwise it is set to 1. For the lobulation sign, a label of 0 indicates no lobulation sign, and a label of 1 indicates an obvious lobulation sign; for the spiculation sign, a label of 0 indicates no spiculation sign, and a label of 1 indicates an obvious spiculation sign; for the margin feature, a label of 0 indicates that the margin is very unclear, and a label of 1 indicates that the margin is very clear; for the texture feature, a label of 0 indicates the presence of non-solid texture, and a label of 1 indicates the presence of solid texture; for the malignancy degree feature, a label of 0 indicates a benign pulmonary nodule, and a label of 1 indicates a malignant pulmonary nodule.

[0050] Specifically, as Figure 2 shown, the pulmonary nodule classification module includes an image feature extraction module, a semantic feature classification module, and a class activation mapping module. The pulmonary nodule classification module uses the CT images of the pulmonary nodules and the binary labels of the semantic features output by the data import module for training, where:

[0051] The image feature extraction module adopts the encoder part of ResUNet, and uses convolution and downsampling operations to extract features from the CT images of the pulmonary nodules to generate feature maps;

[0052] The semantic feature classification module introduces a low-level semantic feature classification task and a pulmonary nodule malignancy classification task. After the feature map output by the last convolutional layer of the image feature extraction module undergoes global average pooling operation, it passes through two fully connected layers in sequence. The first fully connected layer predicts four low-level semantic features of the pulmonary nodule, and the second fully connected layer predicts the malignancy of the pulmonary nodule. Each semantic feature corresponds to two categories, so the number of neurons in the first layer is 8, and the number of neurons in the second layer is 2. The semantic feature classification module outputs the binary classification result of each semantic feature according to the scores of the two neurons corresponding to each semantic feature, and at the same time outputs the score of the neuron with the higher score of each semantic feature. The low-level semantic feature classification loss function and the malignancy classification loss function of the pulmonary nodule both adopt the cross-entropy loss CE Loss, and the total loss function of the classification task is:

[0053]

[0054] In the formula, is the malignancy classification loss function, is the lobulation sign classification loss function, is the spiculation sign classification loss function, is the edge classification loss function, is the texture classification loss function;

[0055] The class activation mapping module uses the output of the semantic feature classification module, that is, the score of the specific category c of the i-th semantic feature to calculate the gradient with respect to the j-th convolutional layer feature map A j of the image feature extraction module. Then, for the k-th channel j of the feature map A its weight is:

[0056]

[0057] For the specific category c of the i-th semantic feature, the element-wise product with the j-th convolutional layer feature map A j is its corresponding class activation map

[0058]

[0059] In the formula, represents the Hadamard product;

[0060] The class activation mapping module obtains the final class activation map M using the class activation maps of multiple semantic features. For the spatial element with spatial coordinates (x, y, z) in M, its value M(x, y, z) is the maximum value of the class activation maps of each semantic feature at the corresponding position:

[0061]

[0062] In the formula, the total number n = 5, corresponding to five semantic features: lobulation sign, spiculation sign, margin, texture, and malignancy degree; for the class activation map corresponding to the specific category c of the i-th semantic feature is the value of the spatial element with spatial coordinates (x, y, z).

[0063] Specifically, as Figure 4 shown, the pulmonary nodule segmentation module uses ResUNet as the backbone network. The loss function of the segmentation task is the Dice loss. The encoder part of the pulmonary nodule segmentation module has two parallel branches, namely the pulmonary nodule CT image feature extraction branch and the class activation map feature extraction branch. The pulmonary nodule segmentation module is trained using the CT image, mask of the pulmonary nodule output by the data import module, and the final class activation map output by the pulmonary nodule classification module. The anatomical attention module uses the class activation map to provide anatomical attention guidance for the feature extraction of the pulmonary nodule CT image, where:

[0064] Both the pulmonary nodule CT image feature extraction branch and the class activation map feature extraction branch adopt the encoder part of ResUNet. The CT image of the pulmonary nodule is input into the pulmonary nodule CT image feature extraction branch, and the corresponding class activation map output by the pulmonary nodule classification module is input into the class activation map feature extraction branch. Convolution and downsampling operations are used to extract features from the input image to generate feature maps;

[0065] As Figure 5 shown, the anatomical attention module takes the e-th layer feature map A e of the pulmonary nodule CT image feature extraction branch and the e-th layer feature map M e of the class activation map feature extraction branch as inputs. After the feature maps of the two branches are concatenated, two parallel 1×1×1 convolutions are performed to obtain the weight tensor of the feature map of the corresponding pulmonary nodule CT image

[0066]

[0067] In the formula, A e and M e are concatenated in the channel direction to obtain [A e , M e , and They are the parameters of two convolutional layers respectively. * represents the convolution operation, and σ is the Sigmoid activation function.

[0068] The two obtained weight tensors are multiplied element-wise with the two groups of input feature maps respectively. Finally, the weighted feature map of the CT image of the pulmonary nodule and the weighted feature map of the class activation map are added element-wise to output the weighted sum feature map F e :

[0069]

[0070] In the formula, · is the element-wise multiplication.

[0071] As described above, it is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of changes or substitutions, which should all be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claimed rights.

Claims

1. An interpretable joint classification and segmentation system for pulmonary nodules based on class activation mapping, characterized by: include: The data import module is used to load lung nodule CT data according to the annotation file, preprocess the lung nodule CT data, and output a uniform-sized image, mask, and binary labels of semantic features; wherein the semantic features include low-level semantic features and high-level semantic features, and there are four low-level semantic features, namely, lobulation sign, burr sign, edge, and texture, and the high-level semantic feature is the degree of malignancy; The pulmonary nodule classification module is used to extract features from CT images of pulmonary nodules through convolution and downsampling operations, output the results of pulmonary nodule semantic feature classification, so as to provide interpretability for the results of pulmonary nodule malignancy classification from the perspective of model reasoning, generate class activation maps of various semantic features based on the class activation mapping method, so as to provide interpretability for the results of pulmonary nodule semantic feature classification from the perspective of visualization, and output the final class activation map using the class activation maps of various semantic features; The pulmonary nodule segmentation module is used to extract features from the CT image of the pulmonary nodules and the final class activation map output by the pulmonary nodule classification module through convolution and downsampling operations, and use the feature map obtained by the class activation map feature extraction to provide anatomical attention guidance for the feature map obtained by the CT image feature extraction of the pulmonary nodules, and output the three-dimensional segmentation result of the pulmonary nodule.

2. The class activation mapping-based interpretable joint classification and segmentation system for pulmonary nodules according to claim 1, characterized in that: The data import module includes a data loading module and a data preprocessing module, wherein: The data loading module loads the pulmonary nodule CT data according to the annotation file, where the pulmonary nodule CT data includes the CT image of the pulmonary nodule, the mask and the level label of the semantic feature; The data preprocessing module is used to resample, adjust windows and normalize the CT image and mask of the lung nodule CT data, output a fixed-size CT image and mask of the lung nodule, and perform label mapping on the level labels of the semantic features to output the binary labels of the semantic features.

3. The class activation mapping-based interpretable joint classification and segmentation system for pulmonary nodules according to claim 2, characterized in that: The pulmonary nodule classification module includes an image feature extraction module, a semantic feature classification module and a class activation mapping module. The pulmonary nodule classification module is trained using the CT image of the pulmonary nodules and the binary labels of the semantic features output by the data import module, wherein: The image feature extraction module uses the encoder part of ResUNet to extract features from the CT image of the lung nodules using convolution and downsampling operations to generate a feature map; The semantic feature classification module introduces low-level semantic feature classification tasks and lung nodule malignancy classification tasks. After the feature map output by the last convolution layer of the image feature extraction module is subjected to a global average pooling operation, it passes through two fully connected layers in sequence. The first fully connected layer predicts four low-level semantic features of lung nodules, and the second fully connected layer predicts the malignancy of lung nodules. Each semantic feature corresponds to two categories, so the number of neurons in the first layer is 8, and the number of neurons in the second layer is 2; the semantic feature classification module outputs the binary classification result of each semantic feature according to the scores of the two neurons corresponding to each semantic feature, and outputs the score of the neuron with a higher score for each semantic feature at the same time; the low-level semantic feature classification loss function and the malignancy classification loss function of lung nodules both use cross entropy loss CE Loss, and the total loss function of the classification task for: In the formula, is the malignancy classification loss function, is the leaf feature classification loss function, is the burr classification loss function, is the marginal classification loss function, is the texture classification loss function; The class activation mapping module uses the output of the semantic feature classification module, i.e., the score of the specific category c of the i-th semantic feature calculate The feature map A of the jth convolutional layer of the image feature extraction module j The gradient of , then for the feature map A j The kth channel Its weight for: For a specific category c of the i-th semantic feature, and the feature map A of the jth convolutional layer j The element-by-element product of is the corresponding class activation map In the formula, represents the Hadamard product; The class activation mapping module uses the class activation maps of each semantic feature to obtain the final class activation map M. For the spatial element with spatial coordinates (x, y, z) in M, its value M(x, y, z) is the maximum value of the class activation map of each semantic feature in the corresponding position: Where, the total number n = 5, corresponding to the five semantic features of lobulation, burr, edge, texture, and malignancy; the class activation map corresponding to the specific category c of the i-th semantic feature is is the value of the spatial element whose spatial coordinates are (x,y,z).

4. The class activation mapping-based interpretable joint classification and segmentation system for pulmonary nodules according to claim 3, characterized in that: The pulmonary nodule segmentation module uses ResUNet as the main network, the loss function of the segmentation task is Dice loss, and the encoder part of the pulmonary nodule segmentation module is two parallel branches, namely, the pulmonary nodule CT image feature extraction branch and the class activation map feature extraction branch; The pulmonary nodule segmentation module is trained using the pulmonary nodule CT image, mask output by the data import module, and the final class activation map output by the pulmonary nodule classification module. The anatomical attention module uses the class activation map to provide anatomical attention guidance for feature extraction of pulmonary nodule CT images, where: The lung nodule CT image feature extraction branch and the class activation map feature extraction branch both use the encoder part of ResUNet, input the lung nodule CT image into the lung nodule CT image feature extraction branch, input the corresponding class activation map output by the lung nodule classification module into the class activation map feature extraction branch, and use convolution and downsampling operations to extract features from the input image to generate a feature map; The anatomical attention module takes the e-th layer feature map A of the lung nodule CT image feature extraction branch e The e-th layer feature map M of the class activation map feature extraction branch e As input, the feature maps of the two branches are concatenated and then subjected to two parallel 1×1×1 convolutions to obtain the weight tensor of the feature map corresponding to the CT image of the lung nodule. and the weight tensor of the feature map corresponding to the class activation map In the formula, A e and M e By splicing in the channel direction, we get [A e ,M e ],W a e and W m e They are the parameters of the two convolutional layers, * represents the convolution operation, and σ is the Sigmoid activation function; The two sets of weight tensors obtained are respectively multiplied element-by-element with the two sets of input feature maps. Finally, the weighted feature map of the CT image of the lung nodule and the weighted feature map of the class activation map are added element-by-element, and the weighted sum feature map F is output. e : Where · is the element-wise multiplication.

Images