Pulmonary nodule image classification method and system based on multi-scale feature fusion attention

Abstract

The invention belongs to the technical field of image processing, and particularly relates to a pulmonary nodule image classification method and system based on multi-scale feature fusion attention, and the method comprises the steps: carrying out the deep extraction of features of different scales through the channel convolution processing, and carrying out the fusion of multi-scale feature maps, so as to reduce the influence of the scale change of the feature maps. And finally, a multi-head attention mechanism is adopted to process and learn global information, highlight important areas and suppress irrelevant parts, so that accurate capture of the pulmonary nodule image is realized, and accurate classification is further realized.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and specifically relates to a lung nodule image classification method and system based on multi-scale feature fusion attention. Background Art

[0002] Deep learning technology has been widely used in the detection and classification of lung nodules. Among them, convolutional neural network is a deep learning model that can automatically learn complex and abstract features from data sets. This ability eliminates the need for tedious manual feature design processes. Although researchers have proposed a variety of lung nodule classification models using convolutional neural networks, these models still have some limitations, which limit their potential in classification accuracy.

[0003] Defects and shortcomings of the existing technology:

[0004] 1. The existing lung nodule classification model mainly relies on small kernel convolution, but the receptive field of small kernel convolution is limited and it can usually only focus on the local features of the image, which leads to the loss of image information during the processing.

[0005] 2. When processing images, not all areas are equally important. Existing networks only use convolution operations to extract image features and are unable to distinguish between primary and secondary information in the image, resulting in the loss of key information.

[0006] 3. The interaction between lung tumors and their surrounding tissues may show different patterns at different scales. Existing classification models do not integrate multi-scale features and cannot fully understand these interactions. Summary of the invention

[0007] In order to solve the problems raised by the background technology, the present invention provides a lung nodule image classification method and system based on multi-scale feature fusion attention.

[0008] The technical solution of the present invention is as follows:

[0009] The present invention provides a lung nodule image classification method based on multi-scale feature fusion attention, comprising the following steps:

[0010] S1: Obtain the image data of the lung nodules to be detected, perform convolution processing, and then perform the first feature extraction, the second feature extraction, and the third feature extraction in sequence to obtain the first scale feature map, the second scale feature map, and the third scale feature map respectively;

[0011] The feature extraction is performed as follows:

[0012] After several convolutions, the feature map to be processed is processed by layer normalization, channel convolution, and activation function to obtain a scale feature map.

[0013] The channel-wise convolution processing is specifically as follows:

[0014] The feature map after layer normalization is divided into a first feature map, a second feature map, a third feature map and a fourth feature map according to the number of channels. The first feature map, the second feature map and the third feature map are respectively processed by a first depth-separable convolution, a second depth-separable convolution and a third depth-separable convolution to obtain the corresponding first convolution feature map, the second convolution feature map and the third convolution feature map;

[0015] The first convolution feature map, the second convolution feature map, the third convolution feature map, and the fourth feature map are connected to obtain a connected feature map;

[0016] The concatenated feature map is added element by element with the layer normalized feature map to obtain the channel-wise convolution feature map;

[0017] S2: The first scale feature map, the second scale feature map and the third scale feature map are subjected to feature fusion to obtain a fused feature map;

[0018] The fused feature map is processed by the multi-head attention mechanism to obtain a fused attention feature map;

[0019] The fused attention feature map is added element by element with the third-scale feature map to obtain the feature map to be detected;

[0020] S3: The feature graph to be detected is classified through full connection to obtain the classification result.

[0021] The step S2 feature fusion is performed as follows:

[0022] After the first scale feature map is processed by convolution, it is fused with the second scale feature map, and the fused feature map is processed by convolution to obtain a first fused feature map;

[0023] The second scale feature map is processed by convolution to obtain a second fused feature map;

[0024] The first fused feature map, the second fused feature map, and the third scale feature map are added element by element to obtain a fused feature map.

[0025] In step S1, the first depth-wise separable convolution processing, the second depth-wise separable convolution processing, and the third depth-wise separable convolution processing use convolution kernels of sizes 7×1×1, 1×7×1, and 1×1×7, respectively.

[0026] In step S2, the fused feature map is processed by a multi-head attention mechanism to obtain a fused attention feature map, which is specifically:

[0027] After the fused feature map is feature mapped to obtain the query matrix, key matrix and value matrix, the attention weight matrix of each attention head is calculated, the attention weight matrices of all attention heads are concatenated to obtain the multi-head attention weight matrix, and the fused feature map and the multi-head attention weight matrix are weighted normalized to obtain the fused attention feature map.

[0028] The present invention also provides a pulmonary nodule image classification system based on multi-scale feature fusion attention, comprising:

[0029] Feature extraction module: used to obtain the image data of the lung nodules to be detected, and after convolution processing, sequentially perform the first feature extraction, the second feature extraction and the third feature extraction to obtain the first scale feature map, the second scale feature map and the third scale feature map respectively;

[0030] The feature extraction is performed as follows:

[0031] After several convolutions, the feature map to be processed is processed by layer normalization, channel convolution, and activation function to obtain a scale feature map.

[0032] The channel-wise convolution processing is specifically as follows:

[0033] The feature map after layer normalization is divided into a first feature map, a second feature map, a third feature map and a fourth feature map according to the number of channels. The first feature map, the second feature map and the third feature map are respectively processed by a first depth-separable convolution, a second depth-separable convolution and a third depth-separable convolution to obtain the corresponding first convolution feature map, the second convolution feature map and the third convolution feature map;

[0034] The first convolution feature map, the second convolution feature map, the third convolution feature map, and the fourth feature map are connected to obtain a connected feature map;

[0035] The concatenated feature map is added element by element with the layer normalized feature map to obtain the channel-wise convolution feature map;

[0036] Feature fusion module: used to fuse the first scale feature map, the second scale feature map and the third scale feature map to obtain a fused feature map;

[0037] The fused feature map is processed by the multi-head attention mechanism to obtain a fused attention feature map;

[0038] The fused attention feature map is added element by element with the third-scale feature map to obtain the feature map to be detected;

[0039] Classification module: It is used to classify the feature graph to be detected through full connection to obtain the classification result.

[0040] Beneficial Effects

[0041] The present invention firstly extracts features of different scales in depth through channel-wise convolution processing, then fuses multi-scale feature maps to reduce the impact of scale changes of feature maps, and finally adopts a multi-head attention mechanism to learn global information, highlight important areas, and suppress irrelevant parts, thereby achieving accurate capture of lung nodule images and then accurate classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A flowchart of a lung nodule image classification method based on multi-scale feature fusion attention is provided in an embodiment.

[0043] Figure 2 This is the confusion matrix of the lung nodule image classification method based on multi-scale feature fusion attention adopted in the embodiment of the present application. DETAILED DESCRIPTION

[0044] The following examples are intended to illustrate the present invention rather than to further limit the present invention.

[0045] The present invention provides a lung nodule image classification method based on multi-scale feature fusion attention. Figure 1 As shown, the following steps are included:

[0046] S1: Obtain the image data of the lung nodules to be detected, perform convolution processing, and then perform the first feature extraction, the second feature extraction, and the third feature extraction in sequence to obtain the first scale feature map, the second scale feature map, and the third scale feature map, respectively.

[0047] Specifically:

[0048] The feature map after convolution processing is first subjected to the first feature extraction to obtain the first scale feature map;

[0049] The first scale feature map is subjected to a second feature extraction to obtain a second scale feature map;

[0050] The second-scale feature map is finally subjected to a third feature extraction to obtain a third-scale feature map.

[0051] The feature extraction of the first feature extraction, the second feature extraction and the third feature extraction is performed as follows:

[0052] The feature map to be processed includes the feature map after convolution processing, the first scale feature map, and the second scale feature map. After several convolution processes, the feature map is processed by layer normalization, channel convolution processing, and activation function processing in sequence to obtain a scale feature map.

[0053] For example, when the feature map to be processed is a feature map after convolution processing, the feature map after convolution processing is processed by several convolution processes, and then processed by layer normalization, channel convolution processing, and activation function processing (such as GELU activation function) in sequence to obtain a first-scale feature map.

[0054] When the feature map to be processed is a first-scale feature map, the first-scale feature map is subjected to several convolution processes, and then to layer normalization, channel-wise convolution, and activation function processing (such as GELU activation function) in sequence to obtain a second-scale feature map.

[0055] When the feature map to be processed is a second-scale feature map, the second-scale feature map is subjected to several convolution processes, and then to layer normalization, channel-wise convolution, and activation function processing (such as GELU activation function) in sequence to obtain a third-scale feature map.

[0056] During the feature extraction process, a series of convolutions are used to extract deep feature information from image data, and activation functions are used to capture high-level representations that are critical for accurate classification.

[0057] Furthermore, the channel-wise convolution processing is specifically as follows:

[0058] The feature map after layer normalization is divided into the first feature map, the second feature map, the third feature map and the fourth feature map according to the number of channels, where the dimension of the feature map after layer normalization is expressed as C×D×W×H, C is the number of channels, D is the depth, W is the width, and H is the height; in terms of channel dimension, it is divided into the first feature map, the second feature map, the third feature map and the fourth feature map according to the number of channels, where the dimensions of the first feature map, the second feature map, the third feature map and the fourth feature map are expressed as

[0059] The first feature map, the second feature map, and the third feature map are respectively processed by the first depth-separable convolution, the second depth-separable convolution, and the third depth-separable convolution to obtain the corresponding first convolution feature map, the second convolution feature map, and the third convolution feature map. Among them, the first convolution feature map can be obtained by convolution in the depth dimension, the second convolution feature map can be obtained by convolution in the width dimension, and the third convolution feature map can be obtained by convolution in the height dimension.

[0060] During the channel-wise convolution process, a convolution kernel of size 7 is used to extract the pulmonary nodule image information. The first feature map, the second feature map, the third feature map, and the fourth feature map are mapped to different convolution branches. Preferably, the first depth-separable convolution process, the second depth-separable convolution process, and the third depth-separable convolution process use convolution kernels of sizes 7×1×1, 1×7×1, and 1×1×7, respectively. The fourth feature map is identically mapped.

[0061] The first convolution feature map, the second convolution feature map, the third convolution feature map and the fourth feature map are connected, for example, by skipping connections, to retain underlying features and reduce information loss, thereby obtaining a connected feature map.

[0062] The concatenated feature map is added element by element to the layer normalized feature map to obtain the channel-wise convolutional feature map.

[0063] The present invention adopts a large kernel convolution method in the feature extraction stage to selectively extract information from key channels. In order to improve processing efficiency, convolution is performed on a channel-by-channel basis to reduce feature loss.

[0064] S2: The first scale feature map, the second scale feature map and the third scale feature map are subjected to feature fusion to obtain a fused feature map.

[0065] In medical image processing, structures and lesions in medical images may vary in size and shape. Some subtle lesions may only be visible in a small range, while larger structures or abnormalities may require a larger range to be observed. Therefore, it is necessary to learn image features at multiple different scales to fully understand the interaction between the lesion and its surrounding tissue to ensure the comprehensiveness of the information.

[0066] Since the dimensions of feature maps are inconsistent, the obtained multi-scale feature maps cannot be directly fused. Preferably, the feature fusion is performed as follows:

[0067] After the first scale feature map is processed by convolution, it is fused with the second scale feature map, and the fused feature map is processed by convolution to obtain a first fused feature map;

[0068] The second scale feature map is processed by convolution to obtain a second fused feature map;

[0069] The dimensions of the obtained first fused feature map, the second fused feature map and the third scale feature map are the same. Therefore, the first fused feature map, the second fused feature map and the third scale feature map are added element by element to achieve fusion and obtain the fused feature map.

[0070] After multi-scale feature fusion, the impact of feature map scale changes is reduced. For the fused feature map, it is necessary to further extract global context information through the attention mechanism, as follows:

[0071] The fused feature map is processed by the multi-head attention mechanism to obtain the fused attention feature map.

[0072] Preferably, the fused feature map is processed by a multi-head attention mechanism to obtain a fused attention feature map, specifically:

[0073] The fused feature map is feature mapped by three convolution kernels of size 1×1×1 to obtain the query matrix, key matrix and value matrix. The attention weight matrix of each attention head is calculated, and the attention weight matrices of all attention heads are concatenated to obtain the multi-head attention weight matrix. The fused feature map and the multi-head attention weight matrix are weighted normalized to obtain the fused attention feature map.

[0074] The fused attention feature map is added element by element with the third-scale feature map to obtain the feature map to be detected.

[0075] After fusing multi-scale features, the present invention adopts a multi-head attention mechanism to capture global information and distinguish primary and secondary information in the image, so as to accurately detect and classify lung nodule images.

[0076] S3: The feature graph to be detected is classified through full connection to obtain the classification result.

[0077] The present invention firstly extracts features of different scales in depth through channel-wise convolution processing, then fuses multi-scale feature maps to reduce the impact of scale changes of feature maps, and finally adopts a multi-head attention mechanism to learn global information, highlight important areas, and suppress irrelevant parts, thereby achieving accurate capture of lung nodule images and then accurate classification.

[0078] The present invention also provides a pulmonary nodule image classification system based on multi-scale feature fusion attention, comprising:

[0079] Feature extraction module: used to obtain the image data of the lung nodules to be detected, and after convolution processing, sequentially perform the first feature extraction, the second feature extraction and the third feature extraction to obtain the first scale feature map, the second scale feature map and the third scale feature map respectively;

[0080] The feature extraction is performed as follows:

[0081] After several convolutions, the feature map to be processed is processed by layer normalization, channel convolution, and activation function to obtain a scale feature map.

[0082] The channel-wise convolution processing is specifically as follows:

[0083] The feature map after layer normalization is divided into a first feature map, a second feature map, a third feature map and a fourth feature map according to the number of channels. The first feature map, the second feature map and the third feature map are respectively processed by a first depth-separable convolution, a second depth-separable convolution and a third depth-separable convolution to obtain the corresponding first convolution feature map, the second convolution feature map and the third convolution feature map;

[0084] The first convolution feature map, the second convolution feature map, the third convolution feature map, and the fourth feature map are connected to obtain a connected feature map;

[0085] The concatenated feature map is added element by element with the layer normalized feature map to obtain the channel-wise convolution feature map;

[0086] Feature fusion module: used to fuse the first scale feature map, the second scale feature map and the third scale feature map to obtain a fused feature map;

[0087] The fused feature map is processed by the multi-head attention mechanism to obtain a fused attention feature map;

[0088] The fused attention feature map is added element by element with the third-scale feature map to obtain the feature map to be detected;

[0089] Classification module: It is used to classify the feature graph to be detected through full connection to obtain the classification result.

[0090] Experimental Results

[0091] Using the Lung Image Database Consortium and Image Database Resource Initiative (LIDC-IDRI) dataset, the pulmonary nodule image classification method based on multi-scale feature fusion attention in this application is configured to the classification model, and its classification performance is tested and evaluated based on the accuracy (ACC), sensitivity (SEN), specificity (SPE) and F1 score (F1score) evaluation indicators. The specific calculation method is as follows:

[0092]

[0093] In the formula, TP is true positive, which means that both the actual sample and the predicted result are positive; FN is false negative, which means that the actual sample is positive, but the predicted result is negative; FP is false positive, which means that the actual sample is negative, but the predicted result is positive; TN is true negative, which means that both the actual sample and the predicted result are negative; ACC is accuracy, which is the most intuitive indicator for evaluating classification performance, indicating the proportion of correctly predicted data to all data; SEN is sensitivity, which indicates the proportion of correctly predicted positive samples to all actual positive samples; SPE is specificity, which refers to the proportion of correctly predicted negative samples to all actual negative samples. The value range of F1score is 0 to 1, where 1 represents the best model output and 0 represents the worst model output.

[0094] The classification results are as follows Figure 2 As shown, the confusion matrix shows that the number of TPs, the number of FNs, the number of FPs, and the number of TNs in the classification results obtained on this dataset is 349, 5, 2, and 127.

[0095] According to the confusion matrix, the experimental results are further calculated and shown in Table 1.

[0096] Table 1 Experimental results

[0097]

[0098] As can be seen from Table 1, the final ACC value is 0.9855, the SEN value is 0.9859, the SPE value is 0.9845, and the F1score value is 0.9901. This shows that the pulmonary nodule image classification method based on multi-scale feature fusion attention in this application has high accuracy, high sensitivity, and high specificity, and can accurately classify pulmonary nodule images.

Claims

1. A lung nodule image classification method based on multi-scale feature fusion attention, characterized in that: The following steps are involved: S1: Obtain the image data of the lung nodules to be detected, perform convolution processing, and then perform the first feature extraction, the second feature extraction, and the third feature extraction in sequence to obtain the first scale feature map, the second scale feature map, and the third scale feature map respectively; The feature extraction is performed as follows: After several convolutions, the feature map to be processed is processed by layer normalization, channel convolution, and activation function to obtain a scale feature map. The channel-wise convolution processing is specifically as follows: The feature map after layer normalization is divided into a first feature map, a second feature map, a third feature map and a fourth feature map according to the number of channels. The first feature map, the second feature map and the third feature map are respectively processed by a first depth-separable convolution, a second depth-separable convolution and a third depth-separable convolution to obtain the corresponding first convolution feature map, the second convolution feature map and the third convolution feature map; The first convolution feature map, the second convolution feature map, the third convolution feature map, and the fourth feature map are connected to obtain a connected feature map; The concatenated feature map is added element by element with the layer normalized feature map to obtain the channel-wise convolution feature map; S2: The first scale feature map, the second scale feature map and the third scale feature map are subjected to feature fusion to obtain a fused feature map; The fused feature map is processed by the multi-head attention mechanism to obtain a fused attention feature map; The fused attention feature map is added element by element with the third-scale feature map to obtain the feature map to be detected; S3: The feature graph to be detected is classified through full connection to obtain the classification result.

2. The lung nodule image classification method based on multi-scale feature fusion attention according to claim 1 is characterized in that: The step S2 feature fusion is performed as follows: After the first scale feature map is processed by convolution, it is fused with the second scale feature map, and the fused feature map is processed by convolution to obtain a first fused feature map; The second scale feature map is processed by convolution to obtain a second fused feature map; The first fused feature map, the second fused feature map, and the third scale feature map are added element by element to obtain a fused feature map.

3. The lung nodule image classification method based on multi-scale feature fusion attention according to claim 1 is characterized in that: In step S1, the first depth-wise separable convolution processing, the second depth-wise separable convolution processing, and the third depth-wise separable convolution processing use convolution kernels of sizes 7×1×1, 1×7×1, and 1×1×7, respectively.

4. The lung nodule image classification method based on multi-scale feature fusion attention according to claim 1 is characterized in that: In step S2, the fused feature map is processed by a multi-head attention mechanism to obtain a fused attention feature map, which is specifically: After the fused feature map is feature mapped to obtain the query matrix, key matrix and value matrix, the attention weight matrix of each attention head is calculated, the attention weight matrices of all attention heads are concatenated to obtain the multi-head attention weight matrix, and the fused feature map and the multi-head attention weight matrix are weighted normalized to obtain the fused attention feature map.

5. A pulmonary nodule image classification system based on multi-scale feature fusion attention, characterized in that: include: Feature extraction module: used to obtain the image data of the lung nodules to be detected, and after convolution processing, sequentially perform the first feature extraction, the second feature extraction and the third feature extraction to obtain the first scale feature map, the second scale feature map and the third scale feature map respectively; The feature extraction is performed as follows: After several convolutions, the feature map to be processed is processed by layer normalization, channel convolution, and activation function to obtain a scale feature map. The channel-wise convolution processing is specifically as follows: The feature map after layer normalization is divided into a first feature map, a second feature map, a third feature map and a fourth feature map according to the number of channels. The first feature map, the second feature map and the third feature map are respectively processed by a first depth-separable convolution, a second depth-separable convolution and a third depth-separable convolution to obtain the corresponding first convolution feature map, the second convolution feature map and the third convolution feature map; The first convolution feature map, the second convolution feature map, the third convolution feature map, and the fourth feature map are connected to obtain a connected feature map; The concatenated feature map is added element by element with the layer normalized feature map to obtain the channel-wise convolution feature map; Feature fusion module: used to fuse the first scale feature map, the second scale feature map and the third scale feature map to obtain a fused feature map; The fused feature map is processed by the multi-head attention mechanism to obtain a fused attention feature map; The fused attention feature map is added element by element with the third-scale feature map to obtain the feature map to be detected; Classification module: It is used to classify the feature graph to be detected through full connection to obtain the classification result.

Images