Lung nodule image classification method and system based on multi-scale feature fusion attention

Abstract

The application belongs to the technical field of image processing, and particularly relates to a lung nodule image classification method and system based on multi-scale feature fusion attention. Different scale features are extracted in depth through split-channel convolution processing, then multi-scale feature maps are fused to reduce the influence of feature map scale change, and finally a multi-head attention mechanism is used for processing, global information is learned, important regions are highlighted, and irrelevant parts are suppressed, so that the lung nodule image can be accurately captured and then accurately classified.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and specifically relates to a lung nodule image classification method and system based on multi-scale feature fusion attention. Background Art

[0002] Deep learning technology has been widely used in the detection and classification of lung nodules. Convolutional neural networks (CNNs) are a type of deep learning model that can automatically learn complex and abstract features from datasets. This capability eliminates the need for tedious manual feature design. Although researchers have proposed various pulmonary nodule classification models using CNNs, these models still have limitations that limit their potential for classification accuracy.

[0003] Defects and shortcomings of existing technology:

[0004] 1. Existing lung nodule classification models mainly rely on small kernel convolution, but the receptive field of small kernel convolution is limited and it can usually only focus on local features of the image, which leads to the loss of image information during the processing.

[0005] 2. When processing images, not all areas are equally important. Existing networks only use convolution operations to extract image features, which cannot distinguish between primary and secondary information in the image, resulting in the loss of key information.

[0006] 3. The interactions between lung tumors and their surrounding tissues may exhibit different patterns at different scales. Existing classification models do not integrate multi-scale features and cannot fully understand these interactions. Summary of the Invention

[0007] To solve the problems raised by the background technology, the present invention provides a lung nodule image classification method and system based on multi-scale feature fusion attention.

[0008] The technical solutions of the present invention are as follows:

[0009] The present invention provides a lung nodule image classification method based on multi-scale feature fusion attention, comprising the following steps:

[0010] S1: Obtain the image data of the lung nodule to be detected, perform convolution processing, and then sequentially perform the first feature extraction, the second feature extraction, and the third feature extraction to obtain the first scale feature map, the second scale feature map, and the third scale feature map respectively;

[0011] The feature extraction operation is as follows:

[0012] After several convolutions, the feature map to be processed is processed by layer normalization, channel convolution, and activation function to obtain a scale feature map.

[0013] The channel-wise convolution processing is specifically as follows:

[0014] The feature map after layer normalization is divided into the first feature map, the second feature map, the third feature map and the fourth feature map according to the number of channels. The first feature map, the second feature map and the third feature map are respectively processed by the first depth-separable convolution, the second depth-separable convolution and the third depth-separable convolution to obtain the corresponding first convolution feature map, the second convolution feature map and the third convolution feature map;

[0015] The first convolution feature map, the second convolution feature map, the third convolution feature map, and the fourth feature map are connected to obtain a connected feature map;

[0016] The concatenated feature map is added element-by-element to the layer-normalized feature map to obtain the channel-wise convolution feature map.

[0017] S2: The first scale feature map, the second scale feature map and the third scale feature map are fused to obtain a fused feature map;

[0018] The fused feature map is processed by the multi-head attention mechanism to obtain the fused attention feature map;

[0019] The fused attention feature map is added element-by-element with the third-scale feature map to obtain the feature map to be detected;

[0020] S3: The feature map to be detected is classified through full connection to obtain the classification result.

[0021] The feature fusion in step S2 is performed as follows:

[0022] After the first-scale feature map is convolved, it is fused with the second-scale feature map, and the fused feature map is convolved to obtain a first fused feature map;

[0023] The second scale feature map is processed by convolution to obtain a second fused feature map;

[0024] The first fused feature map, the second fused feature map, and the third scale feature map are added element by element to obtain a fused feature map.

[0025] In step S1, the first depth-wise separable convolution process, the second depth-wise separable convolution process, and the third depth-wise separable convolution process use convolution kernels of sizes 7×1×1, 1×7×1, and 1×1×7, respectively.

[0026] In step S2, the fused feature map is processed by the multi-head attention mechanism to obtain a fused attention feature map, specifically:

[0027] After the fused feature map is feature mapped to obtain the query matrix, key matrix and value matrix, the attention weight matrix of each attention head is calculated, and the attention weight matrices of all attention heads are spliced ​​to obtain the multi-head attention weight matrix. The fused feature map and the multi-head attention weight matrix are weighted normalized to obtain the fused attention feature map.

[0028] The present invention also provides a pulmonary nodule image classification system based on multi-scale feature fusion attention, comprising:

[0029] Feature extraction module: used to obtain the image data of the lung nodules to be detected, perform convolution processing, and then sequentially perform the first feature extraction, the second feature extraction, and the third feature extraction to obtain the first scale feature map, the second scale feature map, and the third scale feature map respectively;

[0030] The feature extraction operation is as follows:

[0031] After several convolutions, the feature map to be processed is processed by layer normalization, channel convolution, and activation function to obtain a scale feature map.

[0032] The channel-wise convolution processing is specifically as follows:

[0033] The feature map after layer normalization is divided into the first feature map, the second feature map, the third feature map and the fourth feature map according to the number of channels. The first feature map, the second feature map and the third feature map are respectively processed by the first depth-separable convolution, the second depth-separable convolution and the third depth-separable convolution to obtain the corresponding first convolution feature map, the second convolution feature map and the third convolution feature map;

[0034] The first convolution feature map, the second convolution feature map, the third convolution feature map, and the fourth feature map are connected to obtain a connected feature map;

[0035] The concatenated feature map is added element-by-element to the layer-normalized feature map to obtain the channel-wise convolution feature map.

[0036] Feature fusion module: used to fuse the first-scale feature map, the second-scale feature map, and the third-scale feature map to obtain a fused feature map;

[0037] The fused feature map is processed by the multi-head attention mechanism to obtain the fused attention feature map;

[0038] The fused attention feature map is added element-by-element with the third-scale feature map to obtain the feature map to be detected;

[0039] Classification module: It is used to classify the feature map to be detected through full connection to obtain the classification result.

[0040] Beneficial effects

[0041] The present invention first uses channel-wise convolution processing to deeply extract features of different scales, then fuses multi-scale feature maps to reduce the impact of scale changes in feature maps, and finally adopts a multi-head attention mechanism to learn global information, highlight important areas, and suppress irrelevant parts, thereby achieving accurate capture of lung nodule images and then accurate classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A flowchart of a lung nodule image classification method based on multi-scale feature fusion attention is provided in an embodiment.

[0043] Figure 2 This is the confusion matrix of the lung nodule image classification method based on multi-scale feature fusion attention adopted in the embodiment of the present application. DETAILED DESCRIPTION

[0044] The following examples are intended to illustrate the present invention rather than to further limit the present invention.

[0045] The present invention provides a lung nodule image classification method based on multi-scale feature fusion attention. Figure 1 As shown, the following steps are included:

[0046] S1: Obtain the image data of the lung nodule to be detected, perform convolution processing, and then perform the first feature extraction, second feature extraction, and third feature extraction in sequence to obtain the first scale feature map, the second scale feature map, and the third scale feature map, respectively.

[0047] Specifically:

[0048] The feature map after convolution processing is first subjected to the first feature extraction to obtain the first-scale feature map;

[0049] The first-scale feature map is subjected to a second feature extraction to obtain a second-scale feature map;

[0050] The second-scale feature map is finally subjected to a third feature extraction to obtain a third-scale feature map.

[0051] The feature extraction of the first feature extraction, the second feature extraction and the third feature extraction is performed as follows:

[0052] The feature map to be processed includes the feature map after convolution processing, the first scale feature map, and the second scale feature map. After several convolution processes, the feature map is sequentially processed by layer normalization, channel convolution processing, and activation function processing to obtain a scale feature map.

[0053] For example, when the feature map to be processed is a feature map after convolution processing, the feature map after convolution processing is subjected to several convolution processes, and then layer normalization, channel convolution processing, and activation function processing (such as GELU activation function) are performed in sequence to obtain a first-scale feature map.

[0054] When the feature map to be processed is a first-scale feature map, the first-scale feature map is subjected to several convolution processes, and then layer normalization, channel-wise convolution, and activation function processing (such as GELU activation function) in sequence to obtain a second-scale feature map.

[0055] When the feature map to be processed is the second-scale feature map, the second-scale feature map is subjected to several convolution processes, and then layer normalization, channel-wise convolution, and activation function processing (such as GELU activation function) are performed in sequence to obtain the third-scale feature map.

[0056] During the feature extraction process, a series of convolution processes are used to extract deep feature information from image data, and activation functions are used to capture high-level representations that are crucial for accurate classification.

[0057] Furthermore, the channel-wise convolution processing is specifically as follows:

[0058] The feature map after layer normalization is divided into the first feature map, the second feature map, the third feature map and the fourth feature map according to the number of channels. The dimension of the feature map after layer normalization is expressed as C×D×W×H, where C is the number of channels, D is the depth, W is the width, and H is the height. In terms of channel dimension, it is divided into the first feature map, the second feature map, the third feature map and the fourth feature map according to the number of channels. The dimensions of the first feature map, the second feature map, the third feature map and the fourth feature map are expressed as

[0059] The first feature map, the second feature map, and the third feature map are respectively subjected to a first depthwise separable convolution process, a second depthwise separable convolution process, and a third depthwise separable convolution process to obtain corresponding first convolution feature maps, second convolution feature maps, and third convolution feature maps. The first convolution feature map may be obtained by convolution in the depth dimension, the second convolution feature map may be obtained by convolution in the width dimension, and the third convolution feature map may be obtained by convolution in the height dimension.

[0060] During the channel-wise convolution process, a convolution kernel of size 7 is used to extract lung nodule image information. The first, second, third, and fourth feature maps are mapped to different convolution branches. Preferably, the first, second, and third depthwise separable convolution processes use convolution kernels of sizes 7×1×1, 1×7×1, and 1×1×7, respectively. The fourth feature map is subjected to identity mapping.

[0061] The first convolution feature map, the second convolution feature map, the third convolution feature map and the fourth feature map are connected, for example, by skipping connections, to retain the underlying features and reduce information loss, thereby obtaining a connected feature map.

[0062] The concatenated feature map is added element-by-element to the layer-normalized feature map to obtain the channel-wise convolutional feature map.

[0063] The present invention adopts a large kernel convolution method in the feature extraction stage to selectively extract information from key channels. In order to improve processing efficiency, convolution is performed on different channels to reduce feature loss.

[0064] S2: The first-scale feature map, the second-scale feature map, and the third-scale feature map are fused to obtain a fused feature map.

[0065] In medical image processing, structures and lesions within medical images can vary in size and shape. Some subtle lesions may only be visible within a small range, while larger structures or abnormalities may require a wider range to be observed. Therefore, it is necessary to learn image features at multiple scales to fully understand the interaction between lesions and surrounding tissues and ensure comprehensive information.

[0066] Since the dimensions of feature maps are inconsistent, the obtained multi-scale feature maps cannot be directly fused. Preferably, the feature fusion operation is as follows:

[0067] After the first-scale feature map is convolved, it is fused with the second-scale feature map, and the fused feature map is convolved to obtain a first fused feature map;

[0068] The second scale feature map is processed by convolution to obtain a second fused feature map;

[0069] The dimensions of the obtained first fused feature map, the second fused feature map and the third scale feature map are the same. Therefore, the first fused feature map, the second fused feature map and the third scale feature map are added element by element to achieve fusion and obtain a fused feature map.

[0070] After multi-scale feature fusion, the impact of feature map scale changes is reduced. For the fused feature map, it is necessary to further extract global context information through the attention mechanism, as follows:

[0071] The fused feature map is processed by the multi-head attention mechanism to obtain the fused attention feature map.

[0072] Preferably, the fused feature map is processed by a multi-head attention mechanism to obtain a fused attention feature map, specifically:

[0073] The fused feature map is feature mapped by three convolution kernels of size 1×1×1 to obtain the query matrix, key matrix and value matrix. The attention weight matrix of each attention head is calculated, and the attention weight matrices of all attention heads are spliced ​​to obtain the multi-head attention weight matrix. The fused feature map and the multi-head attention weight matrix are weighted and normalized to obtain the fused attention feature map.

[0074] The fused attention feature map is added element-by-element with the third-scale feature map to obtain the feature map to be detected.

[0075] After fusing multi-scale features, the present invention adopts a multi-head attention mechanism to capture global information and distinguish primary and secondary information in the image, so as to accurately detect and classify lung nodule images.

[0076] S3: The feature map to be detected is classified through full connection to obtain the classification result.

[0077] The present invention first uses channel-wise convolution processing to deeply extract features of different scales, then fuses multi-scale feature maps to reduce the impact of scale changes in feature maps, and finally adopts a multi-head attention mechanism to learn global information, highlight important areas, and suppress irrelevant parts, thereby achieving accurate capture of lung nodule images and then accurate classification.

[0078] The present invention also provides a pulmonary nodule image classification system based on multi-scale feature fusion attention, comprising:

[0079] Feature extraction module: used to obtain the image data of the lung nodules to be detected, perform convolution processing, and then sequentially perform the first feature extraction, the second feature extraction, and the third feature extraction to obtain the first scale feature map, the second scale feature map, and the third scale feature map respectively;

[0080] The feature extraction operation is as follows:

[0081] After several convolutions, the feature map to be processed is processed by layer normalization, channel convolution, and activation function to obtain a scale feature map.

[0082] The channel-wise convolution processing is specifically as follows:

[0083] The feature map after layer normalization is divided into the first feature map, the second feature map, the third feature map and the fourth feature map according to the number of channels. The first feature map, the second feature map and the third feature map are respectively processed by the first depth-separable convolution, the second depth-separable convolution and the third depth-separable convolution to obtain the corresponding first convolution feature map, the second convolution feature map and the third convolution feature map;

[0084] The first convolution feature map, the second convolution feature map, the third convolution feature map, and the fourth feature map are connected to obtain a connected feature map;

[0085] The concatenated feature map is added element-by-element to the layer-normalized feature map to obtain the channel-wise convolution feature map.

[0086] Feature fusion module: used to fuse the first-scale feature map, the second-scale feature map, and the third-scale feature map to obtain a fused feature map;

[0087] The fused feature map is processed by the multi-head attention mechanism to obtain the fused attention feature map;

[0088] The fused attention feature map is added element-by-element with the third-scale feature map to obtain the feature map to be detected;

[0089] Classification module: It is used to classify the feature map to be detected through full connection to obtain the classification result.

[0090] Experimental results

[0091] Using the Lung Image Database Consortium and Image Database Resource Initiative (LIDC-IDRI) dataset, the proposed lung nodule image classification method based on multi-scale feature fusion attention was configured into the classification model to test its classification performance and evaluate it based on the accuracy (ACC), sensitivity (SEN), specificity (SPE) and F1 score (F1score) evaluation indicators. The specific calculation method is as follows:

[0092]

[0093] Where TP is true positive, indicating that both the actual sample and the predicted result are positive; FN is false negative, indicating that the actual sample is positive but the predicted result is negative; FP is false positive, indicating that the actual sample is negative but the predicted result is positive; TN is true negative, indicating that both the actual sample and the predicted result are negative; ACC is accuracy, the most intuitive metric for evaluating classification performance, indicating the proportion of correctly predicted data to all data; SEN is sensitivity, indicating the proportion of correctly predicted positive samples to all actual positive samples; SPE is specificity, indicating the proportion of correctly predicted negative samples to all actual negative samples. The F1 score ranges from 0 to 1, where 1 represents the best model output and 0 represents the worst model output.

[0094] The classification results are as follows Figure 2 As shown in Figure 2, the confusion matrix shows that the number of TPs, the number of FNs, the number of FPs, and the number of TNs in the classification results obtained on this dataset is 349, 5, 2, and 127.

[0095] According to the confusion matrix, the experimental results are further calculated and shown in Table 1.

[0096] Table 1 Experimental results

[0097]

[0098] As shown in Table 1, the final ACC value is 0.9855, the SEN value is 0.9859, the SPE value is 0.9845, and the F1score value is 0.9901. This shows that the lung nodule image classification method based on multi-scale feature fusion attention in this application has high accuracy, high sensitivity, and high specificity, and can accurately classify lung nodule images.

Claims

1. A lung nodule image classification method based on multi-scale feature fusion attention, characterized in that: The following steps are involved: S1: Obtain the image data of the lung nodule to be detected, perform convolution processing, and then sequentially perform the first feature extraction, the second feature extraction, and the third feature extraction to obtain the first scale feature map, the second scale feature map, and the third scale feature map respectively; The feature extraction operation is as follows: After several convolutions, the feature map to be processed is processed by layer normalization, channel convolution, and activation function to obtain a scale feature map. The channel-wise convolution processing is specifically as follows: The feature map after layer normalization is divided into the first feature map, the second feature map, the third feature map and the fourth feature map according to the number of channels. The first feature map, the second feature map and the third feature map are respectively processed by the first depth-separable convolution, the second depth-separable convolution and the third depth-separable convolution to obtain the corresponding first convolution feature map, the second convolution feature map and the third convolution feature map; The first convolution feature map, the second convolution feature map, the third convolution feature map, and the fourth feature map are connected to obtain a connected feature map; The concatenated feature map is added element-by-element to the layer-normalized feature map to obtain the channel-wise convolution feature map. S2: The first scale feature map, the second scale feature map and the third scale feature map are fused to obtain a fused feature map; The fused feature map is processed by the multi-head attention mechanism to obtain the fused attention feature map; The fused attention feature map is added element-by-element with the third-scale feature map to obtain the feature map to be detected; S3: The feature map to be detected is classified through full connection to obtain the classification result.

2. The lung nodule image classification method based on multi-scale feature fusion attention according to claim 1 is characterized in that: The feature fusion in step S2 is performed as follows: After the first-scale feature map is convolved, it is fused with the second-scale feature map, and the fused feature map is convolved to obtain a first fused feature map; The second scale feature map is processed by convolution to obtain a second fused feature map; The first fused feature map, the second fused feature map, and the third scale feature map are added element by element to obtain a fused feature map.

3. The lung nodule image classification method based on multi-scale feature fusion attention according to claim 1 is characterized in that: In step S1, the first depth-wise separable convolution process, the second depth-wise separable convolution process, and the third depth-wise separable convolution process use convolution kernels of sizes 7×1×1, 1×7×1, and 1×1×7, respectively.

4. The lung nodule image classification method based on multi-scale feature fusion attention according to claim 1, characterized in that: In step S2, the fused feature map is processed by the multi-head attention mechanism to obtain a fused attention feature map, specifically: After the fused feature map is feature mapped to obtain the query matrix, key matrix and value matrix, the attention weight matrix of each attention head is calculated, and the attention weight matrices of all attention heads are spliced ​​to obtain the multi-head attention weight matrix. The fused feature map and the multi-head attention weight matrix are weighted normalized to obtain the fused attention feature map.

5. A pulmonary nodule image classification system based on multi-scale feature fusion attention, characterized in that: include: Feature extraction module: used to obtain the image data of the lung nodules to be detected, perform convolution processing, and then sequentially perform the first feature extraction, the second feature extraction, and the third feature extraction to obtain the first scale feature map, the second scale feature map, and the third scale feature map respectively; The feature extraction operation is as follows: After several convolutions, the feature map to be processed is processed by layer normalization, channel convolution, and activation function to obtain a scale feature map. The channel-wise convolution processing is specifically as follows: The feature map after layer normalization is divided into the first feature map, the second feature map, the third feature map and the fourth feature map according to the number of channels. The first feature map, the second feature map and the third feature map are respectively processed by the first depth-separable convolution, the second depth-separable convolution and the third depth-separable convolution to obtain the corresponding first convolution feature map, the second convolution feature map and the third convolution feature map; The first convolution feature map, the second convolution feature map, the third convolution feature map, and the fourth feature map are connected to obtain a connected feature map; The concatenated feature map is added element-by-element to the layer-normalized feature map to obtain the channel-wise convolution feature map. Feature fusion module: used to fuse the first-scale feature map, the second-scale feature map, and the third-scale feature map to obtain a fused feature map; The fused feature map is processed by the multi-head attention mechanism to obtain the fused attention feature map; The fused attention feature map is added element-by-element with the third-scale feature map to obtain the feature map to be detected; Classification module: It is used to classify the feature map to be detected through full connection to obtain the classification result.

Images