Pulmonary nodule risk assessment method based on multi-scale multi-receptive-field multi-attention fusion

Abstract

The invention proposes a pulmonary nodule risk assessment method based on multi-scale multi-receptive-field multi-attention fusion, and the method comprises the steps: firstly constructing a pulmonary nodule image classification model based on a deep learning network, and the pulmonary nodule image classification model based on the deep learning network comprises an image preprocessing module, a deep learning network module TMFM, and a classification module; then acquiring a data set containing a pulmonary nodule CT scanning image and a category number label of a pulmonary nodule malignant risk level corresponding to the image, and training a pulmonary nodule image classification model based on a deep learning network; and inputting a to-be-evaluated pulmonary nodule CT scanning image into the trained pulmonary nodule image classification model based on the deep learning network for feature extraction, fusion and classification to obtain a pulmonary nodule risk evaluation classification result of the pulmonary nodule CT scanning image. The method has higher adaptability and accuracy in risk assessment of pulmonary nodules.

Description

Technical Field

[0001] The present invention relates to the field of electronic information technology, and in particular to a lung nodule risk assessment method based on multi-scale, multi-receptive field, and multi-attention fusion. Background Art

[0002] Lung cancer has always been one of the most common cancers threatening human health worldwide. Therefore, early diagnosis of lung cancer is extremely important for early treatment and cure of the disease. Early lung cancer is characterized by small nodules, which can be detected as round and spherical structures in computed tomography images (CT scan images). The identification and diagnosis of benign and malignant nodules usually require doctors to extract multiple features of nodules from CT scan images for analysis, such as size, shape, contour, interval growth, location and calcification of lung nodules between two CT scans. These features help to classify nodules as benign (non-cancerous) or malignant (cancerous). For example, malignant nodules usually have more irregular borders / edges compared to benign nodules, while benign nodules usually have smoother borders. The disadvantage of manual identification is that it takes a lot of time and effort, and it is easy to miss or misdiagnose if it relies solely on personal experience.

[0003] In recent years, convolutional neural networks (CNNs) have achieved great success in the field of image processing. As a core method, convolutional neural networks (CNNs) have been widely used in tasks such as target detection, image segmentation, and image classification. In recent studies, they have also been applied to the classification / detection of pulmonary nodules. Existing classification methods include: (1) applying three Resnet network topologies / structures to nodule classification tasks, where the three Resnet networks are applied to different views / planes of the nodules, namely, axial, coronal, and sagittal; then the outputs of the three different Resnet networks are passed to a fully connected multilayer perceptron network. (2) Using three-dimensional (3D) CNNs for pulmonary nodule risk stratification to utilize volume information in CT scans. (3) A multi-view CNN applied to the problem of lung nodule detection used three scales and four views, generating a total of 12 different images. A convolutional neural network was trained for each image separately, and then all the models were merged and the convolutional neural network was retrained. In the preprocessing stage, linear interpolation, normalized spherical sampling using an icosahedron located at the center of the nodule, and the threshold method were used to approximate the nodule radius. This method achieved a classification rate of 92.1% for the benign and malignant binary classification.

[0004] However, the existing technology still has the following problems: (1) One of the main problems faced by computer-aided diagnosis (CAD) schemes for pulmonary nodule detection and classification is the risk level classification of malignant nodules. Generally speaking, there are differences in risk levels between malignant nodules. However, all the CAD schemes proposed in previous studies did not provide a solution to the problem of malignant nodule classification, but only distinguished between benign and malignant nodules.

[0005] (2) Since lung nodules in CT images are of different sizes and shapes, it is difficult to fully capture the effective information in the image by extracting features at a single scale or designing a fixed receptive field. In particular, the performance of the model is often limited when dealing with scenes with complex backgrounds or multiple targets.

[0006] (3) Existing methods often only optimize a single attention mechanism or specific scale features, and it is difficult to establish an effective feature fusion strategy between multi-scale and multi-attention mechanisms. Summary of the invention

[0007] The main purpose of the present invention is to propose a lung nodule risk assessment method based on multi-scale, multi-receptive field and multi-attention fusion to solve the deficiencies mentioned in the above background technology.

[0008] The present invention proposes a lung nodule risk assessment method based on multi-scale, multi-receptive field, and multi-attention fusion, which comprises the following steps: Step S1, constructing a lung nodule image classification model based on a deep learning network, wherein the lung nodule image classification model based on a deep learning network includes an image preprocessing module, a deep learning network module TMFM, and a classification module. The image preprocessing module is used to perform feature pre-extraction on the input image to obtain an initial feature tensor of the input image; The deep learning network module TMFM includes a feature extraction module and a feature fusion module; the feature extraction module is used to perform multi-scale feature extraction on the feature tensor obtained by the image preprocessing module; the feature fusion module is used to fuse the feature tensors extracted by the feature extraction module; Among them, the feature fusion module includes a spatial attention mechanism SA, a channel attention mechanism CA, a pixel attention mechanism PA and a feature weighted output module; the spatial attention mechanism SA is used to enhance the significant area in the spatial dimension of the feature tensor; the channel attention mechanism CA is used to assign weights on the channel dimension of the feature tensor, adaptively highlight channels with higher relevance, and reduce the influence of invalid channels; the pixel attention mechanism PA is used to refine the feature weights at the pixel level, perform weighted optimization according to the importance of each pixel point, and highlight the features of key pixels; the feature weighted output module is used to uniformly optimize the feature tensors processed and fused by the spatial attention mechanism SA, the channel attention mechanism CA, and the pixel attention mechanism PA, and generate a key feature tensor containing complete multi-scale, multi-receptive field and multi-attention optimization features; The classification module is used to process the key feature tensor and output the pulmonary nodule risk assessment classification result; Step S2, obtaining a data set including CT scan images of lung nodules and category number labels of the malignant risk levels of lung nodules corresponding to the images, training the lung nodule image classification model based on the deep learning network constructed in step S1, and obtaining a trained lung nodule image classification model based on the deep learning network; In step S3, the CT scan image of the lung nodule to be evaluated is input into the lung nodule image classification model based on the deep learning network trained in step S2 for feature extraction, fusion and classification, so as to obtain the lung nodule risk assessment classification result of the lung nodule CT scan image.

[0009] Furthermore, the image preprocessing module includes an input convolution layer and four residual block groups connected in sequence, wherein the input convolution layer includes a convolution layer, a batch normalization layer, an activation layer and a maximum pooling layer; the four residual block groups respectively contain 3, 4, 6 and 3 convolution layers with skip connections.

[0010] Furthermore, the feature extraction module is a multi-branch structure, which includes six parallel branches, namely a convolution operation, three hole convolution operations, an average pooling layer and a wavelet convolution operation. The features output by the six branches are spliced ​​in the channel dimension as the output of the feature extraction module.

[0011] Furthermore, the feature fusion module performs feature extraction of the spatial dimension and channel dimension on the feature tensor extracted by the feature extraction module through the spatial attention mechanism SA and the channel attention mechanism CA, respectively, to obtain the spatial feature tensor and the channel feature tensor, and adds the spatial feature tensor and the channel feature tensor to obtain the prior feature tensor, performs weighted optimization on the feature tensor and the prior feature tensor extracted by the feature extraction module through the pixel attention mechanism PA, to obtain the pixel feature tensor, normalizes the dimension of the pixel feature tensor, and fuses it with the spatial feature tensor and the channel feature tensor to obtain the attention fusion feature tensor, and then processes the attention fusion feature tensor through the feature weighted output module to obtain the key feature tensor.

[0012] Furthermore, the spatial attention mechanism SA includes three parallel spatial pooling modules, a 1×1 convolution operation and a Sigmoid function. The three parallel spatial pooling modules are a spatial average pooling module, a spatial maximum pooling module, and a spatial median pooling module; the outputs of the three spatial pooling modules are concatenated in the channel dimension and input into the 1×1 convolution operation for tensor dimension recovery, and then normalized by the Sigmoid function. The output of the Sigmoid function is multiplied element-by-element with the input of the spatial attention mechanism SA as the output of the spatial attention mechanism SA.

[0013] Furthermore, the channel attention mechanism CA includes three parallel channel pooling modules and an expand function, wherein the three parallel channel pooling modules are a channel average pooling module, a channel maximum pooling module and a channel median pooling module, respectively; the expand function includes a fully connected layer fc1, a Relu activation function and a fully connected layer fc2 connected in sequence, and the output of the fully connected layer fc2 is dot-multiplied with the input of the channel attention mechanism CA to obtain the output of the expand function; wherein the outputs of the three channel pooling modules are respectively weighted by the expand function and fused with the input of the channel attention mechanism CA to obtain three feature tensors after adjusting the weights; and the three feature tensors after adjusting the weights are added to obtain the final output of the channel attention mechanism CA.

[0014] Furthermore, the pixel attention mechanism PA includes a high-dimensional transformation operation, a splicing operation, a rearrangement operation, a convolution operation and a Sigmoid function.

[0015] Furthermore, the classification module includes an average pooling layer, a flattening operation layer and a fully connected layer. The average pooling layer is used to perform feature sorting on the key feature tensor output by the deep learning network module TMFM. The flattening operation layer is used to reshape the feature dimension of the feature tensor to conform to the input dimension of the fully connected layer. The fully connected layer is used to calculate the probability scores of the lung nodule samples in the input image belonging to five lung nodule malignancy risk levels, generate a classification result vector, and convert the classification result vector into a probability distribution vector. The category number of the lung nodule malignancy risk level corresponding to the item with the largest probability value in the probability distribution vector is taken as the lung nodule risk assessment classification result of the lung nodule CT scan image.

[0016] Starting from multi-scale, multi-receptive field and multi-attention, the present invention designs a multi-scale multi-receptive field multi-attention fusion module, named as deep learning network module TMFM, taking into account the size, morphology and malignant characteristics of lung nodules. And on the basis of the deep learning network module TMFM, a lung nodule image classification model based on deep learning network is constructed, so that the multi-attention mechanism can be guided by prior knowledge in the deep learning network, which is conducive to the deep learning network to learn and classify lung nodules more carefully.

[0017] Compared with the prior art, the present invention has the following beneficial effects: (1) The present invention uses a multi-scale, multi-receptive field feature extraction method to perform comprehensive feature analysis and optimization on lung nodule CT scan image data. The core innovation of this design is that by introducing convolution structures of different scales and receptive fields, the lung nodule image classification model based on the deep learning network can more comprehensively understand the detailed information and large-scale contextual information in the image, thereby greatly improving the classification accuracy. This feature extraction method avoids the defect of traditional methods that easily ignore local detail information at a single scale, and also overcomes the problem that overly simple convolution structures are insufficient in capturing large-scale contextual information.

[0018] (2) Combining three different attention mechanisms, namely channel attention, spatial attention and pixel attention, the features are dynamically optimized and enhanced. Traditional convolutional neural networks often rely on fixed convolution kernels to process images. Although this method is simple and effective, it may still ignore important features such as the boundaries and positions of lung nodules. The present invention introduces multiple attention mechanisms, so that the deep learning network module TMFM can weight features according to the importance of different regions and scales in the image, so that the lung nodule image classification model based on the deep learning network can capture key information more accurately, not only focusing on the key parts of the image, but also maintaining strong robustness in complex backgrounds and noise. Compared with traditional feature extraction methods, the attention mechanism of the deep learning network module TMFM significantly improves the generalization ability and noise resistance of the model, making the method of the present invention more adaptable and accurate in the risk assessment of lung nodules.

[0019] (3) In the risk assessment of lung nodules, important information in the image is often scattered at different levels and scales. Therefore, how to effectively fuse multi-level and multi-scale feature information has become one of the key issues to improve the performance of the classification model. The deep learning network module TMFM adopts an innovative feature fusion strategy. It generates prior information of the image by using the pixel attention mechanism PA, uses the prior information as a guide to fuse the output of the spatial attention mechanism SA and the channel attention mechanism CA, splices feature tensors from different dimensions and scales, and optimizes them in combination with the attention mechanism, so that the network can comprehensively understand the characteristics of lung nodules from multiple levels and perspectives, and finally obtain more accurate classification results, and also enhance the interpretability of the classification results. Since the lung nodule image classification model based on the deep learning network constructed by the present invention can focus on the key features of the image at multiple levels and scales, the decision-making process of the model becomes clearer and more transparent. This is particularly important for deep learning models in medical image analysis because it can help doctors better understand the judgment basis of the model, thereby increasing the credibility of the model in actual clinical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a schematic diagram of the overall structure of the deep learning network module TMFM in the lung nodule risk assessment method based on multi-scale, multi-receptive field and multi-attention fusion proposed in the present invention.

[0021] Figure 2 Schematic diagram of the structure of the spatial attention mechanism SA in the deep learning network module TMFM of the present invention.

[0022] Figure 3 It is a structural diagram of the channel attention mechanism CA in the deep learning network module TMFM of the present invention.

[0023] Figure 4 Schematic diagram of the structure of the pixel attention mechanism PA in the deep learning network module TMFM of the present invention.

[0024] Figure 5 This is a distribution diagram of image samples corresponding to the malignancy risk level of lung nodules in the LIDC-IDRI dataset used for model training in the present invention.

[0025] Figure 6 The heat map generated by benchmarking some lung nodule image samples in the LIDC-IDRI dataset using the lung nodule image classification model based on the deep learning network of the present invention.

[0026] Figure 7 The heat map generated by benchmarking the lung nodule image classification model based on the deep learning network of the present invention with the existing deep learning network on the same lung nodule image sample in the LIDC-IDRI dataset. DETAILED DESCRIPTION

[0027] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0028] With reference to the following description and accompanying drawings, embodiments of the present invention will be clear. In these descriptions and accompanying drawings, some specific implementations in the embodiments of the present invention are specifically disclosed to represent some ways of implementing the principles of the embodiments of the present invention, but it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0029] like Figure 1 As shown, an embodiment of the present invention provides a lung nodule risk assessment method based on multi-scale multi-receptive field multi-attention fusion, and the method includes the following steps: Step S1, constructing a lung nodule image classification model based on a deep learning network, wherein the lung nodule image classification model based on a deep learning network includes an image preprocessing module, a deep learning network module TMFM, and a classification module.

[0030] Among them, the image preprocessing module is used to pre-extract features of the input image. In this embodiment, the image preprocessing module includes an input convolution layer and four residual block groups, wherein the input convolution layer includes a convolution layer with a convolution kernel size of 7×7, a batch normalization layer, an activation layer and a maximum pooling layer, and the activation layer is composed of a Relu activation function; the four residual block groups are arranged in sequence according to the input order, and the four residual block groups respectively contain 3, 4, 6 and 3 convolution layers with skip connections. The skip connection means that the network can skip the layer to transmit data when the threshold is reached, so that the network can alleviate the gradient vanishing problem through identity mapping, thereby better training the deep network.

[0031] The deep learning network module TMFM is used for feature extraction. The deep learning network module TMFM includes a feature extraction module and a feature fusion module. The feature extraction module is a multi-branch structure and is used to perform multi-scale feature extraction on the feature tensor obtained by the image preprocessing module; the feature fusion module is used to fuse the feature tensors extracted by the feature extraction module. The feature fusion module includes a spatial attention mechanism SA, a channel attention mechanism CA, a pixel attention mechanism PA and a feature weighted output module; wherein the spatial attention mechanism SA is used to enhance the significant areas in the spatial dimension of the feature tensor, and enhances the features related to the target area in the feature map by weighted reinforcement, while suppressing irrelevant or redundant background information, thereby improving the attention to the target area; the channel attention mechanism CA is used to allocate weights on the channel dimension of the feature tensor, adaptively highlighting the features with Channels with higher correlation reduce the influence of invalid channels, thereby improving the efficiency and accuracy of feature expression; the pixel attention mechanism PA is used to refine the feature weights at the pixel level, perform weighted optimization according to the importance of each pixel, and highlight the features of key pixels. It is particularly suitable for feature extraction of small targets or target areas with complex boundaries; the feature weighted output module is used to uniformly optimize the feature tensors processed and fused by the spatial attention mechanism SA, channel attention mechanism CA, and pixel attention mechanism PA, so that the weights assigned by the spatial attention mechanism SA, channel attention mechanism CA, and pixel attention mechanism PA can be combined to generate the final key feature tensor. The key feature tensor contains complete multi-scale, multi-receptive field and multi-attention optimized features, which are displayed in the form of a heat map, providing higher quality input features for subsequent classification or regression tasks.

[0032] The classification module is used to output the lung nodule risk assessment classification result of the lung nodule CT scan image. The classification module includes an average pooling layer, a flattening operation layer and a fully connected layer. The classification module performs feature sorting on the output key feature tensor obtained by the deep learning network module TMFM through the average pooling layer, and then reshapes the feature dimension to meet the input dimension of the fully connected layer through the flattening operation layer. Finally, the feature tensor with changed dimension after processing by the flattening operation layer is input into the fully connected layer. The probability score of the lung nodule samples in the input image belonging to the five lung nodule malignancy risk levels is calculated through the fully connected layer, and the lung nodule risk assessment classification result of the lung nodule CT scan image is obtained.

[0033] The specific process of processing the input image by the constructed lung nodule image classification model based on deep learning network is as follows: Step S11, input the input image into the image preprocessing module for feature pre-extraction to obtain a feature tensor L; then perform multi-scale feature extraction on the feature tensor L through a multi-branch structure. The multi-branch structure includes six parallel branches, which are a convolution operation, three parallel dilated convolution operations, an average pooling layer, and a wavelet convolution operation. In the convolution operation, the size of the convolution kernel is 1×1, and in the three parallel dilated convolution operations, the size of the convolution kernel is 3×3, and the dilation rates are 3, 5, and 7, respectively. The features output by the six branches are concatenated in the channel dimension to obtain a feature tensor X. The dimension of the feature tensor X is B×C×H×W, where B is the batch size, C is the number of channels, and H and W are the height and width of the feature tensor, respectively.

[0034] Among them, the process of wavelet convolution operation is as follows: using db1 wavelet basis, the feature tensor L is decomposed in the frequency domain to extract multi-band feature information; then, the inverse wavelet transform is used to fuse the multi-band feature information in the frequency domain to obtain the fused feature information; finally, after a convolution operation with a convolution kernel size of 1×1, the features of the fused feature information are re-extracted.

[0035] Step S12, perform feature fusion operation on the feature tensor X obtained in step S11. The spatial dimension and channel dimension features of the feature tensor X are extracted by the spatial attention mechanism SA and the channel attention mechanism CA respectively to obtain the spatial feature tensor Y S and the channel feature tensor Y C , and transform the spatial feature tensor Y S and the channel feature tensor Y C Add them together to get the prior feature tensor P1; then use the pixel attention mechanism PA to perform a pixel-level dot multiplication operation on the feature tensor X and the prior feature tensor P1 to get the pixel feature tensor Y P , and the pixel feature tensor Y PNormalize the dimension to get the feature tensor P2; then compare the feature tensor P2 with the spatial feature tensor Y S and the channel feature tensor Y C Perform a dot multiplication operation, and then add the two feature tensors obtained after the dot multiplication operation to the feature tensor X obtained in step S11, so as to use the prior feature tensor P1 to guide the feature tensors obtained by the spatial attention mechanism SA and the channel attention mechanism CA to fuse, and input the fused feature tensor into the convolution operation with a convolution kernel size of 1×1 for processing to obtain the attention fusion feature tensor Y. The specific steps of the feature fusion operation are as follows: Step S121: Input the feature tensor X obtained in step S11 into the spatial attention mechanism SA and the channel attention mechanism CA to extract features in the spatial dimension and channel dimension respectively, and obtain the spatial feature tensor Y S and the channel feature tensor Y C . The spatial feature tensor Y S and the channel feature tensor Y C Add them together to get the prior feature tensor P1.

[0036] like Figure 2 As shown in FIG. 1 , the spatial attention mechanism SA includes three parallel spatial pooling modules, a 1×1 convolution operation and a Sigmoid function. The three parallel spatial pooling modules are a spatial average pooling module, a spatial maximum pooling module and a spatial median pooling module. The outputs of the three spatial pooling modules are concatenated in the channel dimension and input into the 1×1 convolution operation for tensor dimension recovery. The output of the 1×1 convolution operation is normalized by the Sigmoid function. The output of the Sigmoid function is multiplied element-by-element with the input of the spatial attention mechanism SA as the output of the spatial attention mechanism SA, i.e., the final spatial feature tensor Y is obtained. S .

[0037] Among them, the specific steps of extracting and fusing the feature tensor X through the spatial attention mechanism SA of the feature fusion module are as follows: Step S1211: The feature tensor X obtained in step S11 is extracted through three spatial pooling modules at the spatial level to obtain three spatially pooled feature tensors X avg , X max , X median The three spatial pooling modules are: spatial average pooling module, spatial maximum pooling module and spatial median pooling module; Among them, the spatial average pooling module is defined as: , Among them, X represents the feature tensor, X avgrepresents the feature tensor after spatial average pooling, C represents the number of channels of the feature tensor X, Represents the spatial set of the cth channel dimension of the feature tensor X. The feature tensor X obtained by processing the spatial average pooling module avg The dimensions are B×1×H×W.

[0038] The spatial maximum pooling module is defined as: , Among them, X represents the feature tensor, X max Represents the feature tensor after spatial maximum pooling, C represents the number of channels of the feature tensor X, and max(X[c,:,:]) represents the maximum value of the spatial set of the feature tensor X in the cth channel dimension. The feature tensor X obtained after processing by the spatial maximum pooling module max The dimensions are B×1×H×W.

[0039] The spatial median pooling module is defined as: , Among them, X represents the feature tensor, X median Represents the feature tensor after spatial median pooling, C represents the number of channels of the feature tensor X, and median(X[c,:,:]) represents the median value of the spatial set of the feature tensor X in the cth channel dimension. The feature tensor X obtained by the spatial median pooling module median The dimensions are B×1×H×W.

[0040] Step S1212: The spatially pooled feature tensor X obtained in step S1211 is avg , feature tensor X max , feature tensor X median The concatenation is performed on the channel dimension, and then a 1×1 convolution operation and a Sigmoid function are performed to restore the tensor dimension and normalize it to generate the spatial feature tensor A, which is specifically expressed as: , Among them, A represents the spatial feature tensor after normalization, and the dimension of the spatial feature tensor A is B×1×H×W; Represents the Sigmoid function; Conv 1×1 Represents a convolution operation with a convolution kernel size of 1×1; Represents the operation of concatenating feature tensors along the channel dimension.

[0041] Step S1213, multiply the spatial feature tensor A obtained in step S1212 by the feature tensor X element by element to obtain the final spatial feature tensor Y S , specifically expressed as: , Among them, Y S Represents the final spatial feature tensor obtained by the spatial attention mechanism SA, the spatial feature tensor Y S The dimensions are B×1×H×W, Represents the Hadamard product of tensors, that is, the multiplication operation between elements in corresponding positions one by one.

[0042] like Figure 3 As shown, the channel attention mechanism CA includes three parallel channel pooling modules and an expand function, wherein the three parallel channel pooling modules are a channel average pooling module, a channel maximum pooling module and a channel median pooling module, respectively; the expand function includes a fully connected layer fc1, a Relu activation function and a fully connected layer fc2 connected in sequence, and the output of the fully connected layer fc2 is multiplied by the input of the channel attention mechanism CA to obtain the output of the expand function; wherein the outputs of the three channel pooling modules are respectively adjusted by the expand function for weight adjustment and fused with the input of the channel attention mechanism CA, namely, the feature tensor X, to obtain three feature tensors Y1, Y2 and Y3 after weight adjustment; the three feature tensors Y1, Y2 and Y3 after weight adjustment are added to obtain the final output of the channel attention mechanism CA, namely, the channel feature tensor Y C .

[0043] Among them, the specific steps of extracting and fusing the feature tensor X through the channel attention mechanism CA of the feature fusion module are as follows: Step S1214: The feature tensor X extracted in step S11 is respectively subjected to feature extraction by three channel pooling modules at the channel level to obtain a feature tensor F after three channel pooling. avg 、F max 、F median The three channel pooling modules are: channel average pooling module, channel maximum pooling module and channel median pooling module; Among them, the channel average pooling module is defined as: , Among them, X represents the feature tensor, F avg It represents the feature tensor after channel average pooling. H and W represent the height and width of the feature tensor X. X[:,i,j] represents the channel set of the spatial dimension formed by taking the feature tensor X at the i-th height and j-th width. The feature tensor F obtained by the channel average pooling module is avg The dimensions are B×C×1×1.

[0044] The channel maximum pooling module is defined as: , Among them, X represents the feature tensor, F max represents the feature tensor after channel maximum pooling, max(X[:,i,j]) means taking the maximum value of the channel set of the feature tensor X at the i-th height and j-th width to form the spatial dimension. The feature tensor F obtained after processing by the channel maximum pooling module max The dimensions are B×C×1×1.

[0045] The channel median pooling module is defined as: , Among them, X represents the feature tensor, F median Represents the feature tensor after channel median pooling. median(X[:,i,j]) means taking the median value of the channel set of the feature tensor X at the i-th height and j-th width to form the spatial dimension. The feature tensor F obtained after processing by the channel median pooling module median The dimensions are B×C×1×1.

[0046] Step S1215, create an expand function, which is used to adjust the channel weights of the feature tensors after the three channels are pooled and fuse the information of the feature tensor X. The specific expression of the expand function is: , Where expand(·) represents the expand function, X represents the feature tensor, and F represents the feature tensor F after three channel pooling. avg 、F max 、F median Any one of them, fc1 represents a fully connected layer that compresses the feature tensor after channel pooling in the channel dimension, fc2 represents a fully connected layer that restores the feature tensor to the dimension of the feature tensor X; Relu represents the Relu activation function, X[:,i,j] represents the channel set of the spatial dimension formed by the i-th height and j-th width of the feature tensor X, and F[:,i,j] represents the channel set of the spatial dimension formed by the i-th height and j-th width of the feature tensor F after channel pooling.

[0047] The feature tensors F after the three channel pooling obtained in step S1214 are respectively expanded by the expand function avg 、F max 、F median Adjust the channel weights to obtain three feature tensors Y1, Y2, and Y3 after weight adjustment. Finally, add the three feature tensors Y1, Y2, and Y3 after weight adjustment to obtain the channel feature tensor Y C , specifically expressed as: , Among them, Y C Represents the final channel feature tensor obtained by the channel attention mechanism CA, channel feature tensor Y C The dimensions are B×C×1×1.

[0048] Step S1216: transform the spatial feature tensor Y S and the channel feature tensor Y C The addition results in the prior feature tensor P1, which has a dimension of B×C×H×W.

[0049] Step S122, extract pixel-level global information from the feature tensor X obtained in step S11 and the feature tensor P1 obtained in step S121 through the pixel attention mechanism PA, and then obtain the pixel feature tensor Y after rearrangement operation, convolution operation and Sigmoid function normalization. P .

[0050] like Figure 4 As shown in FIG. 1 , the pixel attention mechanism PA includes two key steps: pixel-level global information extraction and feature fusion. First, the feature tensor X obtained in step S11 and the feature tensor P1 obtained in step S121 are respectively transformed into high dimensions, and then the two transformed feature tensors are concatenated to form a tensor S. Then, the dimensions of the tensor S are rearranged by a rearrangement operation to obtain the rearranged tensor S. R , so as to facilitate the subsequent feature fusion operation. Then, the rearranged tensor S is subjected to a convolution operation with a convolution kernel size of 1×1. R Perform global pixel feature extraction and use the Sigmoid function to normalize the convolution result to obtain the final output of the pixel attention mechanism PA, that is, the pixel feature tensor Y P .

[0051] Among them, the specific steps of extracting pixel-level global information through the pixel attention mechanism PA of the feature fusion module are as follows: Step S1221, concatenate the feature tensor X obtained in step S11 and the feature tensor P1 obtained in step S121 (both with dimensions of B×C×H×W) in high dimensions to form a new tensor S, whose dimensions are B×C×2×H×W. Then, use the rearrangement operation to rearrange the dimensions of tensor S, converting tensor S from dimensions of B×C×2×H×W to dimensions of B×2C×H×W, and obtaining the rearranged tensor S R , the rearranged tensor S R The dimensions are B×2C×H×W.

[0052] Step S1222, the rearranged tensor S RPerform convolution operation, and finally use Sigmoid function to normalize the convolution result to obtain the pixel feature tensor Y P , specifically expressed as: , Among them, Y P Represents the pixel feature tensor, pixel feature tensor Y P The dimensions are B×C×H×W.

[0053] Step S1223, the pixel feature tensor Y is then processed by a Sigmoid function. P Normalize the dimension and get the feature tensor P2.

[0054] Step S123, the feature tensor X obtained in step S11, the feature tensor P2 obtained in step S122, and the spatial feature tensor Y obtained in step S121 S and the channel feature tensor Y C Perform feature fusion, that is, the feature tensor P2 is combined with the spatial feature tensor Y S and the channel feature tensor Y C Perform a dot multiplication operation, and then add the two feature tensors obtained after the dot multiplication operation to the feature tensor X obtained in step S11. The feature tensor obtained by the addition is subjected to a convolution operation with a convolution kernel size of 1×1 to restore the tensor dimension and normalize it to obtain the attention fusion feature tensor Y; specifically expressed as: , Among them, Y represents the attention fusion feature tensor.

[0055] Step S13, the feature weighted output module includes a dot multiplication operation and a convolution operation with a convolution kernel size of 1×1. The attention fusion feature tensor Y obtained in step S12 is dot-multiplied with the feature tensor X obtained in step S11 to obtain a feature weighted feature tensor, and then the feature weighted feature tensor is subjected to a convolution operation with a convolution kernel size of 1×1 to reduce the dimension and further extract features, and finally generate a key feature tensor that can distinguish the malignant risk level of pulmonary nodules, and convert it into a heat map for evaluating the malignant risk level of pulmonary nodules. The key feature tensor is named feature tensor M, and the dimension of feature tensor M is B×C×H×W.

[0056] Step S14, the classification module consists of an average pooling layer, a flattening operation layer and a fully connected layer. The classification module is used to calculate the probability score of the lung nodule samples in the input image belonging to the five lung nodule malignant risk levels based on the key feature tensor (i.e., the feature tensor M). The feature tensor M output by the deep learning network module TMFM is input into the classification module, and the features are sorted through the average pooling layer. The feature dimension is reshaped to meet the input dimension of the fully connected layer through the flattening operation layer. Finally, the feature tensor with changed dimension after processing obtained by the flattening operation layer is input into the fully connected layer. The probability score of the lung nodule samples in the input image belonging to the five lung nodule malignant risk levels is calculated through the fully connected layer, and the lung nodule risk assessment classification result of the lung nodule CT scan image is obtained. Among them, the specific steps of calculating the probability score of the lung nodule samples in the input image belonging to the five lung nodule malignant risk levels based on the feature tensor M are as follows: Step S141, input the feature tensor M with the dimension of B×C×H×W obtained by the deep learning network module TMFM into the average pooling layer of the classification module, perform global average pooling on the feature tensor M in the channel dimension, and generate a feature tensor M after feature sorting avg , the feature tensor M avg The dimension is B×C×1×1, and the specific process is expressed as: , Among them, M represents the feature tensor M obtained by the deep learning network module TMFM, H and W represent the height and width of the feature tensor M, and M[:,i,j] represents the channel set of the spatial dimension formed by taking the feature tensor M at the i-th height and j-th width. avg Represents the feature tensor obtained after the average pooling layer of the classification module is averaged in the channel dimension, the feature tensor M avg The dimensions are B×C×1×1.

[0057] Step S142: The feature tensor M obtained in step S141 is avg Input to the flattening operation layer, the feature tensor M avg Reshape the dimension to obtain the flattened feature tensor M flat , the flattened feature tensor M flat The dimension is B×C.

[0058] Step S143: The fully connected layer uses a uniform distribution or a normal distribution to initialize the weight matrix W. R and the bias vector b r Specifically, the feature tensor M is used flat The weight matrix W is initialized with a uniform distribution of the number of channels C R , so that the weight matrix W R exist Randomly generate within the range of, initialize the bias vector b r Based on the feature tensor M flat The number of channels C is a uniformly distributed random value or all zeros, that is, the bias vector b r exist Randomly generated within a range or all zeros.

[0059] The flattened feature tensor M obtained in step S142 flat Input to the fully connected layer, the fully connected layer passes through the weight matrix W R and the bias vector b r Calculate the probability scores of the lung nodule samples in each input image belonging to the five lung nodule malignancy risk levels and generate the classification result vector R. The specific process is expressed as follows: , Among them, R represents the classification result vector, the dimension of the classification result vector R is 5 dimensions, W R Represents the weight matrix updated for each training network iteration, the weight matrix W R The dimensions are 5×C, b r Represents the 5-dimensional bias vector updated for each training network iteration.

[0060] Among them, the five levels of malignancy risk of lung nodules include: benign lung nodules, benign lung nodules with a certain malignant tendency, lung nodules with unclear malignancy, lung nodules with average malignancy, and highly malignant lung nodules. The corresponding category numbers are 1 to 5.

[0061] Step S144, input the classification result vector R obtained in step S143 into the Softmax function, and convert the classification result vector R into a probability distribution vector R1, which is specifically expressed as: , Wherein, e represents a natural constant, R[:,i] represents the predicted value of the i-th category of the classification result vector R, and R1 represents a probability distribution vector with a dimension of 5. The probability distribution vector R1 contains the probability that the lung nodule sample in the input image belongs to each lung nodule malignancy risk level.

[0062] Step S145, compare the probability distribution vector R1 obtained in step S144 with the five lung nodule malignancy risk levels item by item, confirm the category number of the lung nodule malignancy risk level corresponding to the item with the largest probability value in the probability distribution vector R1, and use the category number of the lung nodule malignancy risk level as the final lung nodule risk assessment classification result.

[0063] Step S2, obtain a data set containing lung nodule CT scan images and category number labels of the lung nodule malignancy risk levels corresponding to the images, and train the lung nodule image classification model based on the deep learning network constructed in step S1; after reaching the maximum number of iterations, obtain the model weight parameters of the last round of training; load the model weight parameters obtained after training into the lung nodule image classification model based on the deep learning network, and obtain the trained lung nodule image classification model based on the deep learning network as the final lung nodule risk assessment classification network model for lung nodule CT scan images of the present invention.

[0064] In this embodiment, the existing data set is used to train the lung nodule image classification model based on the deep learning network of the present invention. Figure 5 The number of samples of lung nodules of different risk levels in the LIDC-IDRI dataset used for model training is shown. In the LIDC-IDRI dataset, there are 1018 cases of chest computed tomography images (CT scan images) for diagnosis and lung cancer screening. Each case is labeled by 4 experienced chest radiologists. Each lung CT scan image has at least one lung nodule sample. In the LIDC-IDRI dataset, there are category number labels for the malignant risk level of lung nodules in lung CT scan images marked by radiologists. The category numbers range from 1 to 5, which are used to indicate the suspicious degree of malignant risk of lung nodules. Among them, category number 5 indicates highly malignant lung nodules, category number 4 indicates lung nodules with average malignancy, category number 3 indicates lung nodules with unclear malignancy, category number 2 indicates benign lung nodules with a certain malignant tendency, and category number 1 indicates benign lung nodules.

[0065] In step S3, the CT scan image of the lung nodule to be evaluated is input into the lung nodule image classification model based on the deep learning network trained in step S2 for feature extraction, fusion and classification, so as to obtain the lung nodule risk assessment classification result of the lung nodule CT scan image.

[0066] In this embodiment, the LIDC-IDRI dataset is used to perform a lung nodule risk assessment test. The lung nodule image classification model based on the deep learning network of the present invention is compared with the existing deep learning networks Resnet34, Resnet50, Resnet101, Resnet152, Densenet121, Densenet264 and ViT in terms of data and model attention, and the respective sample accuracy index, sample weighted accuracy index, sample receiver operating characteristic ROC curve area index (AUC), and sample average accuracy index (AP) are calculated. The performance comparison test results are shown in Table 1.

[0067] Table 1 Test results of the deep learning network-based lung nodule image classification model of the present invention and the existing deep learning network benchmark test on the LIDC-IDRI dataset:

[0068] According to the test results in Table 1, it can be seen that the method of the present invention not only has a significant advantage in the overall classification accuracy, but also can show excellent performance in the area under the ROC curve (AUC) and the sample average accuracy (AP), especially in the classification task of high-risk categories, showing extremely high robustness and sensitivity. For example, in the classification results of the malignancy level category numbered 2 and category numbered 3, which are difficult to distinguish and are relatively vague, the method of the present invention greatly improves the accuracy of the model classification and the corresponding average precision, verifying the superiority of the lung nodule image classification model based on the deep learning network of the present invention in the risk assessment of lung nodules.

[0069] Samples were randomly selected from the LIDC-IDRI dataset and classified using the lung nodule image classification model based on deep learning network of the present invention. A heat map was drawn before the fully connected layer of the model to obtain Figure 6 The heat map results shown; Figure 6 The first line in the figure is the original image of the lung nodule CT scan image sample in the LIDC-IDRI dataset. Figure 6 The second line in the figure is a heat map generated by the present invention. The closer the color is to red, the more attention the model pays to the area here, which means that the deep learning network module TMFM pays more attention to the image structure of this part. The heat map generated by the deep learning network-based lung nodule classification model of the present invention and the existing deep learning networks Resnet34, Resnet50, Resnet101, Resnet152, Densenet121, Densenet264 and ViT on the same lung nodule image are compared. The comparison results are shown in Figure 1. Figure 7 As shown in the results, compared with the existing deep learning network, the model of the present invention can not only accurately find the approximate location of lung nodules, but also Figure 7 The pathological image signs of lung nodules such as blood vessels passing through and lung wall adhesion shown in the original image of the CT scan image sample of the middle lung nodule can also be found.

[0070] The above-described embodiment only expresses one implementation of the present invention, but it should not be understood as limiting the scope of the present invention. For those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A lung nodule risk assessment method based on multi-scale, multi-receptive field and multi-attention fusion, characterized in that: The steps include: Step S1, constructing a lung nodule image classification model based on a deep learning network, wherein the lung nodule image classification model based on a deep learning network includes an image preprocessing module, a deep learning network module TMFM, and a classification module; The image preprocessing module is used to perform feature pre-extraction on the input image to obtain the initial feature tensor of the input image; The deep learning network module TMFM includes a feature extraction module and a feature fusion module; The feature extraction module is used to perform multi-scale feature extraction on the feature tensor obtained by the image preprocessing module; The feature fusion module is used to fuse the feature tensors extracted by the feature extraction module; Among them, the feature fusion module includes a spatial attention mechanism SA, a channel attention mechanism CA, a pixel attention mechanism PA and a feature weighted output module; the spatial attention mechanism SA is used to enhance the significant area in the spatial dimension of the feature tensor; the channel attention mechanism CA is used to assign weights on the channel dimension of the feature tensor, adaptively highlight channels with higher relevance, and reduce the influence of invalid channels; the pixel attention mechanism PA is used to refine the feature weights at the pixel level, perform weighted optimization according to the importance of each pixel point, and highlight the features of key pixels; the feature weighted output module is used to uniformly optimize the feature tensors processed and fused by the spatial attention mechanism SA, the channel attention mechanism CA, and the pixel attention mechanism PA, and generate a key feature tensor containing complete multi-scale, multi-receptive field and multi-attention optimization features; The classification module is used to process the key feature tensor and output the pulmonary nodule risk assessment classification result; Step S2, obtaining a data set including CT scan images of lung nodules and category number labels of the malignant risk levels of lung nodules corresponding to the images, training the lung nodule image classification model based on the deep learning network constructed in step S1, and obtaining a trained lung nodule image classification model based on the deep learning network; In step S3, the CT scan image of the lung nodule to be evaluated is input into the lung nodule image classification model based on the deep learning network trained in step S2 for feature extraction, fusion and classification, so as to obtain the lung nodule risk assessment classification result of the lung nodule CT scan image.

2. The pulmonary nodule risk assessment method based on multi-scale, multi-receptive field and multi-attention fusion according to claim 1 is characterized in that: The image preprocessing module includes an input convolution layer and four residual block groups connected in sequence, wherein the input convolution layer includes a convolution layer, a batch normalization layer, an activation layer and a maximum pooling layer; the four residual block groups respectively contain 3, 4, 6 and 3 convolution layers with skip connections.

3. The lung nodule risk assessment method based on multi-scale, multi-receptive field and multi-attention fusion according to claim 2 is characterized in that: The feature extraction module is a multi-branch structure, which includes six parallel branches, namely a convolution operation, three hole convolution operations, an average pooling layer and a wavelet convolution operation. The features output by the six branches are spliced ​​in the channel dimension as the output of the feature extraction module.

4. The lung nodule risk assessment method based on multi-scale multi-receptive field multi-attention fusion according to claim 3 is characterized in that: The feature fusion module extracts features of the spatial dimension and channel dimension of the feature tensor extracted by the feature extraction module through the spatial attention mechanism SA and the channel attention mechanism CA, respectively, to obtain the spatial feature tensor and the channel feature tensor, and adds the spatial feature tensor and the channel feature tensor to obtain the prior feature tensor, and performs weighted optimization on the feature tensor and the prior feature tensor extracted by the feature extraction module through the pixel attention mechanism PA to obtain the pixel feature tensor, and then normalizes the dimension of the pixel feature tensor and fuses it with the spatial feature tensor and the channel feature tensor to obtain the attention fusion feature tensor. The attention fusion feature tensor is then processed through the feature weighted output module to obtain the key feature tensor.

5. The lung nodule risk assessment method based on multi-scale multi-receptive field multi-attention fusion according to claim 4 is characterized in that: The spatial attention mechanism SA includes three parallel spatial pooling modules, a 1×1 convolution operation and a Sigmoid function. The three parallel spatial pooling modules are a spatial average pooling module, a spatial maximum pooling module, and a spatial median pooling module. The outputs of the three spatial pooling modules are concatenated in the channel dimension and input into the 1×1 convolution operation for tensor dimension recovery, and then normalized by the Sigmoid function. The output of the Sigmoid function is multiplied element-by-element with the input of the spatial attention mechanism SA as the output of the spatial attention mechanism SA.

6. The lung nodule risk assessment method based on multi-scale, multi-receptive field, and multi-attention fusion according to claim 5 is characterized in that: The channel attention mechanism CA includes three parallel channel pooling modules and an expand function, wherein the three parallel channel pooling modules are a channel average pooling module, a channel maximum pooling module and a channel median pooling module, respectively; the expand function includes a fully connected layer fc1, a Relu activation function and a fully connected layer fc2 connected in sequence, and the output of the fully connected layer fc2 is dot-multiplied with the input of the channel attention mechanism CA to obtain the output of the expand function; wherein the outputs of the three channel pooling modules are respectively weighted by the expand function and fused with the input of the channel attention mechanism CA to obtain three feature tensors after weight adjustment; and the three feature tensors after weight adjustment are added together to obtain the final output of the channel attention mechanism CA.

7. The lung nodule risk assessment method based on multi-scale, multi-receptive field and multi-attention fusion according to claim 6 is characterized in that: The pixel attention mechanism PA includes a high-dimensional conversion operation, a splicing operation, a rearrangement operation, a convolution operation and a Sigmoid function.

8. The lung nodule risk assessment method based on multi-scale, multi-receptive field, and multi-attention fusion according to claim 7, characterized in that: The classification module includes an average pooling layer, a flattening operation layer and a fully connected layer. The average pooling layer is used to perform feature sorting on the key feature tensor output by the deep learning network module TMFM. The flattening operation layer is used to reshape the feature dimension of the feature tensor to conform to the input dimension of the fully connected layer. The fully connected layer is used to calculate the probability score of the lung nodule samples in the input image belonging to five lung nodule malignant risk levels, generate a classification result vector, and convert the classification result vector into a probability distribution vector. The category number of the lung nodule malignant risk level corresponding to the item with the largest probability value in the probability distribution vector is taken as the lung nodule risk assessment classification result of the lung nodule CT scan image.

Images