Lightweight lung CT image super-resolution method based on multi-scale feature adaptive aggregation

Abstract

The invention relates to the technical field of medical image processing and computer vision, and discloses a lightweight lung CT image super-resolution method based on multi-scale feature adaptive aggregation. The lightweight lung CT image super-resolution method based on multi-scale feature adaptive aggregation comprises the following steps: S1, a shallow feature extraction stage: extracting low-level features in a low-resolution lung CT image through an initial convolutional layer; s2, in a deep learning network MHAG-Net stage, deep features of the lung CT image are extracted, and image detail expression is enhanced; according to the lightweight lung CT image super-resolution method based on multi-scale feature adaptive aggregation, the resolution and detail quality of the reconstructed image can be remarkably improved, a doctor can more accurately evaluate the lung injury degree of a patient and intervene treatment in time, and the method can be widely applied to the field of medical image processing and has a wide application prospect. Particularly, the method has a wide application prospect in the aspects of super-resolution reconstruction, focus detection, fine structure analysis and the like of the lung CT image.

Description

Technical Field

[0001] The present invention relates to the field of medical image processing and computer vision technology, and in particular to a lightweight lung CT image super-resolution method with multi-scale feature adaptive aggregation. Background Art

[0002] In recent years, with the continuous development of medical imaging technology, lung CT (computed tomography) has become one of the important tools for diagnosing respiratory diseases, and CT has been widely used in the detection and evaluation of lung lesions. Lung CT can clearly display typical lung lesion features such as ground-glass opacity (GGO) and solid lesions through X-ray imaging, which provides an important basis for clinicians to assess the degree of lung damage in patients and intervene in treatment in a timely manner. However, in actual diagnosis, the image resolution obtained by traditional CT equipment is limited and cannot fully reflect the detailed information of the lesion. High-resolution (HR) CT images have higher detail display capabilities and can significantly improve the accuracy of doctors' judgment of lesions. However, obtaining high-resolution images usually relies on expensive equipment upgrades and complex scanning technologies, which imposes a large economic burden on some medical institutions.

[0003] Traditional SR algorithms can generally be divided into three categories: reconstruction-based methods, interpolation-based methods, and learning-based methods.

[0004] Reconstruction-based methods use the complementary information of multiple frames of low-resolution images to restore high-resolution images, and achieve image reconstruction by solving the inverse problem of the imaging process. This type of method usually works best when multiple low-resolution observations are required, but its algorithm complexity is high and consumes a lot of computing resources. Interpolation-based methods use interpolation techniques to supplement image details at the pixel level, such as bilinear interpolation and bicubic interpolation. However, interpolation methods are prone to losing details when enlarging images and cannot effectively restore high-frequency information of images.

[0005] In contrast, learning-based methods include SRCNN, VDSR, and ESPCN. These methods can achieve more accurate reconstruction of spatial details by learning image features from large amounts of data. Compared with reconstruction-based and interpolation-based methods, learning-based SR methods have better image reconstruction effects, especially in the recovery of high-frequency details. However, with the increase in image resolution requirements and the increase in the complexity of medical images, existing learning-based methods face some challenges, especially when processing multi-level image information. Existing methods rely on feature extraction at a single scale and fail to fully capture the multi-level information in the image, resulting in insufficient performance in processing complex medical image details.

[0006] In actual medical image analysis, lung CT images are often affected by noise interference and complex texture structures, and existing super-resolution methods are insufficient in detail recovery and image clarity. In addition, the current complex deep learning models consume large computing resources, and the training and inference time is long, which makes it difficult to meet the needs of clinical applications for rapid response.

[0007] Existing super-resolution reconstruction methods are mostly designed for natural images. There are significant differences between lung CT images and natural images, which makes existing methods inadequate in capturing multi-level feature information of CT images and fails to fully utilize the structural information in medical images, thus limiting the effect of super-resolution reconstruction of lung CT images.

[0008] Therefore, it is necessary to provide a lightweight lung CT image super-resolution method with multi-scale feature adaptive aggregation to solve the above technical problems. Summary of the invention

[0009] To solve the above technical problems, the present invention provides a lightweight lung CT image super-resolution method with multi-scale feature adaptive aggregation.

[0010] The lightweight lung CT image super-resolution method with multi-scale feature adaptive aggregation provided by the present invention comprises the following steps:

[0011] S1, shallow feature extraction stage: extract low-level features from low-resolution lung CT images through the initial convolutional layer;

[0012] S2, deep learning network MHAG-Net stage: used to extract deep features of lung CT images and enhance the expression of image details. The network consists of multiple Multi-scale Hybrid Attention Transformer Blocks (MHATBs) and a 1×1 convolutional layer. A residual structure is introduced in the network structure to ensure the stability of training.

[0013] S3, image reconstruction stage: The pixel shuffling algorithm is used to convert the low-resolution feature map into a high-resolution image.

[0014] Preferably, each Multi-scale Hybrid Attention Transformer Block (MHATB) in the MHAG-Net network includes: 2 layer layers, 2 DropPath layers, 1 multi-layer perceptron (MLP) layer, 1 Hybrid Channel Local Window Attention layer, and 1 Multi-scale Feature Adaptive Aggregation layer.

[0015] Preferably, in the Hybrid Channel Local Window Attention layer, a channel attention modulation (CAM) module and a local window attention (LLA) module are introduced to adaptively adjust the importance of different channels and enhance the expression of details in the local space through a local window mechanism.

[0016] Preferably, in the channel attention modulation (CAM) module, the input features are first globally average pooled (AVGPool), and then a one-dimensional convolution layer and a Sigmoid activation function are used to generate channel attention weights. Finally, the channel attention weights are element-wise multiplied with the input feature map; the channel attention modulation (CAM) module is used to learn the interdependence between channels and enhance the feature expression capability by adaptively adjusting the weights of different channels in the feature map.

[0017] Preferably, the local window attention (LLA) module is used to calculate self-attention within a local spatial range, reduce the amount of calculation, and improve calculation efficiency; in the local window attention (LLA) module, the input features are first mapped into three groups of features: query matrix, key matrix and value matrix through a two-dimensional convolution layer; then, LLA divides the input features into non-overlapping windows, and the features within the windows are reshaped into a two-dimensional matrix; finally, the output features of all windows are reorganized and restored to the original spatial dimension, and the number of channels is restored through a projection layer.

[0018] Preferably, the Multi-scale Feature Adaptive Aggregation (MSAS) strategy is used to improve the network's ability to capture multi-level information and aggregate features at different scales, thereby better reconstructing high-resolution lung CT images.

[0019] Preferably, the MSAS module uses a multi-scale structure to capture multi-level information of the image while enhancing the expression of local features; the MSAS attention module consists of three main parts: channel segmentation, multi-scale weighted unit (MSWU) and channel attention; the MSAS strategy includes:

[0020] The input feature map is split in the channel dimension to generate multiple sub-feature map sets;

[0021] Perform deep convolution on each sub-feature map to capture features of different scales;

[0022] Generate feature maps of different scales through adaptive average pooling, spatial weighting and upsampling operations;

[0023] Feature fusion is performed using learnable weighting coefficients and concatenation is performed in the channel dimension.

[0024] Preferably, the image reconstruction stage realizes the improvement of resolution through a pixel shuffling algorithm, and the specific process is as follows:

[0025] The extracted low-resolution feature map is subjected to a pixel shuffle operation to rearrange the information in the channel dimension into the spatial dimension;

[0026] Realize high-resolution image reconstruction.

[0027] Compared with the related art, the lightweight lung CT image super-resolution method with multi-scale feature adaptive aggregation provided by the present invention has the following beneficial effects:

[0028] The present invention proposes a MHAG-NET deep learning network model, which integrates a hybrid attention mechanism and a multi-scale feature adaptive aggregation strategy. The hybrid attention mechanism can adaptively enhance the expression of local features by combining channel attention and local window attention, reduce the amount of calculation and accurately capture key structural information. The multi-scale feature adaptive aggregation strategy can extract multi-level information of lung CT images, and effectively fuse features of different scales through adaptive weight allocation, making full use of the structural information in medical images, significantly improving the performance of the model in complex texture and detail recovery, and thus improving the quality of super-resolution reconstruction.

[0029] This method can be widely used in the field of medical image processing, especially in super-resolution reconstruction, lesion detection and fine structure analysis of lung CT images, meeting the needs of clinical applications for high-quality image reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A lung CT super-resolution reconstruction flow chart provided by the present invention;

[0031] Figure 2 A schematic diagram of the MHAG-Net network model provided by the present invention;

[0032] Figure 3 A schematic diagram of a CAM module provided by the present invention;

[0033] Figure 4 A schematic diagram of the LLA module provided by the present invention;

[0034] Figure 5 A schematic diagram of the MSAS module provided by the present invention;

[0035] Figure 6 This is a comparison chart of the effects of using different super-resolution reconstruction algorithms provided by the present invention. DETAILED DESCRIPTION

[0036] The present invention will be further described below in conjunction with the accompanying drawings and implementation modes.

[0037] The lightweight lung CT image super-resolution method based on multi-scale feature adaptive aggregation provided by the present invention is as follows: Figure 1 As shown, it is mainly divided into three stages: shallow feature extraction stage, MHAG-Net deep feature extraction stage and image reconstruction stage. For the convenience of explanation, the present invention uses lung CT images to illustrate the whole process. The input low-resolution (LR) lung CT image is extracted by the shallow convolution layer to extract the initial features, and then sent to the MHAG-Net network for deep feature extraction to capture the details and global information of the image. Finally, the super-resolution image is reconstructed in combination with the pixel shuffling algorithm.

[0038] The three stages will be explained in detail below.

[0039] 1. Shallow feature extraction stage

[0040] At the initial stage of the super-resolution reconstruction process, shallow feature extraction is required for the input low-resolution (LR) image. An initial convolutional layer is used to extract low-level features (such as edges, textures, and local structures) in the low-resolution image. This process keeps the spatial dimension of the image unchanged, making it easier for MHAG-Net to perform more complex deep feature extraction.

[0041] 2. MHAG-Net deep feature extraction stage

[0042] The MHAG-Net deep learning network is mainly used to extract deep features of lung CT images and enhance the detail expression of images in super-resolution reconstruction. The MHAG-Net network structure mainly consists of multiple Multi-scale HybridAttentionTransformer Blocks (MHATBs) and a 1×1 convolutional layer ( Figure 2), each Multi-scaleHybridAttention Transformer Block is mainly composed of 2 layers, 2 DropPath layers, 1 multi-layer perceptron (MLP) layer, 1 Hybrid Channel LocalWindowAttention layer, and 1 Multi-scaleFeatureAdaptive Aggregation layer. In order to ensure the stability of training, a residual structure is introduced in the network structure. In the Hybrid Channel Local WindowAttention layer, the channel attention modulation (CAM) and local window attention (LLA) modules are introduced to adaptively adjust the importance of different channels and enhance the expression of details in the local space through the local window mechanism. The Multi-scale Feature Adaptive Aggregation Strategy (MSAS) strategy is introduced to improve the network's ability to capture multi-level information and aggregate features at different scales, so as to better reconstruct high-resolution lung CT images.

[0043] (a) Channel Attention Modulation (CAM) module

[0044] CAM is a channel attention mechanism module (such as Figure 3 ), which is used to learn the interdependence between channels and enhance the feature expression ability by adaptively adjusting the weights of different channels in the feature map. In the channel attention module, the input features are first globally averaged pooled (AVGPool), and then a one-dimensional convolution layer and a Sigmoid activation function are used to generate channel attention weights. Finally, the channel attention weights are multiplied element by element with the input feature map. The calculation process is summarized as follows:

[0045] X'=Sigmoid(Conv(Avgpool(X)))⊙X

[0046] Among them, X and X' represent the input and output of the channel attention module respectively, Sigmoid() represents the Sigmoid function, Conv() represents the one-dimensional convolution, Avgpool() represents the average pooling operation, and ⊙ is the element-wise product.

[0047] (b) LLA attention module

[0048] LLA is a local window attention mechanism module (such as Figure 4), calculate self-attention in the local space. The advantage of LLA is that it introduces channel dimensionality reduction operations, reduces the amount of calculation, and optimizes the local window division and calculation strategy to ensure the aggregation effect of local features and reduce the consumption of hardware resources, so that the calculation efficiency is significantly improved while maintaining efficient feature expression. First, the input features are mapped into three sets of features: query matrix, key matrix and value matrix (represented as Q, K and V respectively) through a two-dimensional convolutional layer. This process is defined as:

[0049] Q=X'W Q ,K=X'W K ,V=X'W V

[0050] Among them, X' represents the input of the local window attention module, W Q ,W k ,W v ∈R C×(C / r) is the weight matrix used to generate queries, keys, and values.

[0051] Subsequently, LLA divides the input features into non-overlapping windows, each of which is M×M in size, and the features within the window are reshaped into a two-dimensional matrix Q s ,K s ,V s ∈R M2×(C / r) Finally, the output features of all windows are reorganized to restore the original spatial dimensions, and the number of channels is restored through the projection layer, where the output of the local window attention module can be expressed as:

[0052]

[0053] Where LLA(X) represents the output of the local window attention module, SoftMax() is the SoftMax function, d is the dimension of the key, and W p ∈R (C / r)×C is the linear projection matrix used to map the features back to the original channel dimension.

[0054] (c) MSAS module

[0055] The MSAS module is an attention mechanism module that uses a multi-scale structure to capture multi-level information of an image while enhancing the expression of local features. The MSAS attention module consists of three main parts: channel segmentation, multi-scale weighted unit (MSWU), and channel attention (e.g. Figure 5 ).

[0056] First, the input feature map is split in the channel dimension to generate four sub-feature map sets {X0, X1, X2, X3}. Each sub-feature map is processed by deep convolution DW-Conv 3×3() is processed to capture features of different scales. Then, after adaptive average pooling, spatial weighting and upsampling operations, feature maps of different scales are generated. Then, these feature maps are fused through learnable weighting coefficients and spliced ​​in the channel dimension. The calculation process can be expressed as:

[0057]

[0058]

[0059] Avgpool() represents the pooling of the input feature map, DW-Conv 3×3 is a 1×1 depthwise convolution,↑ b () indicates that the upsampled features at a specific level are raised to the initial resolution through bilinear interpolation. Among them, Cat() represents the concatenation operation. Then, channel attention is applied (after 2 1×1 convolution layers, an average pooling operation, and a ReLU activation function) to obtain the channel attention weights, and finally, the extracted features are aggregated using 1×1 convolution, and then normalized through a Swish activation function. The final output of MSAS can be expressed as:

[0060]

[0061]

[0062] in represents the channel attention weight, Sigmoid() represents the Sigmoid activation function, ReLU() represents the ReLU activation function, Represents the aggregated features, Conv 1×1 is a 1×1 convolution, is the final output of MSAS, SiLU() represents the SiLU activation function, and ⊙ is the element product.

[0063] 3. Image reconstruction stage

[0064] The image reconstruction process uses the pixel shuffling algorithm to convert low-resolution feature maps into high-resolution images. First, the extracted low-resolution feature maps are subjected to a pixel shuffling operation through a deep learning network to rearrange the information in the channel dimension to the spatial dimension to achieve resolution improvement. The pixel shuffling algorithm improves the clarity and detail expression of the image by reallocating the position of each pixel. The entire image reconstruction process can be expressed as:

[0065] I HR =PixelShuffle(I HR )

[0066] Among them I HRrepresents the reconstructed high-resolution image, I HR Represents the input low-resolution feature map, and PixelShuffle() represents the pixel shuffle operation.

[0067] Below, the present invention takes the super-resolution reconstruction of medical lung CT images as an example. In the experiment, 600 COVID-19 CT images were used to train the model, 100 images were used for verification, and 46 images were used for testing. The algorithm performance was evaluated by the peak signal-to-noise ratio (PSNR) and the structural similarity index (SSIM). We used the MHAG-Net algorithm to compare with Bicubic, SRCNN, ESPCN, VDSR, IMDN, PAN, DRIDSR, MFHAN and other algorithms at magnification factors of ×3 and ×4. The algorithm comparison results are shown in Table 1. In addition, we visualized the algorithm results, as shown in Table 1. Figure 6 As shown in the figure, the experimental results show that the MHAG-Net algorithm in this embodiment has the best super-resolution reconstruction performance.

[0068] Table 1 shows the comparison of the effects of using different super-resolution reconstruction algorithms under a magnification factor of ×3

[0069]

[0070]

[0071] Table 2 shows the comparison of the effects of using different super-resolution reconstruction algorithms under a magnification factor of ×4.

[0072]

[0073] Compared with the related art, the lightweight lung CT image super-resolution method with multi-scale feature adaptive aggregation provided by the present invention has the following beneficial effects:

[0074] The present invention proposes a MHAG-NET deep learning network model, which integrates a hybrid attention mechanism and a multi-scale feature adaptive aggregation strategy. The hybrid attention mechanism can adaptively enhance the expression of local features by combining channel attention and local window attention, reduce the amount of calculation and accurately capture key structural information. The multi-scale feature adaptive aggregation strategy can extract multi-level information of lung CT images, and effectively fuse features of different scales through adaptive weight allocation, making full use of the structural information in medical images, significantly improving the performance of the model in complex texture and detail recovery, and thus improving the quality of super-resolution reconstruction.

[0075] This method can be widely used in the field of medical image processing, especially in super-resolution reconstruction, lesion detection and fine structure analysis of lung CT images, meeting the needs of clinical applications for high-quality image reconstruction.

[0076] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A lightweight lung CT image super-resolution method with multi-scale feature adaptive aggregation, which mainly includes the following steps: S1, shallow feature extraction stage: extract low-level features from low-resolution lung CT images through the initial convolutional layer; S2, deep learning network MHAG-Net stage: used to extract deep features of lung CT images and enhance the expression of image details. The network consists of multiple Multi-scale Hybrid Attention TransformerBlocks and a 1×1 convolutional layer; a residual structure is introduced in the network structure to ensure the stability of training. S3, image reconstruction stage: The pixel shuffling algorithm is used to convert the low-resolution feature map into a high-resolution image.

2. The lightweight lung CT image super-resolution method with multi-scale feature adaptive aggregation as claimed in claim 1, characterized in that: Each Multi-scale Hybrid AttentionTransformerBlock in the MHAG-Net network includes: 2 layer layers, 2 DropPath layers, 1 multi-layer perceptron layer, 1 HybridChannel Local Window Attention layer, and 1 Multi-scale Feature Adaptive Aggregation layer.

3. The lightweight lung CT image super-resolution method with multi-scale feature adaptive aggregation as claimed in claim 2, characterized in that: In the Hybrid Channel Local Window Attention layer, a channel attention modulation module and a local window attention module are introduced to adaptively adjust the importance of different channels and enhance the expression of details in the local space through the local window mechanism.

4. The lightweight lung CT image super-resolution method with multi-scale feature adaptive aggregation as claimed in claim 3, characterized in that: The channel attention modulation module first performs global average pooling on the input features, uses a one-dimensional convolutional layer and a Sigmoid activation function to generate channel attention weights, and finally multiplies the channel attention weights by the input feature map element by element; the channel attention modulation module aims to learn the interdependence between channels and enhance the feature expression capability by adaptively adjusting the weights of different channels in the feature map.

5. The lightweight lung CT image super-resolution method with multi-scale feature adaptive aggregation as claimed in claim 3, characterized in that: The local window attention module is used to calculate self-attention within a local spatial range, reduce the amount of calculation and improve calculation efficiency; the local window attention module firstly inputs features through a two-dimensional convolution layer to be mapped into three sets of features: a query matrix, a key matrix and a value matrix; then, the local window attention module divides the input features into non-overlapping windows, and the features within the windows are reshaped into a two-dimensional matrix; finally, the output features of all windows are reorganized and restored to the original spatial dimension, and the number of channels is restored through a projection layer.

6. The lightweight lung CT image super-resolution method with multi-scale feature adaptive aggregation as claimed in claim 2, characterized in that: Multi-scale Feature Adaptive Aggregation (MSAS) is introduced to improve the network's ability to capture multi-level information and aggregate features at different scales, thereby better reconstructing high-resolution lung CT images.

7. The lightweight lung CT image super-resolution method with multi-scale feature adaptive aggregation as claimed in claim 6, characterized in that: The MSAS module uses a multi-scale structure to capture multi-level information of the image while enhancing the expression of local features; MSAS consists of three main parts: channel segmentation, multi-scale weighted unit and channel attention; the MSAS includes: The input feature map is split in the channel dimension to generate multiple sub-feature map sets; Perform deep convolution on each sub-feature map to capture features of different scales; Generate feature maps of different scales through adaptive average pooling, spatial weighting and upsampling operations; Feature fusion is performed using learnable weighting coefficients and concatenation is performed in the channel dimension.

8. The lightweight lung CT image super-resolution method with multi-scale feature adaptive aggregation as claimed in claim 1, characterized in that: The image reconstruction stage achieves resolution improvement through a pixel shuffling algorithm, and the specific process is as follows: The extracted low-resolution feature map is subjected to a pixel shuffle operation to rearrange the information in the channel dimension into the spatial dimension; Realize high-resolution image reconstruction.

Images