Macrokernel decomposition attention network for lung CT image super-resolution reconstruction

Abstract

The invention relates to the technical field of medical image processing, and discloses a large kernel decomposition attention network for lung CT image super-resolution reconstruction. The invention provides a novel multi-scale large-kernel decomposition attention (MLDA), and the detail features of various structural tissues in a lung CT image are captured by combining a multi-scale strategy and the large-kernel decomposition attention (LDA). Meanwhile, the LDA avoids block artifacts caused by using expansion convolution in large kernel decomposition. In addition, the MLDA provides three receptive fields with different sizes, so that the MLDA can flexibly cope with multi-granularity feature extraction from micro tissues to large lung anatomical structures. According to the method, the MLDA serves as the Token Mixer to be integrated into the MetaFormer architecture, local lung anatomical structure details can be captured, and the recovery quality of the whole image can be improved. Compared with a traditional convolutional neural network, the combination of the MetaFormer architecture and the MLDA enables the model to model local features and a remote dependency relationship at the same time, and maintains a good calculation amount.

Description

Technical Field

[0001] The present invention relates to the field of medical image processing technology, and in particular to a large kernel decomposition attention network for super-resolution reconstruction of lung CT images. Background Art

[0002] Computed tomography (CT) is an important non-invasive medical imaging technology that is widely used in clinical diagnosis and disease screening, and lung CT has become a major diagnostic tool. However, factors such as the limited accuracy of imaging equipment or patient movement can result in low-resolution (LR) lung CT images. This affects medical staff's acquisition of lung pathology information, resulting in delayed or misdiagnosed diagnosis. Generally speaking, there are two ways to improve the resolution of CT images: one is through more advanced hardware equipment, such as high-performance X-ray sources or high-sensitivity detectors; the other is to directly restore HR images from LR images through computational methods. Although upgrading hardware can directly improve image quality, it is costly. Therefore, computational methods have become an important research direction in the field of medical image processing.

[0003] In order to restore HR images, researchers have proposed a variety of super-resolution (SR) methods, many of which have been applied in the field of medical imaging. Traditional SR methods can be divided into three categories: i) Interpolation-based SR methods, which obtain the pixel values ​​of the HR image by weighted calculation within the pixel neighborhood of the LR image. However, this type of method is simple and efficient, but it cannot restore the lost details and easily leads to image blur. ii) Reconstruction-based SR methods, which restore HR images by explicitly modeling the image degradation process and combining prior knowledge. However, this type of method has difficulty in simulating the complex nonlinear mapping between LR and HR images in dynamic scenes. iii) Shallow learning-based SR methods, which restore HR images by learning the mapping relationship between LR and HR images from a large amount of training data. Although this type of method achieves better SR performance than the first two methods, it still has many limitations, such as limited learning ability for complex nonlinear mapping relationships and difficulty in adaptively adjusting algorithm parameters. Recently, convolutional neural networks (CNNs) have made significant progress in image super-resolution tasks, especially methods such as SRCNN, VDSR, and EDSR, which are superior to traditional methods in restoring HR images. Although these methods are mainly developed for natural images, some researchers have tried to apply deep learning to the SR task of medical images. Compared with CNN-based methods, Transformer-based methods (such as SwinIR, HAT, and CAT) have performed better in image restoration performance. However, such methods have not been widely used in the super-resolution task of lung CT images. Although the Transformer-based architecture has excellent image restoration capabilities, its parameter scale and computational complexity are also greatly increased. For the super-resolution task of lung CT images, the ideal deep learning model is one that can restore the details of the lung structure while remaining lightweight.

[0004] In the prior art, there are still problems that significantly affect the super-resolution performance of lung CT images: First, the tissue structure and lesion areas in lung CT images provide important pathological information, but due to the differences in shape and size of these areas, it is difficult for standard CNN to recover their details, especially when the size of these areas exceeds the limited receptive field of CNN. In addition, the edges of lung tissue structures are usually irregular curves, which are difficult for standard CNN to capture with square filters. Second, traditional CNNs mainly focus on the processing of local information, so it is difficult to capture global information in lung CT images, such as airways, lung parenchyma, blood vessels and their long-range associations with surrounding tissues.

[0005] Therefore, it is necessary to provide a large kernel decomposition attention network for super-resolution reconstruction of lung CT images to solve the above technical problems. Summary of the invention

[0006] To solve the above technical problems, the present invention provides a large kernel decomposition attention network for super-resolution reconstruction of lung CT images.

[0007] The large kernel decomposition attention network for super-resolution reconstruction of lung CT images provided by the present invention includes: a multi-scale large kernel decomposition attention module (MLDAM); a simple gated feedforward module (SGFM); and an anatomically-aware dynamic fusion module (ADFM).

[0008] Preferably, the multi-scale large kernel decomposition attention module (MLDAM) is composed of a multi-scale large kernel decomposition attention (MLDA) and a simple gated feedforward module (SGFM); MLDA combines a multi-scale strategy through a large kernel decomposition attention (LDA), enabling the model to adaptively capture multi-granular features from local details to global anatomical structures.

[0009] Preferably, the simple gated feedforward module (SGFM) optimizes feature expression through a gating mechanism, while significantly reducing the number of network parameters and computational requirements, and introduces a single-layer deep convolution weighted processing feature map, thereby improving the model's spatial feature capture capability and computational efficiency.

[0010] Preferably, the anatomical-aware dynamic fusion module (ADFM) consists of point-by-point convolution and 3×3 deformable convolution, the deformable convolution dynamically adjusts the sampling position of the convolution kernel according to the input features, and the 1×1 convolution is used for pixel-by-pixel feature fusion.

[0011] The present invention also proposes a method for super-resolution reconstruction of lung CT images, comprising the large kernel decomposition attention network for super-resolution reconstruction of lung CT images, and the method further comprises the following steps:

[0012] S1. For the input low-resolution lung CT image, a simple convolutional layer is used to extract shallow features;

[0013] S2, the shallow features are input into a series of stacked multi-scale large kernel decomposition attention blocks (MLDAMs) and an anatomically aware dynamic fusion module (ADFM) to extract deep features;

[0014] S3, after completing the deep feature extraction, the shallow features and the deep features are summed to combine the features;

[0015] S4. Reconstruct the final high-resolution lung CT image through the pixshuffle operation.

[0016] Compared with related technologies, the large kernel decomposition attention network for super-resolution reconstruction of lung CT images provided by the present invention has the following beneficial effects:

[0017] (1) The present invention proposes a novel multi-scale large kernel decomposition attention (MLDA), which captures the detailed features of various structural tissues in lung CT images by combining the large kernel decomposition attention mechanism (LDA) with a multi-scale strategy. At the same time, the LDA of the present invention avoids the block artifacts caused by the use of dilated convolution in large kernel decomposition. In addition, MLDA provides three different sizes of receptive fields, which enables it to flexibly cope with multi-granularity feature extraction from micro-tissues to large lung anatomical structures.

[0018] (2) The present invention integrates MLDA as a Token Mixer into the MetaFormer architecture. This design not only captures the local lung anatomical details, but also improves the overall image restoration quality. Compared with traditional convolutional neural networks, the combination of MetaFormer architecture and MLDA enables the model to simultaneously model local features and long-range dependencies while maintaining a good amount of computation.

[0019] (3) The present invention proposes an anatomically aware dynamic fusion module (ADFM) for restoring irregular edge details in lung CT images. ADFM dynamically adjusts the sampling position of the convolution kernel through deformable convolution and adaptively perceives complex anatomical structures. Combined with point-by-point convolution, ADFM can efficiently fuse multi-scale features to ensure accurate restoration of edge details. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a visual comparison diagram of x3 SR on the COVID-CT test image in the present invention;

[0021] Figure 2 It is a visual comparison diagram of x4 SR on the COVID-CT test image in the present invention;

[0022] Figure 3 It is a visual comparison diagram of x4 SR on the GGO test image in the present invention;

[0023] Figure 4 This is a visual comparison of x3 SR on the Non-COVID-CT test image in the present invention;

[0024] Figure 5 This is a visual comparison of x4 SR on the Non-COVID-CT test image in the present invention;

[0025] Figure 6The architecture diagram of the MLDAN proposed in the present invention, wherein the design of LDA includes cascaded horizontal and vertical one-dimensional deep convolution (DwConv), dilated deep convolution (DwDConv) and 1x1 convolution (PwConv). The expansion rate-aware deep convolution is incorporated into the LDA branch. k represents the maximum receptive field, r represents the dilation rate, and n = (1, 2, 3...). DETAILED DESCRIPTION

[0026] The present invention will be further described below in conjunction with the accompanying drawings and implementation modes.

[0027] A large kernel decomposition attention network for super-resolution reconstruction of lung CT images proposed in the present invention includes three main parts: a multi-scale large kernel decomposition attention module (MLDAM), a simple gated feedforward module (SGFM) and an anatomically aware dynamic fusion module (ADFM). Figure 6 shows the basic architecture of our proposed MLDAN.

[0028] like Figure 6 As shown in (a), given the input low-resolution lung CT image I LR ∈R C×H×W , first use a simple 3×3 convolutional layer to extract shallow features F p ∈R C×H×W . Then, the shallow features are input into a series of stacked multi-scale large kernel decomposition attention blocks (MLDAMs) and an anatomical-aware dynamic fusion module (ADFM) to extract deep features. Each MLDAM contains a LayerNorm (LN) layer, a multi-scale large kernel decomposition attention (MLDA) and a simple gated feedforward module (SGFM). After completing the deep feature extraction, we sum the shallow features with the deep features to combine the features. Finally, the final high-resolution lung CT image is reconstructed through a pixshuffle operation. The whole process can be formulated as:

[0029]

[0030] Among them, f SF (·) represents the 3×3 convolutional layer used for shallow feature extraction. The shallow feature obtained is F O ;f DF (·) represents stacked MLDAMs and ADFM for deep feature extraction; deep features F are extracted DF ;f Rec (·) denotes the reconstruction module implemented by 3×3 convolution and pixshuffle layers to improve the reconstruction efficiency.

[0031] 1. Multi-scale Large-kernel Decomposition Attention Module (MLDAM)

[0032] We use multi-scale large kernel decomposition attention (MLDA) as TokenMixer to model local features and long-range dependencies in lung CT images. Figure 6 As shown in (a), MLDAM consists of two parts: multi-scale large kernel decomposition attention (MLDA) and simple gated feedforward module (SGFM). In particular, we introduce two learnable scaling factors λ1 and λ2 to flexibly capture long-range dependencies of various scales, thereby adaptively extracting image features of different granularities. For a given intermediate feature X, the processing of MLDAM can be formulated as:

[0033]

[0034] where LN(·) denotes layer normalization; MLDA(·) and SGFM(·) are MLDA and SGFM introduced in the following sections, respectively.

[0035] 2. Multi-scale Large Kernel Decomposition Attention (MLDA)

[0036] MLDA combines the large kernel decomposition attention (LDA) with a multi-scale strategy, enabling the model to adaptively capture multi-granular features from local details to global anatomical structures, such as Figure 6 (b) as shown.

[0037] 3. Large Core Decomposition Attention (LDA)

[0038] The K×K convolution is decomposed into five serial components: 1×(2d-1) and (2d-1)×1 depth convolution (DwConv), denoted as f 1-DW (·),f DW-1 (·); 1×[k / r] and [k / r]×1 extended depthwise convolution (DwDConv), denoted as f 1-DWD (·),f DWD-1 (·); 1×1 point-wise convolution (PwConv), denoted as f PW (·), where k represents the maximum receptive field and r represents the dilation rate. However, although this decomposition mode can expand the receptive field and reduce the computational overhead, the potential block effect of the dilated convolution is not conducive to the SR task. Figure 6 As shown in (c), we propose an LDA that is more suitable for SR tasks by introducing a dilation rate-aware (r+n)×(r+n) depth convolution represented as f DW-r (·) to recalibrate the attention map to avoid the influence of blocking effects, where n = (1, 2, 3...). Given an input feature map LDA can be expressed as:

[0039]

[0040] 4. Multi-scale mechanism

[0041] Single-scale LDA is not flexible for the complex structure of lung CT images. We enhance LDA by grouping multi-scale mechanisms and use residual learning to supplement global context information. Figure 6 As shown in (b), the input feature map is firstly expressed as It can be further divided into three groups of features For each set of features X i (where i = 1, 2, 3), we use three different decomposition strategies {k, r} to generate attention maps LDA i Specifically, LDA1 is implemented by {10,2} decomposition, which is suitable for capturing detailed features such as bronchi and small blood vessels; LDA2 is implemented by {21,3} decomposition, which is used to extract features such as blood vessels and lung parenchyma; LDA3 is implemented by {36,4} decomposition, which is suitable for restoring the overall structure of the lung and the global texture of the lesion area. i After the attention maps are concatenated, they are element-wise multiplied with X′ to obtain the weighted feature maps. In addition, we also use a point-by-point convolution to linearly combine the weighted feature maps to better fuse information of different scales. Given an input The whole process can be expressed as:

[0042] X′=f pw (X),[X1,X2,X3]=X′,

[0043]

[0044] Among them, f pw (·) indicates point-by-point convolution, Concat(·) indicates concatenation operation, and LDA i (.) represents LDA with three different decomposition strategies.

[0045] 5. Simple Gated Feedforward Module (SGFM)

[0046] SGFM optimizes feature expression through a gating mechanism, while significantly reducing the number of network parameters and computational requirements. In addition, we introduced a single-layer deep convolution to weight feature maps, improving the model's spatial feature capture capability and computational efficiency. Figure 6 As shown in (d), given the intermediate features The process of SGFM can be expressed as:

[0047] X′=PW(X),[X′1,X′2]=X′,

[0048]

[0049] Where PW(.) represents point-by-point convolution and DW(.) represents depth-wise convolution.

[0050] 6. Anatomy-aware Dynamic Fusion Module (ADFM)

[0051] In many previous SR tasks, ordinary 3×3 convolution is often used in the final stage of deep feature extraction. However, due to its fixed receptive field and convolution sampling mode, it is difficult to adjust adaptively when facing complex and irregular lung structure edges, which will lead to insufficient edge recovery or blurred edges. Figure 6 As shown in (e), we proposed ADFM to improve this situation. This module consists of point-by-point convolution and 3×3 deformable convolution. Among them, the deformable convolution can dynamically adjust the sampling position of the convolution kernel according to the input features, allowing the network to adaptively perceive the subtle features between the complex anatomical structures of the lungs and improve the model's ability to restore irregular edge details. 1×1 convolution is used for pixel-by-pixel feature fusion. By compressing and recombining channel information, it can more efficiently fuse multi-scale information, thereby better balancing global and local features. Experimental results show that ADFM can effectively integrate useful information and improve reconstruction quality.

[0052] Our MLDAN is compared with several state-of-the-art methods on the public COVID-CT dataset, including SRCNN, ESPCN, VDSR, IMDN, PAN, DRIDSR, and MFHAN. Table 1 shows the experimental results of different methods, where “bold” indicates the best results and horizontal lines indicate suboptimal results.

[0053] Table 1. PSNR / SSIM comparison of tested methods.

[0054]

[0055]

[0056] As shown in Table 1, MLDAN shows higher values ​​on PSNR and SSIM, which indicates that the SR results generated by MLDAN are better than other methods and are closer to real images in visual perception. In addition, MLDAN requires the least amount of parameters, indicating that the model obtains excellent results without wasting resources and time.

[0057] To further evaluate the reconstruction performance of MLDAN, we present the visual comparison in Figure 1-5 In. Figure 1 and Figure 4As shown in the figure, when the magnification factor is 3, the lung margins and lung nodules restored by MLDAN are clearer, especially the areas indicated by the red arrows, which helps doctors make accurate diagnoses. Figure 3 The SR image of ground-glass opacity (GGO) at a magnification of 4 is shown in Figure 4. GGO is a typical feature in high-resolution CT images of the lungs, which is manifested as a slight increase in lung CT density. The outlines of bronchi and blood vessels can be clearly seen, similar to frosted glass. Compared with other methods, the GGO generated by MLDAN is clearer in outline and details. Figure 2 and Figure 5 The pathological features in lung CT images can be clearly observed.

[0058] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A large kernel decomposition attention network for super-resolution reconstruction of lung CT images, characterized in that: include: Multi-scale Large Kernel Decomposition Attention Module (MLDAM); Simple Gated Feedforward Module (SGFM); Anatomy-aware Dynamic Fusion Module (ADFM).

2. The large kernel decomposition attention network for super-resolution reconstruction of lung CT images as claimed in claim 1, characterized in that The multi-scale large kernel decomposition attention module (MLDAM) consists of a multi-scale large kernel decomposition attention (MLDA) and a simple gated feedforward module (SGFM); MLDA combines a multi-scale strategy through large kernel decomposition attention (LDA), enabling the model to adaptively capture multi-granular features from local details to global anatomical structures.

3. The large kernel decomposition attention network for super-resolution reconstruction of lung CT images as claimed in claim 1, characterized in that The simple gated feedforward module (SGFM) optimizes feature expression through a gating mechanism, while significantly reducing the number of network parameters and computational requirements, and introduces a single-layer deep convolution weighted processing feature map, thereby improving the model's spatial feature capture capability and computational efficiency.

4. The large kernel decomposition attention network for super-resolution reconstruction of lung CT images as claimed in claim 1, characterized in that The anatomical-aware dynamic fusion module (ADFM) consists of point-by-point convolution and 3×3 deformable convolution. The deformable convolution dynamically adjusts the sampling position of the convolution kernel according to the input features, and the 1×1 convolution is used for pixel-by-pixel feature fusion.

5. A method for super-resolution reconstruction of lung CT images, comprising a large kernel decomposition attention network for super-resolution reconstruction of lung CT images as described in any of claims 1-4, characterized in that: The method further comprises the steps of: S1. For the input low-resolution lung CT image, a simple convolutional layer is used to extract shallow features; S2, the shallow features are input into a series of stacked multi-scale large kernel decomposition attention blocks (MLDAMs) and an anatomically aware dynamic fusion module (ADFM) to extract deep features; S3, after completing the deep feature extraction, the shallow features and the deep features are summed to combine the features; S4. Reconstruct the final high-resolution lung CT image through the pixshuffle operation.

Images