An enhanced Swin Transformer super-resolution reconstruction method for industrial CT images

Abstract

The present invention provides an enhanced Swin Transformer super-resolution reconstruction method for industrial CT images, belonging to the fields of computer image processing and industrial nondestructive testing. This method addresses the problem of incomplete recovery of detail information during super-resolution reconstruction of industrial CT images. By combining the advantages of the Swin Transformer with those of a CNN, the method more effectively acquires global and local feature information of the input image through a window-based multi-head self-attention mechanism and a multidimensional attention convolution module. The introduced detail enhancement convolution module combines traditional convolution with differential convolution, which helps improve the network's ability to extract detail features and its generalization. This method is suitable for super-resolution reconstruction of CT images of arbitrarily complex parts, with reliable resolution improvement and strong adaptability.

Description

Technical Field

[0001] The present invention belongs to the fields of computer image processing and industrial non-destructive testing, and in particular relates to an enhanced Swin Transformer super-resolution reconstruction method for industrial CT images. Background Art

[0002] Industrial computed tomography (CT) technology is an advanced non-destructive testing method that integrates multidisciplinary technologies such as radiology, computer science, optics, electronics, and image processing. The industrial CT system is mainly composed of a radiation source, a mechanical motion mechanism, a radiation detector, and a computer data processing unit. During the inspection, X-rays penetrate the object to be tested and generate projection data, which is then processed by computer software to obtain a CT image of the object to be tested. For parts with complex internal structures, industrial CT technology has demonstrated excellent detection performance. However, due to the limitations of the equipment and the influence of factors such as external interference, CT images often have problems such as blurring, lack of details, and unclear edges. In particular, when performing CT scanning on large-scale components, due to the relatively small imaging magnification, the resolution of the reconstructed CT image is not high, resulting in the loss of some detailed information in the CT image.

[0003] One approach to improving the resolution of industrial CT images is to upgrade the hardware, such as using a X-ray source with a smaller focal spot and a detector with a higher pixel density. However, both X-ray sources and detectors currently face both physical and process limitations, making upgrades expensive. Especially when imaging large components, resolution improvements are still constrained by physical limitations. Image super-resolution reconstruction technology uses software algorithms to restore one or more low-resolution images of a given scene into one or more corresponding high-resolution, clear images. This technology offers a new approach to addressing insufficient image resolution, breaking through hardware limitations and significantly reducing equipment costs.

[0004] Classic image super-resolution reconstruction methods include interpolation-based, reconstruction-based, and learning-based methods. Interpolation-based methods, such as nearest neighbor interpolation, bilinear interpolation, and bicubic interpolation, are simple to implement and can improve the quality of reconstructed images to a certain extent, but they can easily blur edges and details. These methods are often used as a preprocessing step for other super-resolution reconstruction methods. In contrast, reconstruction-based methods, such as iterative backprojection, convex set projection, and maximum a posteriori probability, focus more on image degradation and can preserve texture details. However, due to their high computational load and poor image quality, they are not suitable for practical applications. Learning-based methods learn from a pre-constructed library of low-resolution and high-resolution image samples to find a one-to-one mapping from low-resolution to high-resolution images, thereby guiding image super-resolution reconstruction. Traditional image super-resolution reconstruction algorithms rely on artificially defined prior information to extract low-level image features with limited expressive power. However, they struggle to achieve satisfactory reconstruction results when the magnification factor increases. With the rapid development of deep learning in recent years, an increasing number of researchers have actively explored the use of deep learning for image super-resolution reconstruction. Compared with traditional methods, deep learning methods can obtain more expressive image features through training on datasets. They can adaptively learn the mapping relationship between low-resolution and high-resolution images, reconstructing more realistic images. GPUs can also be used for faster results. Currently, a variety of deep learning algorithms have been applied to the field of image super-resolution reconstruction, such as Convolutional Neural Networks (CNN), Residual Networks (ResNet), Generative Adversarial Networks (GAN), and Transformers, and have achieved excellent results on multiple public datasets.

[0005] Industrial CT images typically contain rich texture details and semantic information, as well as numerous fine structures. Selecting a super-resolution reconstruction network suitable for industrial CT images is crucial for improving CT image clarity. Existing deep learning-based image super-resolution techniques mostly extract deep image features by stacking convolutional layers to increase network depth. However, this approach often overlooks the connections between feature information at different levels, making it difficult to capture the global image information. In industrial CT images in particular, it is difficult to extract sufficient deep features from regions of interest, resulting in incomplete recovery of some image information. Summary of the Invention

[0006] The purpose of the present invention is to provide an enhanced Swin Transformer super-resolution reconstruction method based on the characteristics of industrial CT images. By combining the traditional CNN and Transformer structures, the Transformer is used to capture global information, while the CNN focuses on extracting local information, giving full play to the respective advantages of the two to obtain more comprehensive feature information. The method is suitable for super-resolution reconstruction of CT images of arbitrarily complex parts, with reliable resolution improvement and strong adaptability.

[0007] The technical solution adopted by the present invention to solve the technical problem includes the following steps:

[0008] Step 1: Obtain a high-resolution CT image dataset of parts that are the same or similar to the parts to be super-resolution reconstructed;

[0009] Step 2: Construct an industrial CT image degradation model;

[0010] Step 3: Obtain or select a blur kernel dataset and a noise dataset, and apply them to the industrial CT image degradation model to degrade the high-resolution CT image dataset to obtain a paired low-resolution CT image dataset;

[0011] Step 4: Construct an enhanced Swin Transformer network for super-resolution reconstruction of industrial CT images;

[0012] Step 5: Train and test the constructed enhanced Swin Transformer super-resolution reconstruction network;

[0013] Step 6: Use the trained enhanced Swin Transformer super-resolution reconstruction network to perform super-resolution reconstruction on the industrial CT image of the part to be super-resolution reconstructed to obtain its high-resolution CT image.

[0014] Furthermore, there are three methods for obtaining a high-resolution CT image dataset in step 1: (1) using industrial CT simulation software to simulate and reconstruct the part design model to obtain a simulated high-resolution CT image dataset; (2) using actual industrial CT to perform high-resolution scanning on the actual part and reconstruct the actual high-resolution CT image dataset; (3) combining the simulated high-resolution CT image dataset and the actual high-resolution CT image dataset in a certain proportion to form a mixed high-resolution CT image dataset. You can choose one of these methods as needed.

[0015] Furthermore, the specific steps for constructing the industrial CT image degradation model in step 2 are:

[0016] The industrial CT image degradation model is expressed as: First, the high-resolution CT image IHR Convolve with the blur kernel k, then downsample the convolution result to a scale of s, and finally add a certain degree of noise n to obtain a low-quality, low-resolution CT image I LR .

[0017] Furthermore, the specific method for obtaining the blur kernel dataset in step 3 is:

[0018] To obtain a blur kernel k that more closely resembles the degradation of actual CT images, an unsupervised blur kernel estimation network was constructed. This network consists of a generator and a discriminator, both of which employ multi-layer convolutional layers. A real high-resolution CT image is input into the network, then downsampled to a certain multiple by the kernel estimation generator. Image patches of the same size are then cropped from the original high-resolution CT image. The discriminator determines whether the pixel distributions of the downsampled and cropped images are consistent. After training, the degradation function layer of the generator network is extracted as the blur kernel k. A blur kernel is obtained for each real high-resolution CT image, and a blur kernel dataset is available from a dataset of real high-resolution CT images. When the image is degraded, a blur kernel is randomly selected from the blur kernel dataset to downsample the high-resolution CT image.

[0019] Furthermore, the specific method for obtaining the noise dataset in step 3 is:

[0020] To make the CT images after adding noise n appear more natural, a noise dataset is collected from an actual high-resolution CT image dataset. The size, maximum variance, and minimum mean of the noise blocks are first set. During the acquisition process, if the variance of the noise blocks in the actual high-resolution CT image falls within the set variance range, they are included in the noise dataset. When the image degrades, noise blocks are randomly selected from the noise dataset and added to the downsampled image to complete the noise injection.

[0021] Both blur kernel and noise datasets can be saved as public blur kernel and noise datasets, using device, part, and scanning parameter features as identifiers. When the features of the part to be super-resolution reconstructed are similar to those in these two public datasets, they can be directly selected and applied to the industrial CT image degradation model.

[0022] Furthermore, the specific steps of constructing the enhanced SwinTransformer network for industrial CT image super-resolution reconstruction in step 4 are as follows:

[0023] First, a shallow feature extraction module consisting of one layer of 3×3 convolution is used to extract shallow features of the input low-resolution CT image. The extracted feature map is then sent to the deep feature extraction module to further extract deep features. Finally, the extracted shallow features and deep features are fused through a long skip connection and upsampled to obtain a high-resolution CT image.

[0024] The deep feature extraction module contains 6 enhanced Swin Transformer groups and 1 3×3 convolutional layer. Each Swin Transformer group consists of 6 enhanced Swin Transformer blocks, 1 detail enhancement convolutional module, and 1 3×3 convolutional layer with residual connections.

[0025] The enhanced Swin Transformer block adds a multi-dimensional attention convolution module in parallel with the window-based multi-head self-attention module on the basis of the standard Swin Transformer block to improve the representation ability of the network. The multi-dimensional attention convolution module includes two convolutional layers, with a GELU (Gaussian Error Linear Unit) activation function between the two convolutional layers, and a multi-dimensional collaborative attention module after the second convolutional layer. The multi-dimensional collaborative attention module consists of three parallel branches. The top branch is responsible for obtaining the interdependence between features in the spatial dimension W, the middle branch is responsible for obtaining the interdependence between features in the spatial dimension H, and the bottom branch is responsible for obtaining the interaction between channels. The outputs of these three branches are averaged and aggregated, and the attention weights of each dimension are used to adjust them to obtain a more refined feature map.

[0026] The detail enhancement convolution module consists of five parallel convolutional layers: one traditional convolutional layer and four differential convolutional layers. The differential convolutional layers are center differential convolutional layers, angular differential convolutional layers, horizontal differential convolutional layers, and vertical differential convolutional layers. Traditional convolutional layers are used to obtain intensity-level information, while differential convolutional layers are used to enhance gradient-level information. Ultimately, the features extracted by all convolutional layers are merged.

[0027] The beneficial effects of the present invention are: (1) The present invention constructs a degradation model based on the characteristics of industrial CT images, which can obtain high-low resolution CT image pairs that are closer to the actual degradation situation, which is beneficial to improving the super-resolution reconstruction quality of actual industrial CT images.

[0028] (2) The industrial CT image enhanced Swin Transformer super-resolution reconstruction network designed in this invention combines the advantages of Swin Transformer and CNN, and more effectively obtains the global and local feature information of the input image through the window-based multi-head self-attention mechanism and multi-dimensional attention convolution module, and the acquired feature information is more comprehensive.

[0029] (3) The present invention introduces a detail enhancement convolution module into the super-resolution reconstruction network, combining traditional convolution and differential convolution, which is beneficial to improving the network's ability to extract detail features and generalization. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 Flowchart of the present invention.

[0031] Figure 2 Schematic diagram of the degradation process of high-resolution CT images.

[0032] Figure 3 Schematic diagram of the enhanced Swin Transformer super-resolution reconstruction network structure.

[0033] Figure 4 Schematic diagram of the multi-dimensional attention convolution module.

[0034] Figure 5 Schematic diagram of the detail enhancement convolution module. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and beneficial effects of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0036] like Figure 1 FIG. 1 is a flow chart of the present invention, which shows an enhanced Swin Transformer super-resolution reconstruction method for industrial CT images, comprising the following steps:

[0037] Step 1: The part to be super-resolution reconstructed is an additively manufactured metal part. A mixed high-resolution CT image dataset of parts of the same or similar type is obtained, including: using industrial CT simulation software to simulate and reconstruct eight part design models to obtain a 280×280 simulated high-resolution CT image dataset of 1,800 pieces; using actual industrial CT to perform high-resolution scanning and reconstruct eight actual parts to obtain a 280×280 actual high-resolution CT image dataset of 1,800 pieces.

[0038] Step 2: Construct an industrial CT image degradation model, which is expressed as First, the high-resolution CT image I HRConvolve with the blur kernel k, then downsample the convolution result to a scale of s, and finally add a certain degree of noise n to obtain a low-quality, low-resolution CT image I LR .

[0039] Step 3: Obtain or select a blur kernel dataset and a noise dataset, and apply them to the industrial CT image degradation model to degrade the high-resolution CT image dataset, thereby obtaining a paired low-resolution CT image dataset. Figure 2 As shown, first, the high-resolution CT image dataset (mixed in this embodiment) is downsampled by a factor of two using the bicubic interpolation method to remove the original noise and improve the image clarity. Secondly, blur kernels and noise are randomly selected from the blur kernel dataset and the noise dataset and applied to the industrial CT image degradation model to degrade the bicubic downsampled images. Finally, 3600 pairs of high-low resolution image pairs (140×140, 35×35) are obtained for training the super-resolution reconstruction network.

[0040] In step 3, to obtain a blur kernel k that more closely resembles the degradation of actual CT images, an unsupervised blur kernel estimation network is constructed. This network consists of a generator and a discriminator, both of which employ multi-layer convolutional layers. After inputting an actual high-resolution CT image into the network, the kernel estimation generator downsamples the image to a certain multiple. Simultaneously, image patches of the same size are cropped from the original high-resolution CT image. The discriminator determines whether the pixel distributions of the downsampled and cropped images are consistent. After training, the degradation function layer of the generator network is extracted as the blur kernel k. A blur kernel is obtained for each actual high-resolution CT image. A blur kernel dataset can be obtained from an actual high-resolution CT image dataset.

[0041] In step 3, to make the CT images after adding noise n appear more natural, a noise dataset is collected from an actual high-resolution CT image dataset. The noise blocks are first set to 35×35 in size, with a maximum variance of 10 and a minimum mean of 0. During the acquisition process, if the variance of the noise blocks in the actual high-resolution CT images falls within the set variance range, they are included in the noise dataset. When the image degrades, noise blocks are randomly selected from the noise dataset and added to the downsampled image to complete the noise injection.

[0042] In step 3, the blur kernel dataset and noise dataset are saved as a public blur kernel dataset and a public noise dataset based on their characteristic identifiers, such as equipment, parts, and scanning parameters. If the characteristic identifiers of the parts to be super-resolution reconstructed in other applications are similar to those in these two public datasets, they can be directly selected and applied to the industrial CT image degradation model.

[0043] Step 4: Construct an enhanced Swin Transformer network for super-resolution reconstruction of industrial CT images. Figure 3 As shown in Figure 3, the overall network structure consists of a shallow feature extraction module, a deep feature extraction module, and an upsampling module. In the shallow feature extraction module, a single 3×3 convolution layer is used to extract shallow features from the input low-resolution image. The extracted feature map is then fed into a deep feature extraction module consisting of six enhanced Swin Transformer groups (ESTGs) and one 3×3 convolutional layer to further extract deep features. In the upsampling module, the extracted shallow and deep features are fused via long skip connections, and upsampling is performed using the pixel-shuffle method to obtain high-resolution CT images.

[0044] Each enhanced Swin Transformer group (ESTG) in the deep feature extraction module consists of six enhanced Swin Transformer blocks (ESTB), one detail enhancement convolutional block (DECB), and one 3×3 convolutional layer with residual connections. ESTB builds on the standard Swin Transformer block by adding a multi-dimensional attention convolutional block (MACB) in parallel with a window-based multi-head self-attention (W-MSA) block. Within each successive ESTB, a shifted window-based multi-head self-attention (SW-MSA) block is applied periodically. To avoid conflicts between the MACB and W-MSA optimizations, a weight α is assigned to the MACB output. The ESTB processing can be expressed as follows:

[0045] X′=LN(X in )

[0046] X″=(S)W-MSA(X′)+αMACB(X′)+X in

[0047] X out =MLP(LN(X″))+X″

[0048] Among them, X′ and X″ are intermediate features, X in and X out is the input and output of ESTB. MLP is a multi-layer perceptron used to calculate the self-attention module. For the input features of H×W×C, first divide the input into HW / M 2 A window of size M×M is used to adjust the input to (HW / M 2 )×M 2 ×C features, and then calculate the self-attention of each window separately. The query matrix (Q), key matrix (K), and value matrix (V) are calculated through linear mapping, and the window-based self-attention can be expressed as follows:

[0049]

[0050] Where d represents the dimension of the query and key, and B is the relative position code. In addition, a shift window partitioning method is used to establish connections between adjacent non-overlapping windows, and the shift size is set to half the window size.

[0051] like Figure 4 As shown in the figure, the multi-dimensional attention convolution module MACB in the enhanced Swin Transformer block ESTB consists of two convolutional layers with a GELU activation function between the two layers. The second convolutional layer is followed by a multi-dimensional collaborative attention module. This attention module consists of three parallel branches: the top branch is responsible for capturing the interdependencies between features in the spatial dimension W, the middle branch is responsible for capturing the interdependencies between features in the spatial dimension H, and the bottom branch is responsible for capturing the interactions between channels. The outputs of the three branches are averaged and aggregated and adjusted according to the attention weights of each dimension to obtain more refined feature maps. The addition of the multi-dimensional attention convolution module MACB enables the network to obtain better visual representations.

[0052] like Figure 5 As shown in the figure, the detail enhancement convolution module DECB in the enhanced Swin Transformer group ESTG includes five parallel convolution layers, one traditional convolution layer VC and four differential convolution layers. The differential convolution layers are: center differential convolution layer CDC, angle differential convolution layer ADC, horizontal differential convolution layer HDC and vertical differential convolution layer VDC. Traditional convolution is used to obtain intensity level information, differential convolution enhances gradient level information, and finally the features extracted by all convolution layers are merged. On this basis, in order to avoid an increase in parameters, the additivity of convolution is used to make the convolution module equivalent to a standard convolution, that is, the convolution kernels are added at the corresponding positions to obtain an equivalent kernel. The use of the detail enhancement convolution module DECB enhances the network's ability to extract detail features.

[0053] Step 5: Train and test the constructed enhanced Swin Transformer super-resolution reconstruction network. The super-resolution reconstruction network model constructed in Step 4 was trained using the paired industrial CT image dataset obtained in Step 3. The batch size was set to 8, the weight α of the MACB module output was set to 0.01, and the Adam optimizer was used with two exponential decay rates set to β1 = 0.9 and β2 = 0.99, respectively. The initial learning rate was set to 0.0002, and 300,000 iterations were performed. During training, the L1 loss function was gradually reduced, ensuring that the generated high-resolution CT images were similar to the corresponding real high-resolution CT images.

[0054] Step 6: Using the trained enhanced Swin Transformer super-resolution reconstruction network, super-resolution reconstruction is performed on the industrial CT image of an additively manufactured metal part to be super-resolution reconstructed to obtain its high-resolution CT image, verifying the effectiveness of the present invention.

Claims

1. An enhanced Swin Transformer super-resolution reconstruction method for industrial CT images, characterized by The following steps are involved: Step 1: Obtain a high-resolution CT image dataset of parts that are the same or similar to the parts to be super-resolution reconstructed; Step 2: Build an industrial CT image degradation model The meaning is to first convert the high-resolution CT image I HR Convolve with the blur kernel k, then downsample the convolution result to a scale of s, and finally add a certain degree of noise n to obtain a low-quality, low-resolution CT image I LR ; Step 3: Obtain or select a blur kernel dataset and a noise dataset, and apply them to the industrial CT image degradation model to degrade the high-resolution CT image dataset, thereby obtaining a paired low-resolution CT image dataset; Step 4: Construct an enhanced Swin Transformer network for super-resolution reconstruction of industrial CT images. The overall network structure includes a shallow feature extraction module, a deep feature extraction module, and an upsampling module. The shallow feature extraction module, which consists of a single layer of 3×3 convolution, extracts shallow features from the input low-resolution CT image. The extracted feature map is then fed into the deep feature extraction module to further extract deep features. Finally, the extracted shallow and deep features are fused through a long skip connection, and upsampling is performed to obtain a high-resolution CT image. Specifically, the following features are included: (1) The deep feature extraction module consists of 6 enhanced Swin Transformer groups and 1 3×3 convolutional layer. Each enhanced Swin Transformer group consists of 6 enhanced Swin Transformer blocks, 1 detail enhancement convolutional module and 1 3×3 convolutional layer with residual connection; (2) The detail enhancement convolution module consists of five parallel convolutional layers, including one traditional convolutional layer and four differential convolutional layers. The traditional convolutional layer is used to obtain intensity-level information, while the differential convolutional layer is used to enhance gradient-level information. Finally, the features extracted by all convolutional layers are merged. (3) The enhanced Swin Transformer block adds a multi-dimensional attention convolution module in parallel with the window-based multi-head self-attention module on the basis of the standard Swin Transformer block to improve the representation ability of the network. The multi-dimensional attention convolution module consists of two convolutional layers with a GELU activation function between the two convolutional layers. The second convolutional layer is followed by a multi-dimensional collaborative attention module. (4) The multi-dimensional collaborative attention module consists of three parallel branches, which respectively obtain the interdependence between features in different dimensions. Finally, the outputs of the three branches are averaged and aggregated, and the attention weights of each dimension are used to adjust them to obtain a more refined feature map. Step 5: Train and test the constructed enhanced Swin Transformer super-resolution reconstruction network; Step 6: Use the trained enhanced Swin Transformer super-resolution reconstruction network to perform super-resolution reconstruction on the industrial CT image of the part to be super-resolution reconstructed to obtain its high-resolution CT image.

2. The enhanced Swin Transformer super-resolution reconstruction method for industrial CT images according to claim 1, characterized in that: There are three methods for obtaining a high-resolution CT image dataset in step 1. You can select one of the methods according to your needs: (1) Use industrial CT simulation software to simulate the part design model and reconstruct it to obtain a simulated high-resolution CT image dataset; (2) Use actual industrial CT to perform high-resolution scanning on the actual part and reconstruct it to obtain an actual high-resolution CT image dataset; (3) Combine the simulated high-resolution CT image dataset and the actual high-resolution CT image dataset in a certain proportion to form a mixed high-resolution CT image dataset.

3. The enhanced Swin Transformer super-resolution reconstruction method for industrial CT images according to claim 1, characterized in that: The following method is used to obtain the blur kernel dataset in step 3: (1) Construct an unsupervised blur kernel estimation network, which includes a generator and a discriminator. The network structures of the generator and the discriminator are both multi-layer convolutional layer structures; (2) After the actual high-resolution CT image is input into the network, it is downsampled to a certain multiple through the kernel estimation generator. At the same time, image blocks of the same size are cropped from the original high-resolution CT image. The discriminator determines whether the pixel distribution of the downsampled image and the cropped image is consistent; (3) After the training is completed, the degradation function layer of the generator network is extracted as the blur kernel k. One blur kernel is obtained for each actual high-resolution CT image, and a blur kernel dataset can be obtained from the actual high-resolution CT image dataset.

4. The enhanced Swin Transformer super-resolution reconstruction method for industrial CT images according to claim 1, characterized in that: The blur kernel dataset and noise dataset obtained in step 3 are saved as a public blur kernel dataset and a public noise dataset based on the equipment, parts, and scanning parameters. When the feature identifiers of the parts to be super-resolution reconstructed are similar to the feature identifiers of the data in these two public datasets, they can be directly selected and applied to the industrial CT image degradation model.

Images