Enhanced Swin Transform super-resolution reconstruction method for industrial CT image

Abstract

The invention provides an enhanced Swin Transformer super-resolution reconstruction method for an industrial CT (Computed Tomography) image, and belongs to the field of computer image processing and industrial nondestructive testing. The invention provides an enhanced Swin Transform super-resolution reconstruction method for solving the problem that detail information recovery is not complete enough during industrial CT image super-resolution reconstruction, the advantages of Swin Transform and CNN are combined, global and local feature information of an input image is more effectively acquired through a multi-head self-attention mechanism based on a window and a multi-dimensional attention convolution module, and the image super-resolution reconstruction method based on the multi-head self-attention mechanism based on the window and the multi-dimensional attention convolution module based on the multi-head self-attention mechanism based on the multi-head self-attention mechanism based on the multi-head self-attention mechanism based on the multi-dimensional attention convolution module is provided. And an introduced detail enhancement convolution module combines traditional convolution and differential convolution, so that the extraction capability and generalization of the network on detail features can be improved. The method is suitable for super-resolution reconstruction of any complex part CT image, the resolution is improved reliably, and the adaptability is high.

Description

Technical Field

[0001] The invention belongs to the field of computer image processing and industrial nondestructive testing, and in particular relates to an enhanced Swin Transformer super-resolution reconstruction method for industrial CT images. Background Art

[0002] Industrial computed tomography (CT) technology is an advanced non-destructive testing method that integrates multidisciplinary technologies such as radiology, computer science, optics, electronic technology and image processing. The industrial CT system is mainly composed of a radiation source, a mechanical motion mechanism, a radiation detector and a computer data processing unit. During the inspection, X-rays penetrate the object to be tested and generate projection data, which is then processed by computer software to obtain a CT image of the object to be tested. For parts with complex internal structures, industrial CT technology has shown excellent detection performance. However, due to the limitations of the equipment and the influence of external interference and other factors, CT images often have problems such as blur, lack of details and unclear edges. Especially when performing CT scanning on large-sized components, due to the relatively small imaging magnification, the resolution of the reconstructed CT image is not high, resulting in the loss of some detail information in the CT image.

[0003] In order to improve the resolution of industrial CT images, one method is to upgrade the hardware, such as using a ray source with a smaller focal size and a detector with a higher pixel density. However, both the ray source and the detector are currently facing dual physical and process limitations, and the cost of upgrading is high. Especially when imaging large-sized components, the improvement in resolution is still constrained by physical limits. Image super-resolution reconstruction technology uses software algorithms to restore one or more low-resolution images in a given scene to corresponding one or more high-resolution clear images. This technology provides a new idea for solving the problem of insufficient image resolution, breaks through hardware limitations, and greatly reduces equipment costs.

[0004] Classic image super-resolution reconstruction methods include interpolation-based methods, reconstruction-based methods, and learning-based methods. Interpolation-based methods such as nearest neighbor interpolation, bilinear interpolation, and bicubic interpolation are simple to implement and can improve the quality of reconstructed images to a certain extent, but they are prone to blurring of image edges and details. This method is usually used as a preprocessing step for other methods in super-resolution reconstruction tasks. In contrast, reconstruction-based methods include iterative back-projection, convex set projection, and maximum a posteriori probability method, which pay more attention to the image degradation process and can maintain good texture details. However, due to the large amount of calculation and general image quality, it is difficult to meet the needs of practical applications. Learning-based methods learn from pre-constructed low-resolution image and high-resolution image sample libraries to find a one-to-one mapping relationship from low-resolution images to high-resolution images, thereby guiding image super-resolution reconstruction. Traditional image super-resolution reconstruction algorithms reconstruct by extracting some low-level image features with limited expression capabilities based on artificially defined prior information. When the magnification factor increases, it is difficult to obtain a relatively ideal reconstruction effect. With the rapid development of deep learning in recent years, more and more scholars have actively explored the use of deep learning for image super-resolution reconstruction. Compared with traditional methods, deep learning methods can obtain more expressive image features through training of data sets, can adaptively learn the mapping relationship between low-resolution and high-resolution images, reconstruct more realistic images, and can also use GPU to accelerate and obtain results faster. Currently, a variety of deep learning algorithms have been applied to the field of image super-resolution reconstruction, such as Convolutional Neural Networks (CNN), Residual Networks (ResNet), Generative Adversarial Networks (GAN), Transformer, etc., and have achieved good results on multiple public data sets.

[0005] Industrial CT images usually contain rich texture details and semantic information, and also contain many small structures. Selecting a super-resolution reconstruction network suitable for industrial CT images is crucial to improving the clarity of CT images. Most of the existing image super-resolution technologies based on deep learning extract deep features of the image by stacking convolutional layers to increase the network depth. However, this method often ignores the connection between feature information at different levels and has difficulty capturing the global information of the image. Especially in industrial CT images, it is difficult to extract sufficient deep features from the region of interest in a targeted manner, resulting in incomplete recovery of some information in the image. Summary of the invention

[0006] The purpose of the present invention is to provide an enhanced Swin Transformer super-resolution reconstruction method based on the characteristics of industrial CT images. By combining the traditional CNN with the Transformer structure, the Transformer is used to capture global information, while the CNN focuses on the extraction of local information, giving full play to the respective advantages of the two, and obtaining more comprehensive feature information. The method is suitable for super-resolution reconstruction of CT images of any complex parts, with reliable resolution improvement and strong adaptability.

[0007] The technical solution adopted by the present invention to solve the technical problem comprises the following steps:

[0008] Step 1: Obtain a high-resolution CT image dataset of parts that are the same or similar to the parts to be super-resolution reconstructed;

[0009] Step 2: Construct an industrial CT image degradation model;

[0010] Step 3: Obtain or select a blur kernel dataset and a noise dataset, and apply them to the industrial CT image degradation model to degrade the high-resolution CT image dataset to obtain a paired low-resolution CT image dataset;

[0011] Step 4: Construct an enhanced Swin Transformer network for super-resolution reconstruction of industrial CT images;

[0012] Step 5: Train and test the constructed enhanced Swin Transformer super-resolution reconstruction network;

[0013] Step 6: Use the trained enhanced Swin Transformer super-resolution reconstruction network to perform super-resolution reconstruction on the industrial CT image of the part to be super-resolution reconstructed to obtain its high-resolution CT image.

[0014] Furthermore, there are three methods for obtaining a high-resolution CT image dataset in step 1: (1) using industrial CT simulation software to simulate and reconstruct the part design model to obtain a simulated high-resolution CT image dataset; (2) using actual industrial CT to perform high-resolution scanning on the actual part and reconstruct the actual high-resolution CT image dataset; (3) combining the simulated high-resolution CT image dataset and the actual high-resolution CT image dataset in a certain proportion into a mixed high-resolution CT image dataset. When using, select one of the methods according to your needs.

[0015] Furthermore, the specific steps of constructing the industrial CT image degradation model in step 2 are:

[0016] The industrial CT image degradation model is expressed as: First, the high-resolution CT image IHR Convolve with blur kernel k, then downsample the convolution result to a scale of s, and finally add a certain degree of noise n to obtain a low-quality, low-resolution CT image I LR .

[0017] Furthermore, the specific method for obtaining the blur kernel data set in step 3 is:

[0018] In order to obtain a blur kernel k that is closer to the actual CT image degradation, an unsupervised blur kernel estimation network is constructed. The network includes a generator and a discriminator. The network structures of the generator and the discriminator are both multi-layer convolutional layer structures. After the actual high-resolution CT image is input into the network, it is downsampled to a certain multiple of the image through the kernel estimation generator. At the same time, the image blocks of the same size are cropped from the original high-resolution CT image. The discriminator is used to determine whether the pixel distribution of the downsampled image and the cropped image is consistent. After training, the degradation function layer of the generator network is extracted as the blur kernel k. One blur kernel is obtained for each actual high-resolution CT image, and a blur kernel dataset can be obtained from the actual high-resolution CT image dataset. When the image is degraded, a blur kernel is randomly selected from the blur kernel dataset to downsample the high-resolution CT image.

[0019] Furthermore, the specific method for obtaining the noise data set in step 3 is:

[0020] In order to make the CT image after adding noise n more natural, a noise data set is collected from the actual high-resolution CT image data set. First, the size, maximum variance and minimum mean of the noise block are set. During the acquisition process, if the variance of the noise block of the actual high-resolution CT image is within the set variance range, it will be included in the noise data set. When the image is degraded, a noise block in the noise data set is randomly selected and added to the downsampled image to complete the noise injection.

[0021] Both the blur kernel dataset and the noise dataset can be saved as public blur kernel dataset and public noise dataset according to the feature identifiers of equipment, parts, and scanning parameters. When the feature identifiers of the parts to be super-resolution reconstructed are similar to the feature identifiers of the data in these two public datasets, they can be directly selected and applied to the industrial CT image degradation model.

[0022] Furthermore, the specific steps of constructing the enhanced SwinTransformer network for super-resolution reconstruction of industrial CT images in step 4 are:

[0023] Firstly, a shallow feature extraction module consisting of one layer of 3×3 convolution is used to extract shallow features of the input low-resolution CT image. Then the extracted feature map is sent to the deep feature extraction module to further extract deep features. Finally, the extracted shallow features and deep features are fused through a long skip connection, and upsampling is performed to obtain a high-resolution CT image.

[0024] The deep feature extraction module contains 6 enhanced Swin Transformer groups and 1 3×3 convolutional layer. Each Swin Transformer group consists of 6 enhanced Swin Transformer blocks, 1 detail enhancement convolutional module and 1 3×3 convolutional layer with residual connection.

[0025] The enhanced Swin Transformer block adds a multi-dimensional attention convolution module in parallel with the window-based multi-head self-attention module on the basis of the standard Swin Transformer block to improve the representation ability of the network. The multi-dimensional attention convolution module includes two convolutional layers, with a GELU (Gaussian Error Linear Unit) activation function between the two convolutional layers, and a multi-dimensional collaborative attention module after the second convolutional layer. The multi-dimensional collaborative attention module consists of three parallel branches. The top branch is responsible for obtaining the interdependence between features on the spatial dimension W, the middle branch is responsible for obtaining the interdependence between features on the spatial dimension H, and the bottom branch is responsible for obtaining the interaction between channels. The outputs of these three branches are averaged and aggregated, and adjusted using the attention weights of each dimension to obtain a more refined feature map.

[0026] The detail enhancement convolution module consists of 5 parallel convolution layers, including 1 traditional convolution layer and 4 differential convolution layers. The differential convolution layers are center differential convolution layer, angle differential convolution layer, horizontal differential convolution layer and vertical differential convolution layer. The traditional convolution layer is used to obtain intensity level information, the differential convolution layer is used to enhance the gradient level information, and finally the features extracted by all convolution layers are merged.

[0027] The beneficial effects of the present invention are: (1) The present invention constructs a degradation model based on the characteristics of industrial CT images, which can obtain high-low resolution CT image pairs that are closer to the actual degradation situation, which is beneficial to improving the super-resolution reconstruction quality of industrial CT actual images.

[0028] (2) The industrial CT image enhanced Swin Transformer super-resolution reconstruction network designed by the present invention combines the advantages of Swin Transformer and CNN, and more effectively obtains the global and local feature information of the input image through the window-based multi-head self-attention mechanism and the multi-dimensional attention convolution module, and the obtained feature information is more comprehensive.

[0029] (3) The present invention introduces a detail enhancement convolution module into the super-resolution reconstruction network, which combines traditional convolution and differential convolution, which is beneficial to improving the network's ability to extract detail features and generalization. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a flow chart of the present invention.

[0031] Figure 2 Schematic diagram of the degradation process of high-resolution CT images.

[0032] Figure 3 Schematic diagram of the enhanced Swin Transformer super-resolution reconstruction network structure.

[0033] Figure 4 Schematic diagram of the multi-dimensional attention convolution module.

[0034] Figure 5 Schematic diagram of the detail enhancement convolution module. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and beneficial effects of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0036] like Figure 1 As shown, it is a flow chart of the present invention, an enhanced Swin Transformer super-resolution reconstruction method of industrial CT images, comprising the following steps:

[0037] Step 1: The part to be super-resolution reconstructed is an additively manufactured metal part, and a mixed high-resolution CT image dataset of the same or similar parts is obtained, including: using industrial CT simulation software to simulate and reconstruct 8 part design models to obtain 1800 280×280 simulated high-resolution CT image datasets; using actual industrial CT to perform high-resolution scanning and reconstruct 8 actual parts to obtain 1800 280×280 actual high-resolution CT image datasets.

[0038] Step 2: Construct an industrial CT image degradation model, which is expressed as First, the high-resolution CT image I HRConvolve with blur kernel k, then downsample the convolution result to a scale of s, and finally add a certain degree of noise n to obtain a low-quality, low-resolution CT image I LR .

[0039] Step 3: Obtain or select a blur kernel dataset and a noise dataset, and apply them to the industrial CT image degradation model to degrade the high-resolution CT image dataset, thereby obtaining a paired low-resolution CT image dataset. Figure 2 As shown, firstly, the high-resolution CT image dataset (mixed in this embodiment) is downsampled twice by the bicubic interpolation method to remove the original noise and improve the clarity of the image; secondly, blur kernels and noise are randomly selected from the blur kernel dataset and the noise dataset and applied to the industrial CT image degradation model to degrade the image after bicubic downsampling; finally, 3600 pairs of high-low resolution image pairs (140×140, 35×35) are obtained for training the super-resolution reconstruction network.

[0040] In step 3, in order to obtain a blur kernel k that is closer to the actual CT image degradation, an unsupervised blur kernel estimation network is constructed. The network includes a generator and a discriminator. The network structures of the generator and the discriminator are both multi-layer convolutional layer structures. After the actual high-resolution CT image is input into the network, it is downsampled to a certain multiple of the image through the kernel estimation generator. At the same time, the image blocks of the same size are cropped from the original high-resolution CT image. The discriminator is used to determine whether the pixel distribution of the downsampled image and the cropped image is consistent. After training, the degradation function layer of the generator network is extracted as the blur kernel k. One blur kernel is obtained for each actual high-resolution CT image, and a blur kernel dataset can be obtained from the actual high-resolution CT image dataset.

[0041] In step 3, in order to make the CT image after adding noise n more natural, a noise data set is collected from the actual high-resolution CT image data set. First, the size of the noise block is set to 35×35, the maximum variance is 10, and the minimum mean is 0. During the acquisition process, if the variance of the actual high-resolution CT image noise block is within the set variance range, it will be included in the noise data set. When the image is degraded, a noise block in the noise data set is randomly selected and added to the downsampled image to complete the noise injection.

[0042] In step 3, the blur kernel dataset and the noise dataset are saved as a public blur kernel dataset and a public noise dataset according to the feature identifiers of equipment, parts, scanning parameters, etc. When the feature identifiers of the parts to be super-resolution reconstructed in other application situations are similar to the data in these two public datasets, they can be directly selected and applied to the industrial CT image degradation model.

[0043] Step 4: Construct an enhanced Swin Transformer network for super-resolution reconstruction of industrial CT images. Figure 3 As shown in the figure, the overall structure of the network includes: shallow feature extraction module, deep feature extraction module and upsampling module. In the shallow feature extraction module, a layer of 3×3 convolution is used to extract shallow features of the input low-resolution image. The extracted feature map is then sent to the deep feature extraction module containing 6 enhanced Swin Transformer groups ESTG and 1 3×3 convolution layer to further extract deep features. In the upsampling module, the extracted shallow features and deep features are fused through long jump connections, and the pixel-shuffle method is used for upsampling to obtain high-resolution CT images.

[0044] Each enhanced Swin Transformer group ESTG in the deep feature extraction module includes 6 enhanced Swin Transformer blocks ESTB, 1 detail enhancement convolution module DECB and 1 3×3 convolution layer with residual connection. ESTB adds a multi-dimensional attention convolution module MACB in parallel with the window-based multi-head self-attention W-MSA module on the basis of the standard Swin Transformer block. In the continuous ESTB, the multi-head self-attention SW-MSA based on the displacement window is used at intervals. At the same time, in order to avoid conflicts between MACB and W-MSA in optimization, a weight α is assigned to the output of MACB. The processing process of ESTB can be expressed as:

[0045] X′=LN(X in )

[0046] X″=(S)W-MSA(X′)+αMACB(X′)+X in

[0047] X out =MLP(LN(X″))+X″

[0048] Among them, X′ and X″ are intermediate features, X in and X out are the input and output of ESTB. MLP is a multi-layer perceptron used to calculate the self-attention module. For the input features of H×W×C, first divide the input into HW / M 2 M×M windows are used to adjust the input to (HW / M 2 )×M 2 ×C features, and then calculate the self-attention of each window separately. The query matrix (Q), key matrix (K), and value matrix (V) are calculated through linear mapping, and the window-based self-attention can be expressed as follows:

[0049]

[0050] Where d represents the dimension of query and key, and B is the relative position code. In addition, the displacement window partitioning method is used to establish the connection between adjacent non-overlapping windows, and the displacement size is set to half of the window size.

[0051] like Figure 4 As shown in the figure, the multi-dimensional attention convolution module MACB in the enhanced Swin Transformer block ESTB includes two convolutional layers, with a GELU activation function between the two layers, and a multi-dimensional collaborative attention module after the second convolutional layer. The attention module consists of three parallel branches. The top branch is responsible for obtaining the interdependence between features in the spatial dimension W, the middle branch is responsible for obtaining the interdependence between features in the spatial dimension H, and the bottom branch is responsible for obtaining the interaction between channels. The outputs of the three branches are averaged and aggregated, and adjusted according to the attention weights of each dimension, so that a more refined feature map can be obtained. The addition of the multi-dimensional attention convolution module MACB can enable the network to obtain better visual representation.

[0052] like Figure 5 As shown in the figure, the detail enhancement convolution module DECB in the enhanced Swin Transformer group ESTG includes five parallel convolution layers, one traditional convolution layer VC and four differential convolution layers. The differential convolution layers are: center differential convolution layer CDC, angle differential convolution layer ADC, horizontal differential convolution layer HDC and vertical differential convolution layer VDC. Traditional convolution is used to obtain intensity level information, differential convolution is used to enhance gradient level information, and finally the features extracted by all convolution layers are merged. On this basis, in order to avoid the increase of parameters, the additivity of convolution is used to make the convolution module equivalent to a standard convolution, that is, each convolution kernel is added at the corresponding position to obtain an equivalent kernel. The use of the detail enhancement convolution module DECB enhances the network's ability to extract detail features.

[0053] Step 5: Train and test the constructed enhanced Swin Transformer super-resolution reconstruction network. The super-resolution reconstruction network model constructed in step 4 is trained using the paired industrial CT image dataset obtained in step 3. The batch size is set to 8, the weight α of the MACB module output is 0.01, and the Adam optimizer is used. The two exponential decay rates of the optimizer are set to β 1 = 0.9 and β 2= 0.99, the initial learning rate was set to 0.0002, and a total of 300,000 iterations were performed. During the training process, the L1 loss function gradually decreased, making the generated high-resolution CT images similar to the corresponding real high-resolution CT images.

[0054] Step 6: Using the trained enhanced Swin Transformer super-resolution reconstruction network, super-resolution reconstruction is performed on the industrial CT image of an additively manufactured metal part to be super-resolution reconstructed to obtain its high-resolution CT image, thereby verifying the effectiveness of the present invention.

Claims

1. An enhanced Swin Transformer super-resolution reconstruction method for industrial CT images, characterized in that The following steps are involved: Step 1: Obtain a high-resolution CT image dataset of parts that are the same or similar to the parts to be super-resolution reconstructed; Step 2: Constructing industrial CT image degradation model This means that the high-resolution CT image I HR Convolve with blur kernel k, then downsample the convolution result to a scale of s, and finally add a certain degree of noise n to obtain a low-quality, low-resolution CT image I LR ; Step 3: Obtain or select a blur kernel dataset and a noise dataset, and apply them to the industrial CT image degradation model to degrade the high-resolution CT image dataset, thereby obtaining a paired low-resolution CT image dataset; Step 4: Construct an enhanced Swin Transformer network for super-resolution reconstruction of industrial CT images. The overall structure of the network includes a shallow feature extraction module, a deep feature extraction module and an upsampling module. The shallow feature extraction module composed of 1 layer of 3×3 convolution is used to extract shallow features of the input low-resolution CT image, and the extracted feature map is sent to the deep feature extraction module to further extract deep features. Finally, the extracted shallow features and deep features are fused through a long skip connection, and upsampling is performed to obtain a high-resolution CT image. Step 5: Train and test the constructed enhanced Swin Transformer super-resolution reconstruction network; Step 6: Use the trained enhanced Swin Transformer super-resolution reconstruction network to perform super-resolution reconstruction on the industrial CT image of the part to be super-resolution reconstructed to obtain its high-resolution CT image.

2. The enhanced Swin Transformer super-resolution reconstruction method for industrial CT images according to claim 1, characterized in that: There are three methods for obtaining a high-resolution CT image dataset in step 1. You can select one of the methods as needed: (1) Use industrial CT simulation software to simulate and reconstruct the part design model to obtain a simulated high-resolution CT image dataset; (2) Use actual industrial CT to perform high-resolution scanning on the actual part and reconstruct the actual high-resolution CT image dataset; (3) Combine the simulated high-resolution CT image dataset and the actual high-resolution CT image dataset into a mixed high-resolution CT image dataset in a certain proportion.

3. The enhanced Swin Transformer super-resolution reconstruction method for industrial CT images according to claim 1, characterized in that: The following method is used to obtain the blur kernel data set in step 3: (1) Construct an unsupervised blur kernel estimation network, which includes a generator and a discriminator. The network structures of the generator and the discriminator are both multi-layer convolutional layer structures; (2) After the actual high-resolution CT image is input into the network, it is downsampled to an image of a certain multiple through the kernel estimation generator. At the same time, image blocks of the same size are cropped from the original high-resolution CT image. The discriminator determines whether the pixel distribution of the downsampled image and the cropped image is consistent; (3) After the training is completed, the degradation function layer of the generator network is extracted as the blur kernel k. One blur kernel is obtained for each actual high-resolution CT image. The blur kernel dataset can be obtained from the actual high-resolution CT image dataset.

4. The enhanced Swin Transformer super-resolution reconstruction method for industrial CT images according to claim 1, characterized in that: The blur kernel dataset and noise dataset obtained in step 3 can be saved as a public blur kernel dataset and a public noise dataset according to the equipment, parts, and scanning parameters as feature identifiers. When the feature identifiers of the parts to be super-resolution reconstructed are similar to the feature identifiers of the data in these two public datasets, they can be directly selected and applied to the industrial CT image degradation model.

5. The enhanced Swin Transformer super-resolution reconstruction method for industrial CT images according to claim 1, characterized in that: The deep feature extraction module described in step 4 contains 6 enhanced Swin Transformer groups and 1 3×3 convolutional layer. Each enhanced Swin Transformer group consists of 6 enhanced Swin Transformer blocks, 1 detail enhancement convolutional module and 1 3×3 convolutional layer with residual connection.

6. The detail enhancement convolution module according to claim 5, characterized in that It contains 5 convolutional layers arranged in parallel, including 1 traditional convolutional layer and 4 differential convolutional layers. The traditional convolutional layer is used to obtain intensity level information, and the differential convolutional layer is used to enhance the gradient level information. Finally, the features extracted by all convolutional layers are merged.

7. The enhanced Swin Transformer block according to claim 5, characterized in that The enhanced SwinTransformer block adds a multi-dimensional attention convolution module in parallel with the window-based multi-head self-attention module on the basis of the standard Swin Transformer block to improve the representation ability of the network. The multi-dimensional attention convolution module includes two convolutional layers with a GELU activation function between the two convolutional layers, and a multi-dimensional collaborative attention module after the second convolutional layer.

8. The multi-dimensional collaborative attention module according to claim 7, characterized in that It consists of three parallel branches, which respectively obtain the interdependence between features in different dimensions. Finally, the outputs of the three branches are averaged and aggregated, and the attention weights of each dimension are used to adjust them to obtain a more refined feature map.

Images