Terahertz image super-resolution method and system based on generative adversarial network and hybrid Transform
Through generative adversarial networks and hybrid Transformer technology, the problems of low resolution and noise influence of terahertz imaging systems are solved, and efficient image super-resolution and robustness are achieved, suitable for fields such as safe detection and medical imaging.
Patent Information
- Application Number
- CN202410981874.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-22
- Publication Date
- 2024-11-08
AI Technical Summary
Traditional terahertz imaging systems have low resolution and are susceptible to noise and environmental factors, resulting in reduced image quality and making it difficult to achieve high-definition and rich detailed imaging.
Using the terahertz image super-resolution method based on generative adversarial network and hybrid Transformer, the convolution enhancement dense module, residual Swin Transformer module and consolidation information Transformer module are used to remove noise and extract feature information to realize the image Detail feature enhancement and reconstruction.
Effectively remove noise, improve image resolution and robustness, suitable for real-time applications of terahertz images and a variety of scenarios such as safe detection and medical imaging.
Smart Images

Figure CN118918007A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of terahertz security inspection and image super-resolution, and in particular to a terahertz image super-resolution method and system based on a generative adversarial network and a hybrid Transformer. Background Art
[0002] The terahertz band is part of the electromagnetic spectrum, located between microwaves and infrared. Compared with visible light and infrared, the terahertz band has stronger penetration ability and can penetrate some common obstacles such as haze, paper, plastic, etc. to achieve imaging of hidden objects. Due to its strong electromagnetic wave penetration, it has good penetration for many non-metallic and non-conductive materials. At the same time, the energy of the terahertz band is relatively low, which has little impact on the human body and most materials, and will not cause significant biological hazards. Therefore, it is suitable for fields such as biomedicine and safety detection.
[0003] The resolution of traditional terahertz imaging systems is usually low, which makes it difficult to obtain high-definition and detailed images. In particular, in applications that require precise measurement or fine structure analysis, resolution limitations may become a significant problem. In addition, the terahertz band is easily affected by factors such as atmospheric conditions, material surface conditions, and system internal noise. There will be a large amount of Gaussian noise and information distortion in the overall image, resulting in a decrease in imaging quality. The design and debugging of terahertz imaging systems are relatively complex and require highly precise control and calibration to ensure imaging quality. This complexity not only increases the manufacturing cost of the equipment, but may also increase the difficulty of operation and maintenance. Due to wave source fluctuations and sensor noise, electromagnetic wave interference, and low wave source power, terahertz images have low spatial resolution, blurred contour features, and high background noise. Therefore, it is of great significance to develop an image enhancement algorithm with good denoising ability, high robustness, and high imaging resolution for terahertz image dangerous goods detection. Summary of the invention
[0004] In view of the above problems, the present invention provides a terahertz image super-resolution method based on a generative adversarial network and a hybrid Transformer, the method comprising:
[0005] Step S1, using a terahertz imaging device to take a terahertz image of the person to be inspected, and constructing a terahertz image data set;
[0006] Step S2, constructing a terahertz image super-resolution model based on a generative adversarial network and a moderated Transformer;
[0007] Step S3, using the terahertz image data set to train the terahertz image super-resolution model to obtain a trained terahertz image super-resolution model;
[0008] Step S4: Use the trained network to perform image super-resolution on the terahertz image to be detected.
[0009] Optionally, in step S1, a terahertz image is taken of the person to be inspected using a terahertz imaging device, and the contents of constructing a terahertz image data set specifically include:
[0010] Use the terahertz imaging device to take terahertz images of the personnel to be inspected, and obtain the terahertz image dataset T h =[T h1 ,T h2 ,...,T hk ], for T h After 4 times downsampling, we get the data set T l =[T l1 ,T l2 ,...,T lk ], the T h For high-resolution terahertz images, the T l This is a low-resolution terahertz image.
[0011] Optionally, in S2, the content of constructing a terahertz image super-resolution model based on a generative adversarial network and a moderated Transformer specifically includes:
[0012] Step S21, using a generator structure of a generative adversarial network to extract features of the terahertz image data set to obtain a feature map;
[0013] Step S22: performing spatial and channel convolution on the feature maps and concatenating the feature maps to obtain a first fused feature map;
[0014] Step S23, inputting the first fusion feature into a residual Swin Transformer module to obtain a second fusion feature;
[0015] Step S24: input the second fusion feature into the information consolidation Transformer module to obtain the third fusion feature.
[0016] Optionally, in step S22, the feature maps are subjected to spatial and channel convolution and concatenated to obtain the first fused feature map, which specifically includes:
[0017] The feature map is input into the convolution layer for feature extraction and noise information is filtered out to obtain a feature map t1 after feature activation;
[0018] The feature map t1 is subjected to spatial and channel convolution to obtain a feature map t2;
[0019] After weighting the feature map t1 by an optimization factor γ, the feature map t2 is concatenated and fused on dimension 1 to obtain the first fused feature map T C .
[0020] Optionally, in step S23, the first fusion feature is input into a residual Swin Transformer module to obtain the second fusion feature, which specifically includes:
[0021] After the feature map T1 passes through the first layer of residual Swin Transformer module, a convolutional layer is used to perform a residual connection between the input feature and the output feature to obtain the first output feature;
[0022] After the first feature map passes through the second layer of residual Swin Transformer module, a convolution layer is used to perform a residual connection between the input feature and the output feature to obtain the second output feature;
[0023] After the second feature map passes through the second layer residual Swin Transformer module, a convolution layer is used to perform a residual connection between the input feature and the output feature to obtain the third output feature;
[0024] After the third feature map passes through the second layer residual Swin Transformer module, a convolution layer is used to perform a residual connection between the input feature and the output feature to obtain the second fusion feature.
[0025] Optionally, in step S24, the second fused feature is input into a consolidation information transformer module to obtain the third fused feature, which specifically includes:
[0026] The feature map t2 is passed through the information consolidation Transformer module to consolidate the feature information, that is, the third fusion feature E;
[0027] E=τ·LeFF(LN(WMSA(LN(T2))+T2))+(WMSA(LN(T2))+T2)+T2
[0028] Among them, LeFF(·) represents the enhanced feedforward network function; WMSA(·) represents the weighted improved weighted multi-head self-attention mechanism; LN(·) represents the layer normalization function.
[0029] The present invention also discloses a terahertz image super-resolution system based on a generative adversarial network and a hybrid Transformer, the system comprising: a data acquisition module, a model building module, a model optimization module and an image super-resolution detection module;
[0030] The data acquisition module is used to use a terahertz imaging device to take a terahertz image of the person to be inspected and construct a terahertz image data set;
[0031] The model building module is used to build a terahertz image super-resolution model based on a generative adversarial network and a moderated Transformer;
[0032] The model optimization module is used to train the terahertz image super-resolution model using the terahertz image data set to obtain a trained terahertz image super-resolution model;
[0033] The image super-resolution detection module is used to perform image super-resolution on the terahertz image to be detected using a trained network.
[0034] Optionally, the model building module includes an adversarial network feature extraction submodule, a spatial channel convolution submodule, a Transformer submodule and a feature fusion submodule;
[0035] The adversarial network feature extraction submodule is used to extract features of the terahertz image data set using a generator structure of a generative adversarial network to obtain a feature map;
[0036] The spatial channel convolution submodule is used to perform spatial and channel convolution on the feature map and perform splicing to obtain a first fused feature map;
[0037] The Transformer submodule is used to input the first fusion feature into the residual SwinTransformer module to obtain the second fusion feature;
[0038] The feature fusion submodule is used to input the second fused feature into the consolidated information Transformer module to obtain the third fused feature.
[0039] Compared with the prior art, the present invention has the following beneficial effects:
[0040] The present invention provides a terahertz image super-resolution method and system based on a generative adversarial network and a hybrid Transformer. It is difficult to remove the interference of noise in the traditional terahertz image super-resolution algorithm. The present invention effectively removes the noise in the terahertz image and extracts the feature information by designing a residual space and channel reconstruction convolution enhancement dense module, thereby providing feature map information with better robustness for subsequent detail feature enhancement and image reconstruction. At the same time, the residual Swin Transformer module introduced realizes the mobile extraction between feature maps and the fusion of multiple channels, and can stably read key information such as detail contours in poor information. Finally, the consolidated information Transformer module is used in the image reconstruction part to consolidate the amplified feature map information of the model after upsampling. The method is suitable for real-time applications and various scenarios of terahertz image super-resolution. It has broad application prospects in many fields, such as safety detection, non-destructive testing and medical imaging. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0042] Figure 1 Schematic diagram of the composition of the terahertz imaging system according to an embodiment of the present invention, wherein the terahertz imaging device includes a computer 101, a terahertz imaging device 102, an image conversion device 103, a terahertz band transceiver 104, and a person under inspection carrying an item 105;
[0043] Figure 2 This is a schematic diagram of the architecture of a super-resolution network generator in an embodiment of the present invention;
[0044] Figure 3 It is a schematic diagram of the composition of the residual space and channel reconstruction convolution enhancement dense module in the super-resolution network generator architecture of an embodiment of the present invention;
[0045] Figure 4 Schematic diagram of the composition of the residual SwinTransformer module in the super-resolution network generator architecture of an embodiment of the present invention;
[0046] Figure 5 Schematic diagram of the composition of the information consolidation Transformer module in the super-resolution network generator architecture of an embodiment of the present invention. DETAILED DESCRIPTION
[0047] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.
[0048] Embodiment 1
[0049] The present invention provides a terahertz image super-resolution method based on a generative adversarial network and a hybrid Transformer, such as Figure 2 As shown, the method includes:
[0050] Step S1: Use a terahertz imaging device to take a terahertz image of the person to be inspected, and construct a terahertz image data set.
[0051] In step S1, the terahertz imaging device 102 is used to take a terahertz image of the person to be inspected 105, and the contents of constructing the terahertz image data set specifically include: taking a terahertz image of the person to be inspected by the terahertz imaging device, and obtaining a terahertz image data set T h =[T h1 ,T h2 ,...,T hk ], for T h After 4 times downsampling, we get the data set T l =[T l1 ,T l2 ,...,T lk ], the T h For high-resolution terahertz images, the T l is a low-resolution terahertz image. Dataset T h and T l The total number of elements in the dataset is K = 6340. h The image size in the dataset is 3×240×480. l The image size in is 3×60×120. During the training process, the dataset T h As high-resolution image input for training, the dataset T l As low-resolution image input for training.
[0052] In this embodiment, Figure 1 As shown, the terahertz imaging device includes a computer 101, a terahertz imaging device 102, an image conversion device 103, a terahertz band transceiver 104, and a person under inspection carrying items 105; the computer 101 is connected to the terahertz imaging device 102, and is also connected to the image conversion device 103, which is then connected to the terahertz band transceiver 104. The computer obtains the terahertz image of the person under inspection carrying items 105 through a deep learning image super-resolution algorithm and performs super-resolution output display.
[0053] Step S2: construct a terahertz image super-resolution model based on a generative adversarial network and a moderated Transformer.
[0054] In S2, the contents of building a terahertz image super-resolution model based on generative adversarial networks and moderated Transformer specifically include:
[0055] Step S21: Use the generator structure of the generative adversarial network to extract features of the terahertz image data set to obtain a feature map.
[0056] The generator structure of the generative adversarial network in the terahertz image super-resolution method of Transformer, the hybrid Transformer is composed of the residual Swin Transformer and the consolidated information Transformer in the overall structure of the generator network, and the generator network structure includes a shallow feature extraction network, a deep feature extraction network, and an image reconstruction module. The shallow feature extraction network is a convolution layer with a convolution kernel of 3, and the deep feature extraction network includes a series of operations such as residual space and channel reconstruction convolution enhancement dense modules, residual Swin Transformer dense blocks, and convolution. The image reconstruction module is an upsampling, convolution, and consolidated information Transformer module structure.
[0057] During the training process, low-resolution terahertz images T with sizes of 3×60×120 and 3×240×480 were used. l and high-resolution terahertz images T h As input, in the deep feature extraction network of the model, T l The feature map T obtained in the shallow feature extraction network is processed by a series of processes such as residual space and channel reconstruction convolution enhancement dense modules, addition, normalized convolution, residual Swin Transformer dense, upsampling and consolidation information Transformer modules. The super-resolution image T output by the generated network is obtained. SR , with a size of 3×240×480. At the same time, the designed residual space and channel reconstruction convolution enhancement dense module, residual Swin Transformer module and consolidation information Transformer module are used to remove noise interference from the input feature information map, enhance the extraction of detail feature information and restore the contour features in the subsequent image reconstruction, thereby enhancing the super-resolution image generation effect of the generator network.
[0058] Finally, T SR and T hThe information is put into the discriminator of the generative adversarial network and passed to the generator through back propagation, which prompts the generator to adjust its generation strategy to generate more deceptive data, thereby improving the performance of the generator and the quality of the generated data. In the entire generative adversarial network, in order to make the generated super-resolution terahertz images closer to reality and better show the presence of dangerous goods carried by the inspected person, the algorithm uses spectral normalization (SN) U-Net as the discriminator network of the network.
[0059] Step S22: perform spatial and channel convolution on the feature maps and concatenate them to obtain a first fused feature map.
[0060] Residual space and channel reconstruction convolution enhanced dense modules such as Figure 3 As shown in the figure, the residual space and channel reconstruction convolution enhanced dense module is a standard convolution layer after the residual connection. Each residual space and channel reconstruction convolution enhanced dense module can be regarded as a stack of residual blocks, with spatial and channel reconstruction convolutions added on the path of adding residuals. Each module consists of a convolution layer, an activation layer, and a spatial and channel reconstruction convolution, which are concatenated and repeated three times and then passed through the residual. The main goal of this module is to enhance feature representation capabilities, reduce information loss, strengthen mobile extraction between feature maps and multi-channel fusion, and produce better SR effects. Assume that the given input feature map scale is 64×240×480. For the convolution activation part, first pass through the convolution layer Conv 3×3 (·) Reduce the channel dimension by 1 times to extract features and filter out noise information, then activate the output features to get feature map t1, with a scale of 180×240×480, and reconstruct the feature map t2 with a scale of 64×240×480 after spatial and channel convolution. Then weight t1 with an optimization factor γ and concatenate and fuse the features with t2 on dimension 1 to get feature map T C1 Similarly, the feature maps output by the next two residual blocks are T C2 , T C3 .T C1,C2,C3 The size is 180×240×480. The process can be described as:
[0061] t1=LR(Conv 3×3 (T))
[0062] t2=Conv SC (T)
[0063] T C1 =Concate(γ·t1,t2,1)
[0064] In the above formula, LR(·) represents the activation function. Conv(·) represents the convolution function with a convolution kernel of 3. SC(·) represents the spatial and channel reconstruction convolution function. Concate(·) is the concatenation function. T C1 It represents the output feature map of the first residual block, and t1 is the residual space and channel reconstruction convolution enhancement dense module. The channel dimension is first reduced by 1 times through the convolution layer to extract features and filter out noise information, and then the output feature is activated to obtain the feature map t1.
[0065] For the entire dense module part, in particular, this part is connected in series through three residual modules, so that the input feature information can be repeatedly denoised and refined, and then the final refined feature map T C3 After adding a weighted optimization factor γ, the features are concatenated and fused with T on dimension 1 to obtain the output feature map T of the entire dense module. De , the process of this dense module can be described as:
[0066] T De =Concate(γ·T C3 , T, 1)
[0067] In the above formula, T De Represents the final output features of the residual space and channel reconstruction convolution enhancement dense module, and the feature map scale is 180×240×480.
[0068] Step S23, input the first fusion feature into the residual Swin Transformer module to obtain the second fusion feature.
[0069] The residual Swin Transformer module is as follows Figure 4 As shown in Figure 1, first, given the input feature T1, here, T1 is the given input feature map of the residual Swin Transformer module, with a scale of 180×240×480. Let the input feature pass through the four-layer designed residual Swin Transformer module and then pass through a convolution layer to perform a residual connection between the input feature and the output feature to obtain the output feature T RS , T Rs∈180×240×480. Since the hierarchical window mechanism is used in Swin Transformer to divide the image into windows, the designed residual Swin Transformer module can effectively solve the problem of sensitivity to image scale in the original Transformer model and improve computational efficiency. Through window processing, the designed residual Swin Transformer module can maintain the effective transmission of local details and global context information and expand the receptive field. In addition, the residual connection therein helps to improve the context interaction ability in the Transformer model. The process of the first residual Swin Transformer module can be expressed as:
[0070] Z1=MLP(LN(MSA(LN(Z))+Z))+(MSA(LN(Z))+Z)+Z
[0071] In the above formula, LN(·) represents the layer normalization function. MSA(·) represents the multi-head self-attention function. MLP(·) represents the multi-layer perceptron function. Z is the input feature map. Z1 represents the output feature map of the first residual Swin Transformer module.
[0072] At the same time, Z1 also passes through three residual Swin Transformer modules to obtain feature maps Z2, Z3, Z4, and Z 1,2,3,4 ∈180×240×480. Where Z4 is the final output feature map of all residual Swin Transformer modules.
[0073] Step S24: input the second fusion feature into the information consolidation Transformer module to obtain the third fusion feature.
[0074] Consolidate information Transformer modules such as Figure 5As shown in the figure, first, assuming that the output feature T2∈180×960×1920 after upsampling, let the output feature be used as the input feature to pass through the information consolidation Transformer module, which consolidates the model's amplified feature map information and generalization ability. LeWinformer can capture the upsampled feature information during image restoration and help restore missing and damaged features. But there is a problem with LeWinformer. It lacks feature interaction between sub-images. To solve this problem, the module adds a jump connection line between the input layer and the output layer of LeWinformer, and also adjusts the weight matrix of its feedforward neural network. Therefore, the information consolidation Transformer module inherits all the advantages of LeWinformer, such as capturing feature information from a global and local perspective, processing deformation, rotation and scaling in the input image, and also strengthening its global interaction ability. The upsampled feature information is consolidated, and the robustness of the generated image information is improved. The input feature map consolidation process is:
[0075] E=τ·LeFF(LN(WMSA(LN(T2))+T2))+(WMSA(LN(T2))+T2)+T2
[0076] In the above formula, LeFF(·) represents the enhanced feedforward network function. WMSA(·) represents the weighted improved weighted multi-head self-attention function. LN(·) represents the layer normalization function. E is the output feature map of the information consolidation Transformer module, with a size of 180×960×1920.
[0077] Step S3: Use the terahertz image data set to train the terahertz image super-resolution model to obtain a trained terahertz image super-resolution model.
[0078] The training is carried out in steps. First, the generator network is trained and then the generated pre-trained weights are loaded into the entire generative adversarial network for complete training. The generator network is trained separately for 250,000 iter, and then the trained weights are loaded to train the complete model for 250,000 iter. The loss functions used are L1 loss function, perceptual loss function and adversarial loss. Set the network training parameters: learning rate lr = 2e-4, batchsize = 6, the training set validation set test set division ratio is 0.8:0.1:0.1, the optimizer uses Adam, and the training cycle is 150 rounds.
[0079] Step S4: Use the trained network to perform image super-resolution on the terahertz image to be detected.
[0080] Use the trained network to make predictions, input the terahertz test image, and output the terahertz super-resolution image. TL The image size is 3×240×480 when input into the network. After the generator network is inferred, the generated super-resolution terahertz image is obtained, and the size of the output terahertz image is 3×960×1920. The output feature map is input into the trained discriminator network for judgment, and the output of the discriminator is calculated through forward propagation, and then the loss function of the discriminator is calculated. Through back propagation and optimization algorithms, the parameters of the discriminator are updated to reduce the loss. Finally, the thoroughly optimized generator network outputs a realistic super-resolution terahertz image that passes the discriminator. This is the final super-resolution terahertz image with an image size of 3×960×1920.
[0081] Embodiment 2
[0082] A terahertz image super-resolution system based on a generative adversarial network and a hybrid Transformer, the system comprising: a data acquisition module, a model building module, a model optimization module and an image super-resolution detection module;
[0083] The data acquisition module is used to use a terahertz imaging device to take terahertz images of the person to be inspected and construct a terahertz image data set.
[0084] Using the terahertz imaging device 102 to take a terahertz image of the person to be inspected 105, the contents of constructing the terahertz image data set specifically include: using the terahertz imaging device to take a terahertz image of the person to be inspected, obtaining a terahertz image data set T h =[T h1 ,T h2 ,...,T hk ], for T h After 4 times downsampling, we get the data set T l =[T l1 ,T l2 ,...,T lk ], the T h For high-resolution terahertz images, the T l is a low-resolution terahertz image. Dataset T h and T l The total number of elements in the dataset is K = 6340. h The image size in the dataset is 3×240×480. l The image size in is 3×60×120. h As high-resolution image input for training, the dataset T l As low-resolution image input for training.
[0085] In this embodiment, the terahertz imaging device includes a computer 101, a terahertz imaging device 102, an image conversion device 103, a terahertz band transceiver 104, and a person under inspection carrying items 105; the computer 101 is connected to the terahertz imaging device 102, and is also connected to the image conversion device 103, which is then connected to the terahertz band transceiver 104. The computer obtains the terahertz image of the person under inspection carrying items 105 through a deep learning image super-resolution algorithm and performs super-resolution output display.
[0086] Model building module, used to build a terahertz image super-resolution model based on the generative adversarial network and the moderated Transformer. The model building module includes the adversarial network feature extraction submodule, the spatial channel convolution submodule, the Transformer submodule and the feature fusion submodule;
[0087] The adversarial network feature extraction submodule is used to extract the features of the terahertz image data set using the generator structure of the generative adversarial network to obtain a feature map. The generator structure of the generative adversarial network in the terahertz image super-resolution method of Transformer, the hybrid Transformer is composed of the residual SwinTransformer and the consolidated information Transformer in the overall structure of the generator network, and the generator network structure includes a shallow feature extraction network, a deep feature extraction network, and an image reconstruction module. The shallow feature extraction network is a convolution layer with a convolution kernel of 3, and the deep feature extraction network includes a series of operations such as residual space and channel reconstruction convolution enhancement dense modules, residual SwinTransformer dense blocks, and convolution. The image reconstruction module is an upsampling, convolution, and consolidated information Transformer module structure.
[0088] During the training process, low-resolution terahertz images T with sizes of 3×60×120 and 3×240×480 were used. l and high-resolution terahertz images T h As input, in the deep feature extraction network of the model, T l The feature map T obtained in the shallow feature extraction network is processed by a series of processes such as residual space and channel reconstruction convolution enhancement dense modules, addition, normalized convolution, residual SwinTransformer dense, upsampling and consolidation information Transformer modules. The super-resolution image T output by the generated network is obtained. SR, with a size of 3×240×480. At the same time, the designed residual space and channel reconstruction convolution enhancement dense module, residual SwinTransformer module and consolidation information Transformer module are used to remove noise interference from the input feature information map, enhance the extraction of detail feature information and restore the contour features in the subsequent image reconstruction, thereby enhancing the super-resolution image generation effect of the generator network.
[0089] Finally, T SR and T h The information is put into the discriminator of the generative adversarial network and passed to the generator through back propagation, which prompts the generator to adjust its generation strategy to generate more deceptive data, thereby improving the performance of the generator and the quality of the generated data. In the entire generative adversarial network, in order to make the generated super-resolution terahertz images closer to reality and better show the presence of dangerous goods carried by the inspected person, the algorithm uses spectral normalization (SN) U-Net as the discriminator network of the network.
[0090] The spatial channel convolution submodule is used to convolve the feature map in space and channel, and then concatenate the feature map to obtain a first fused feature map. Figure 3 As shown in the figure, the residual space and channel reconstruction convolution enhanced dense module is a standard convolution layer after the residual connection. Each residual space and channel reconstruction convolution enhanced dense module can be regarded as a stack of residual blocks, with spatial and channel reconstruction convolutions added on the path of adding residuals. Each module consists of a convolution layer, an activation layer, and a spatial and channel reconstruction convolution, which are concatenated and repeated three times and then passed through the residual. The main goal of this module is to enhance feature representation capabilities, reduce information loss, strengthen mobile extraction between feature maps and multi-channel fusion, and produce better SR effects. Assume that the given input feature map scale is 64×240×480. For the convolution activation part, first pass through the convolution layer Conv 3×3 (·) Reduce the channel dimension by 1 times to extract features and filter out noise information, then activate the output features to get feature map t1, with a scale of 180×240×480, and reconstruct the feature map t2 with a scale of 64×240×480 after spatial and channel convolution. Then weight t1 with an optimization factor γ and concatenate and fuse the features with t2 on dimension 1 to get feature map T C1 Similarly, the feature maps output by the next two residual blocks are T C2 , T C3 .T C1,C2,C3 The size is 180×240×480. The process can be described as:
[0091] t1=LR(Conv 3×3 (T))
[0092] t2=Conv SC (T)
[0093] T C1 =Concate(γ·t1,t2,1)
[0094] In the above formula, LR(·) represents the activation function. Conv(·) represents the convolution function with a convolution kernel of 3. SC (·) represents the spatial and channel reconstruction convolution function. Concate(·) is the concatenation function. T C1 Represents the output feature map of the first residual block.
[0095] For the entire dense module part, in particular, this part is connected in series through three residual modules, so that the input feature information can be repeatedly denoised and refined, and then the final refined feature map T C3 After adding a weighted optimization factor γ, the features are concatenated and fused with T on dimension 1 to obtain the output feature map T of the entire dense module. De , the process of this dense module can be described as:
[0096] T De =Concate(γ·T C3 , T, 1)
[0097] In the above formula, T De Represents the final output features of the residual space and channel reconstruction convolution enhancement dense module, and the feature map scale is 180×240×480.
[0098] The Transformer submodule is used to input the first fusion feature into the residual Swin Transformer module to obtain the second fusion feature.
[0099] The residual Swin Transformer module is as follows Figure 4 As shown in the figure, first, given the input feature T1, the scale is 180×240×480. Let the input feature pass through the four-layer residual Swin Transformer module and then pass through a convolutional layer to perform a residual connection between the input feature and the output feature to obtain the output feature T RS , T Rs∈180×240×480. Since the hierarchical window mechanism is used in Swin Transformer to divide the image into windows, the designed residual Swin Transformer module can effectively solve the problem of sensitivity to image scale in the original Transformer model and improve computational efficiency. Through window processing, the designed residual Swin Transformer module can maintain the effective transmission of local details and global context information and expand the receptive field. In addition, the residual connection helps to improve the context interaction ability in the Transformer model. The process of the first residual Swin Transformer module can be expressed as:
[0100] Z1=MLP(LN(MSA(LN(Z))+Z))+(MSA(LN(Z))+Z)+Z
[0101] In the above formula, LN(·) represents the layer normalization function. MSA(·) represents the multi-head self-attention function. MLP(·) represents the multi-layer perceptron function. Z is the input feature map. Z1 represents the output feature map of the first residual Swin Transformer module.
[0102] At the same time, Z1 also passes through three residual Swin Transformer modules to obtain feature maps Z2, Z3, Z4, and Z 1,2,3,4 ∈180×240×480. Where Z4 is the final output feature map of all residual Swin Transformer modules.
[0103] The feature fusion submodule is used to input the second fused feature into the information consolidation transformer module to obtain the third fused feature. Figure 5As shown in the figure, first, assuming that the output feature T2∈180×960×1920 after upsampling, let the output feature be used as the input feature to pass through the information consolidation Transformer module, which consolidates the model's amplified feature map information and generalization ability. LeWinformer can capture the upsampled feature information during image restoration and help restore missing and damaged features. But there is a problem with LeWinformer. It lacks feature interaction between sub-images. To solve this problem, the module adds a jump connection line between the input layer and the output layer of LeWinformer, and also adjusts the weight matrix of its feedforward neural network. Therefore, the information consolidation Transformer module inherits all the advantages of LeWinformer, such as capturing feature information from a global and local perspective, processing deformation, rotation and scaling in the input image, and also strengthening its global interaction ability. The upsampled feature information is consolidated, and the robustness of the generated image information is improved. The input feature map consolidation process is:
[0104] E=τ·LeFF(LN(WMSA(LN(T2))+T2))+(WMSA(LN(T2))+T2)+T2
[0105] In the above formula, LeFF(·) represents the enhanced feedforward network function. WMSA(·) represents the weighted improved weighted multi-head self-attention function. LN(·) represents the layer normalization function. E is the output feature map of the information consolidation Transformer module, with a size of 180×960×1920.
[0106] The model optimization module is used to train the terahertz image super-resolution model using the terahertz image dataset to obtain a trained terahertz image super-resolution model; the training is carried out in a step-by-step manner, first training the generator network part and then loading the generated pre-trained weights into the entire generative adversarial network for complete training. The generator network is trained separately for 250,000 iters, and then the trained weights are loaded to train the complete model, and 250,000 iters are trained. The loss functions used are L1 loss function, perceptual loss function, and adversarial loss. Set the network training parameters: learning rate lr = 2e-4, batchsize = 6, the training set validation set test set division ratio is 0.8: 0.1: 0.1, the optimizer uses Adam, and the training cycle is 150 rounds.
[0107] The image super-resolution detection module is used to perform image super-resolution on the terahertz image to be detected using a trained network.
[0108] Use the trained network to make predictions, input the terahertz test image, and output the terahertz super-resolution image. TLThe image size is 3×240×480 when input into the network. After the generator network is inferred, the generated super-resolution terahertz image is obtained, and the size of the output terahertz image is 3×960×1920. The output feature map is input into the trained discriminator network for judgment, and the output of the discriminator is calculated through forward propagation, and then the loss function of the discriminator is calculated. Through back propagation and optimization algorithms, the parameters of the discriminator are updated to reduce the loss. Finally, the thoroughly optimized generator network outputs a realistic super-resolution terahertz image that passes the discriminator. This is the final super-resolution terahertz image with an image size of 3×960×1920.
[0109] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.
Claims
1. A terahertz image super-resolution method based on generative adversarial network and hybrid Transformer, It is characterized in that The method comprises: Step S1, using a terahertz imaging device to take a terahertz image of the person to be inspected, and constructing a terahertz image data set; Step S2, constructing a terahertz image super-resolution model based on a generative adversarial network and a moderated Transformer; Step S3, using the terahertz image data set to train the terahertz image super-resolution model to obtain a trained terahertz image super-resolution model; Step S4: Use the trained network to perform image super-resolution on the terahertz image to be detected.
2. According to claim 1, the terahertz image super-resolution method based on generative adversarial network and hybrid Transformer, It is characterized in that In step S1, a terahertz image is taken of the person to be inspected using a terahertz imaging device, and the contents of constructing a terahertz image data set specifically include: Use the terahertz imaging device to take terahertz images of the personnel to be inspected, and obtain the terahertz image dataset T h =[T h1 ,T h2 ,...,T hk ], for T h After 4 times downsampling, we get the data set T l =[T l1 ,T l2 ,...,T lk ], the T h For high-resolution terahertz images, the T l This is a low-resolution terahertz image.
3. The terahertz image super-resolution method based on generative adversarial network and hybrid Transformer according to claim 1, It is characterized in that In S2, the content of constructing a terahertz image super-resolution model based on a generative adversarial network and a moderated Transformer specifically includes: Step S21, using a generator structure of a generative adversarial network to extract features of the terahertz image data set to obtain a feature map; Step S22: performing spatial and channel convolution on the feature maps and concatenating the feature maps to obtain a first fused feature map; Step S23, inputting the first fusion feature into a residual Swin Transformer module to obtain a second fusion feature; Step S24: input the second fusion feature into the information consolidation Transformer module to obtain the third fusion feature.
4. The terahertz image super-resolution method based on generative adversarial network and hybrid Transformer according to claim 3, It is characterized in that In step S22, the feature maps are subjected to spatial and channel convolution and concatenated to obtain the first fused feature map, which specifically includes: The feature map is input into the convolution layer for feature extraction and noise information is filtered out to obtain the feature map t after feature activation. 1 ; The feature map t 1 After spatial and channel convolution, the feature map t is obtained 2 ; The feature map t 1 After weighting an optimization factor γ and the feature map t 2 Concatenate and fuse the features on dimension 1 to obtain the first fused feature map T C .
5. The terahertz image super-resolution method based on generative adversarial network and hybrid Transformer according to claim 4, It is characterized in that In the step S23, the first fusion feature is input into the residual Swin Transformer module to obtain the second fusion feature, which specifically includes: The feature map T 1 After the first layer of residual Swin Transformer module, a convolutional layer is used to perform a residual connection between the input feature and the output feature to obtain the first output feature; After the first output feature passes through the second layer of residual Swin Transformer module, a convolution layer is used to perform a residual connection between the input feature and the output feature to obtain the second output feature; After the second output feature passes through the second layer residual Swin Transformer module, a convolution layer is used to perform a residual connection between the input feature and the output feature to obtain the third output feature; After the third output feature passes through the second layer residual Swin Transformer module, a residual connection is performed between the input feature and the output feature through a convolution layer to obtain the second fusion feature.
6. The terahertz image super-resolution method based on generative adversarial network and hybrid Transformer according to claim 5, It is characterized in that In the step S24, the second fusion feature is input into the consolidation information Transformer module, and the content of the third fusion feature obtained specifically includes: The feature map t 2 The feature information is consolidated through the information consolidation Transformer module, i.e. the third fusion feature E; E=τ.LeFF(LN(WMSA(LN(T 2 ))+T 2 ))+(WMSA(LN(T 2 ))+T 2 )+T 2 Among them, LeFF(·) represents the enhanced feedforward network function; WMSA(·) represents the weighted improved weighted multi-head self-attention mechanism; LN(·) represents the layer normalization function.
7. A terahertz image super-resolution system based on a generative adversarial network and a hybrid Transformer, wherein the terahertz image super-resolution system is used to implement the terahertz image super-resolution method according to any one of claims 1 to 6, It is characterized in that The system includes: data acquisition module, model building module, model optimization module and image super-resolution detection module; The data acquisition module is used to use a terahertz imaging device to take a terahertz image of the person to be inspected and construct a terahertz image data set; The model building module is used to build a terahertz image super-resolution model based on a generative adversarial network and a moderated Transformer; The model optimization module is used to train the terahertz image super-resolution model using the terahertz image data set to obtain a trained terahertz image super-resolution model; The image super-resolution detection module is used to perform image super-resolution on the terahertz image to be detected using a trained network.
8. The terahertz image super-resolution system based on generative adversarial network and hybrid Transformer according to claim 7, It is characterized in that The model building module includes an adversarial network feature extraction submodule, a spatial channel convolution submodule, a Transformer submodule and a feature fusion submodule; The adversarial network feature extraction submodule is used to extract features of the terahertz image data set using a generator structure of a generative adversarial network to obtain a feature map; The spatial channel convolution submodule is used to perform spatial and channel convolution on the feature map and perform splicing to obtain a first fused feature map; The Transformer submodule is used to input the first fusion feature into the residual Swin Transformer module to obtain the second fusion feature; The feature fusion submodule is used to input the second fused feature into the consolidated information Transformer module to obtain the third fused feature.
Citation Information
Patent Citations
Terahertz image denoising method based on multi-scale mixed attention
CN114581330A
Image super-resolution reconstruction model and method based on residual mixed attention network
CN115222601A
Terahertz image super-resolution method based on deep learning algorithm
CN115375540A
TransSR-Net structured image super-resolution reconstruction method
CN116664397A
Super-resolution image reconstruction method and device
CN118134763A
Cited By
Terahertz image super-resolution reconstruction method and system based on multi-dimensional attention fusion network
CN120147136A
Unsupervised parallax tolerant terahertz image splicing method and system
CN120563317A
