Thermal infrared image optimization method based on multi-channel fusion and semantic information

Abstract

The application provides a thermal infrared image optimization method based on multi-channel fusion and semantic information, and relates to the technical field of image processing. The method comprises the following steps: acquiring VIS and corresponding TIR data sets, inputting the HR image obtained after preprocessing into an MCFGAN model, the MCFGAN model comprising a semantic discriminator and a generator, the generator comprising three skip-connected DISR modules; inputting the HR image data set into an infrared degradation model to obtain an LR image data set; training the generator by using the LR and HR image data sets to obtain an SR image data set; combining the semantic discriminator; training the generator and the discriminator by means of a generative adversarial network to obtain an optimized MCFGAN model, effectively eliminating the noise in the thermal infrared image, improving the extraction efficiency of high-frequency details of the thermal infrared image, improving the information mining depth and the utilization rate of information in attention, thereby improving the information fusion efficiency, improving the image quality and resolution.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a thermal infrared image optimization method based on multi-channel fusion and semantic information. BACKGROUND

[0002] Currently, infrared thermal imaging technology can perform non-contact, high-resolution temperature imaging, providing rich information about the measured target, and thus has been widely used in the power system, civil engineering, automobile, metallurgy, petrochemical and pharmaceutical industries. Limited by the hardware performance of the thermal imager, compared with the visible light (VIS) image, the thermal infrared (TIR) image has the characteristics of low resolution, high noise, blurred edges, and the existence of random bright spots and irregular stripes. In order to overcome the limitations of hardware facilities, it is more cost-effective to improve the resolution of TIR images by using super resolution (SR) reconstruction technology.

[0003] Among them, single image super resolution is a classic computer vision problem, which aims to recover a high resolution (HR) image from a given low resolution (LR) image. In the field of VIS images, with the rapid development of deep neural networks, SR has made considerable breakthroughs. The current mainstream SR methods include: convolutional neural network based, generative adversarial network (GAN) based, Transformers based, blind super resolution and real-world super resolution.

[0004] Currently, in the field of infrared image research, an infrared image single image super resolution method combining compressed sensing theory and deep learning is adopted, which uses the sparsity of low resolution images as the compressed sampling result of high resolution images in compressed sensing to reconstruct higher resolution images, and combines the Transformer with spatial and channel double attention mechanisms to solve the problem that more global edge structure information is needed for infrared SR.

[0005] The existing research has problems such as poor detail recovery and large noise. The present application proposes a thermal infrared image super resolution algorithm (MCFGAN, Multi-Channel Feature Generative Adversarial Network) based on multi-channel fusion and semantic information, and uses a training strategy of a semantic discriminator and a grayscale visible light image guided network training, which exhibits significant advantages in thermal infrared image super resolution compared with other super resolution networks. SUMMARY

[0006] The main purpose of the present application is to provide a thermal infrared image optimization method based on multi-channel fusion and semantic information, which solves the technical problems of poor detail recovery, large noise and lack of semantic information guidance in the field of infrared image processing in the prior art.

[0007] To solve the above technical problems, the technical solution adopted by the present application is: a thermal infrared image optimization method based on multi-channel fusion and semantic information, comprising the following steps:

[0008] Obtain the VIS and corresponding TIR data set, and after data preprocessing, obtain the HR image data set;

[0009] Construct the MCFGAN model; wherein the MCFGAN model includes a generator module with a multi-channel fusion attention module and a semantic discriminator: the generator includes three skip-connection multi-channel fusion attention modules, i.e. DISR modules; the semantic discriminator includes a semantic module and a semantic perception fusion module; wherein the semantic module is used to generate semantic information, and the semantic perception fusion module is used to integrate the semantic information into the discriminator;

[0010] Input the HR image data set into the infrared degradation model to obtain the LR image data set;

[0011] Train the generator using the HR image data set and the LR image data set to generate the SR image data set, and then input the generated SR image data set into the semantic discriminator to determine true or false;

[0012] In the semantic discriminator, the semantic module generates semantic information according to the HR image data set, and integrates the semantic information into the discriminator through the semantic perception fusion module, so as to perform adversarial training on the MCFGAN model, and obtain an optimized MCFGAN model;

[0013] Input the TIR picture to be tested into the optimized MCFGAN model to obtain an optimized thermal infrared image.

[0014] In the preferred scheme, the data preprocessing includes gray processing of the VIS and TIR data set to generate the HR image data set, and then CLAHE processing;

[0015] CLAHE processing: the core of the CLAHE contrast limited adaptive histogram equalization processing lies in local histogram equalization and contrast limitation, and the formula of the local histogram equalization is:

[0016] (1.1);

[0017] In the formula, is the number of pixels with a gray value of in the local area, To represent the cumulative sum of the number of all pixels with gray value less than or equal to , and is the pixel value of the image;

[0018] The local histogram equalization is linearly transformed so that the pixel value distribution of the image is as uniform as possible, and the formula is:

[0019] (1.2);

[0020] In the formula, is the pixel value of the original image at position , is the pixel value of the new image after local histogram equalization processing at position , is the cumulative distribution function value of the minimum pixel value in the local histogram, is the cumulative distribution function value of the maximum pixel value in the local histogram, is the cumulative distribution function value of the pixel value in the local histogram, is the maximum value of the new pixel value range;

[0021] The contrast limiting formula is:

[0022] (1.3);

[0023] In the formula, is the frequency value of the local histogram after contrast limiting processing, is the histogram of the local region, and ClipLimit is the contrast limiting parameter.

[0024] In the preferred scheme, in the semantic discriminator, the semantic module Semantic Model is used to extract accurate and rich semantic features of the infrared image; then the extracted semantic features are combined with the discriminator by using the semantic perception fusion module;

[0025] Among them, the semantic module is a Modified Resnet-50, and the semantic features S h of the third layer are extracted.

[0026] A1: semantic is transmitted to the self-attention module, and the query :

[0027] (7.1);

[0028] where LN, GN, SA and RA are layer normalization, group normalization, self-attention module and rearrangement respectively;

[0029] A2: the generator generates the SR image and the HR image in the HR dataset into the convolutional layer to obtain the original enhanced feature, and the original enhanced feature is rearranged to obtain the key of the SR image and the HR image respectively and and the value and Then the query Q, the key and the value of the SR image and the HR image are input into the Cross Attention module; and and the value and

[0030] A3: the semantic perception image feature output by the Cross Attention is connected with the original enhanced feature to obtain the final feature of the SR image and the HR image and The formula is:

[0031] (7.2);

[0032] (7.3);

[0033] (7.4);

[0034] (7.5);

[0035] (7.6);

[0036] (7.7);

[0037] (7.8);

[0038] (7.9);

[0039] wherein, is the output after the multi-head attention mechanism of the Cross Attention module, and is the output after the BasicTransformerBlock, is a scale factor, is a feedforward network, ​For the original enhanced feature.

[0040] In a preferred embodiment, the generator module comprises a shallow feature extraction module, a multi-channel fusion attention module for deep feature extraction, and a reconstruction module, and the formula is:

[0041] The shallow feature extraction module uses a convolutional layer with a 3x3 convolution kernel, and the formula is: , For the extracted feature map;

[0042] The multi-channel fusion attention module, i.e. the MCF module, is used to extract deep features, and the formula is:

[0043] (6.1);

[0044] In the formula, MCF block, the MCF block is composed of three skip-connected DISR blocks, is the output of the nth DISR block;

[0045] The DISR block contains a densely connected module branch, a context attention module branch, and a statistical spatial channel attention module branch, and the dynamic weights of the three branches are determined by the input features; then, features are extracted from using a 3x3 convolution kernel: , completing the deep feature extraction part;

[0046] The equation for image reconstruction is:

[0047] (6.2);

[0048] In the formula, is the interpolation up-sampling, is the final SR output.

[0049] In a preferred embodiment, each DISR module generates weights for three independent channels by using the same input features, and the formula for the specific operation is:

[0050] (5.1);

[0051] (5.2);

[0052] (5.3);

[0053] (5.4);

[0054] (5.5);

[0055] (5.6);

[0056] wherein, is the output of the dense connection, is the output of the contextual attention module, is the output of the statistical spatial channel attention module; is the weight value of the three channels generated by the dynamic weight module according to the input features.

[0057] In the preferred solution, each DISR module controls the balance through weighted summation, including the dynamic weighted contribution of the dense connection module, the contextual attention module and the statistical spatial channel attention module;

[0058] In the DISR module, the dense connection module Dense is used, and the formula is as follows:

[0059] (2.1);

[0060] (2.2);

[0061] (2.2);

[0062] (2.4);

[0063] (2.5);

[0064] (2.6);

[0065] (2.7);

[0066] (2.8);

[0067] (2.9);

[0068] (2.10);

[0069] wherein, is the convolution layer, is the activation function, is the feature obtained by layer-by-layer convolution processing and splicing with the previous layer, Y is the final output, is the weighting coefficient.

[0070] In the preferred solution, the contextual attention module is used in the DISR module, and the formula is as follows:

[0071] (3.1);

[0072] (3.2);

[0073] (3.3);

[0074] (3.4);

[0075] (3.5);

[0076] (3.6);

[0077] (3.7);

[0078] wherein, is the global feature, is the compressed feature, is the horizontal context information, is the vertical context information, is the length of the horizontal convolution kernel, is the height of the vertical convolution kernel, is the feature remapped to the original channel dimension, is the attention weight, is the final output, is the element-wise multiplication.

[0079] In the preferred solution, in the DISR module, the statistical spatial channel attention module has the following specific formula:

[0080] (4.1);

[0081] (4.2);

[0082] (4.3);

[0083] (4.4);

[0084] (4.5);

[0085] (4.6);

[0086] wherein, is the intra-channel mean, is the squared deviation of the mean, is the stable normalization factor after adding a regularization parameter to the normalization factor, is the normalization factor, is the regularization parameter to prevent the denominator from being too small when calculating the normalization factor, The attention weights are noted.

[0087] In the preferred solution, in the DISR module, the input features are first compressed using global average pooling, then passed through a connection layer containing two fully connected and one ReLU activation layer, and finally the weight of each branch is calculated by the Softmax function.

[0088] In the preferred solution, the HR image dataset and the LR image dataset are used to train the generator, and in the MCFGAN model adversarial training process, the discriminator is used to distinguish whether the image is true or false, and the discriminator is optimized by using the loss function, and the formula is:

[0089] (9.1);

[0090] wherein, and are the distributions of high-quality images and generated images , respectively, is the expectation, and D is the discriminator;

[0091] The generator is optimized by the combination of three losses, including L1 loss, perceptual loss and adversarial loss, and the formula is:

[0092] (9.2);

[0093] wherein, and are the weight coefficients of perceptual loss and adversarial loss, respectively, is the adversarial loss, is the perceptual loss, is the pixel-level supervision loss;

[0094] The goal of the adversarial loss is to make the generator deceive the discriminator, and the optimization goal is opposite to that of :

[0095] (9.3);

[0096] wherein, is the expectation, and are the distributions of high-quality images and generated images , respectively.

[0097] The application provides a thermal infrared image optimization method based on multi-channel fusion and semantic information, acquires VIS and corresponding TIR data sets, and then inputs the preprocessed HR image into an MCFGAN model, the MCFGAN model comprising a semantic discriminator and a generator, the generator comprising three skip-connected DISR modules; the HR image data set is input into an infrared degradation model to obtain an LR image data set, the generator is trained by using the LR and HR image data sets to obtain an SR image data set, the semantic discriminator is combined, the generator and the discriminator are trained by means of a generative adversarial network, and an optimized MCFGAN model is obtained, so that the noise in the thermal infrared image is effectively eliminated, the extraction efficiency of high-frequency details of the thermal infrared image is improved, the information mining depth and the information utilization rate in attention are improved, the information fusion efficiency is improved, and the image quality and resolution are improved. BRIEF DESCRIPTION OF DRAWINGS

[0098] The application will be further described below in combination with the drawings and embodiments:

[0099] Figure 1 is a flow chart of the thermal infrared image optimization method of the application;

[0100] Figure 2 is a schematic diagram of the generator model of the application;

[0101] Figure 3 is a semantic module structure Semantic Model diagram of the application;

[0102] Figure 4 is a semantic perception fusion block MST structure diagram of the application;

[0103] Figure 5 is a semantic discriminator structure diagram of the application;

[0104] Figure 6 is a generator overall network architecture diagram of the application;

[0105] Figure 7 is a comparison effect diagram of the MCFGAN and the existing super-resolution network when the scale factor is 4 in the application;

[0106] Figure 8 is a comparison effect diagram of the MCFGAN and the existing super-resolution network when the scale factor is 2 in the application. DETAILED DESCRIPTION

[0107] Embodiment 1

[0108] As shown in Figures 1-8 , a thermal infrared image optimization method based on multi-channel fusion and semantic information comprises the following steps:

[0109] S1: Obtain the VIS and the corresponding TIR dataset, and after data preprocessing, obtain the HR image dataset.

[0110] S2: Construct the MCFGAN model; wherein the MCFGAN model includes a generator (MCFNet) module with a multi-channel fusion attention (DISR) module and a semantic discriminator: the generator (MCFNet) includes three skip-connected DISR modules, and the semantic discriminator includes a semantic module and a semantic perception fusion module (MST), the semantic module is used to generate the semantic information, and the MST module is used to integrate the semantic information into the discriminator.

[0111] S3: Input the HR image dataset into the infrared degradation model to obtain the LR image dataset.

[0112] S4: Train the generator using the HR image dataset and the LR image dataset to generate the SR image dataset, and then input the generated SR image dataset into the semantic discriminator to determine the true or false.

[0113] S5: In the semantic discriminator, the semantic module generates semantic information according to the HR image dataset, and integrates the semantic information into the discriminator through the semantic perception fusion module, so as to perform adversarial training on the MCFGAN model, and obtain an optimized MCFGAN model.

[0114] S6: Input the to-be-tested VIS and the corresponding TIR picture into the optimized MCFGAN model to obtain an optimized thermal infrared image.

[0115] In this embodiment, the MCFNet is a multi-channel feature network (Multi-Channel Feature Network), and the DISR is a deep image super-resolution with multi-channel fusion (Deep Image Super-Resolution with Multi-Channel Fusion).

[0116] Further, Figure 1 The infrared degradation model used in this embodiment is a high-order thermal infrared degrader used in the application number CN 117372254A and named "a thermal infrared image super-resolution algorithm based on a multi-structure fusion generative adversarial network", which can be adaptively used with other modules that can obtain LR graphics.

[0117] In this embodiment, by acquiring VIS and corresponding TIR data sets, and then inputting the HR image data set obtained after preprocessing into the MCFGAN model, the MCFGAN model includes a semantic discriminator and a generator, and the generator includes three skip-connected DISR modules; the HR image data set is input into the generator to obtain an LR image data set, and then the LR and HR image data sets are used to train the generator (MCFNet) to obtain an SR image data set, in combination with the semantic discriminator, the generator and the discriminator are trained by means of a generative adversarial network (GAN), the semantic discriminator provides semantic information about the image content, an optimized MCFGAN model is obtained, and testing is performed to effectively eliminate noise in the thermal infrared image; the multi-channel fusion structure improves the information mining depth and the utilization rate of information in attention, thereby improving the efficiency of information fusion, improving the image quality and resolution.

[0118] The semantic discriminator is composed of a semantic module, a semantic perception fusion block and a common discriminator.

[0119] In the preferred scheme, in step S1, the data preprocessing includes gray-scale processing of the VIS and TIR data sets to generate an HR image data set, and then CLAHE (Contrast Limited Adaptive Histogram Equalization) processing is performed.

[0120] CLAHE processing: a technique for enhancing local contrast of an image by local histogram equalization and contrast limitation, aiming to avoid excessive amplification of noise while improving the detail information of the image. The core of the CLAHE (Contrast Limited Adaptive Histogram Equalization) processing is local histogram equalization and contrast limitation. The formula of local histogram equalization is:

[0121] (1.1);

[0122] In the formula, is the number of pixels with a gray value of in the local region, is the cumulative sum of the number of all pixels with a gray value less than or equal to , and are the pixel values of the image.

[0123] Then, the local histogram equalization is transformed by linear transformation to make the pixel value distribution of the image as uniform as possible, and the formula is:

[0124] (1.2);

[0125] In the formula, is the pixel value of the original image at position the pixel value of the new image at position the pixel value of the new image at position the pixel value of the new image at position the cumulative distribution function value of the minimum pixel value in the local histogram, the cumulative distribution function value of the maximum pixel value in the local histogram, the pixel value the cumulative distribution function value in the local histogram, the maximum value of the new pixel value range;

[0126] The contrast limiting formula is:

[0127] (1.3);

[0128] In the formula, the frequency value of the local histogram after the contrast limiting processing, the histogram of the local region, the frequency of the gray value of each pixel in the region, and the ClipLimit is a contrast limiting parameter for controlling the contrast enhancement of the local region.

[0129] In this embodiment, the gray-scale visible light image is used to guide the network training: first, the gray-scale visible light image is combined with the simple and clear thermal infrared image to construct a thermal infrared degradation generator to generate an LR image, and the MCFNet is trained using the LR and HR images, and then the semantic module is combined to guide the training of the MCFGAN.

[0130] In this embodiment, the generator (MCFNet) module includes a shallow feature extraction module, a DISR module for deep feature extraction, and a reconstruction module.

[0131] The generator includes three skip-connected DISR modules.

[0132] In the infrared image, there may be a large difference in subtle temperature changes or object shapes. The dense connection helps to capture these details through close cooperation between layers, thereby improving the perception of small changes.

[0133] In the preferred scheme, in the multi-channel fusion attention module, each DISR module controls the balance through weighted summation, including the dynamic weighted contribution of the dense connection module (Dense), the context attention module (DCA), and the statistical spatial channel attention module (SSCA);

[0134] The Dense module is used in the DISR module, and the formula is:

[0135] (2.1);

[0136] (2.2);

[0137] (2.2);

[0138] (2.4);

[0139] (2.5);

[0140] (2.6);

[0141] (2.7);

[0142] (2.8);

[0143] (2.9);

[0144] (2.10);

[0145] wherein, represents a convolution layer, is an activation function, is a feature obtained by layer-by-layer convolution processing and splicing with the previous layer, Y is the final output, is a weighting coefficient, used to enhance residual learning.

[0146] In this embodiment, the Dense module enhances the utilization of features and information flow by layer-by-layer convolution processing and splicing with the previous layer, thereby promoting the flow of information and the transmission of gradients, and reducing the problem of gradient disappearance.

[0147] The DCA module combines multiple convolution operations to extract context information, including horizontal convolution and vertical convolution, and generates adaptive weights related to the input feature map through these convolutions.

[0148] In the preferred scheme, the DCA module is used in the DISR module, and the formula is as follows:

[0149] (3.1);

[0150] (3.2);

[0151] (3.3);

[0152] (3.4);

[0153] (3.5);

[0154] (3.6);

[0155] (3.7);

[0156] wherein, is the global feature, is the compressed feature, is the horizontal context information, is the vertical context information, is the length of the horizontal convolution kernel, is the height of the vertical convolution kernel, is the feature remapped to the original channel dimension, is the attention weight, is the final output, is the element-wise multiplication.

[0157] In this embodiment, as shown in FIG. 3, the main purpose of the DCA module is to extract spatial context information through horizontal and vertical convolution operations, and to weight and integrate these information into the original feature through the attention mechanism. Finally, the context information and the original feature are combined through element-wise multiplication, which enhances the representation of key regions or channels and improves the network's ability to handle complex features. Figure 6

[0158] The SSCA module generates adaptive weights by calculating the mean difference kernel variance information of the image and applies them to the input to enhance the feature response of the key region.

[0159] In the preferred scheme, the specific formula of the SSCA module in the DISR module is as follows:

[0160] (4.1);

[0161] (4.2);

[0162] (4.3);

[0163] (4.4);

[0164] (4.5);

[0165] (4.6);

[0166] wherein, is the mean within the channel, is the square of the deviation of the mean, ​a stable normalization factor with a regularization parameter added to the normalization factor, a normalization factor, a regularization parameter to prevent the denominator from being too small when calculating the normalization factor, an attention weight.

[0167] In this embodiment, the SSCA module calculates the mean and variance of each channel, and performs stable normalization to prevent instability caused by too small values in the calculation; the attention weight is calculated, and the attention weight of each channel is calculated through the Sigmoid activation function, so that the features of the channel can be weighted according to the weight when output, the features of important channels will be amplified, and the features of unimportant channels will be suppressed. Finally, the original features and the calculated attention weight are weighted and fused through element-by-element multiplication to obtain the output features. The SSCA module dynamically focuses on the channels with large statistical changes and suppresses the channels with relatively stable information, thereby improving the effectiveness of feature representation and network performance.

[0168] In this embodiment, as shown in Figure 6 In the DISR module, the dense connection module enhances the details of the thermal infrared image by fusing features of different layers, the context attention module enhances the feature response on different scales to optimize the performance in low-contrast scenes, and the statistical spatial channel attention module captures the distribution of global and local features to improve the attention to key areas while adaptively identifying the degradation features of the thermal infrared image.

[0169] In the generator, the DISR module is used. Each DISR module controls the dynamic weighted contribution of balancing the dense connection module, the context attention module and the statistical spatial channel attention module through weighted summation. The DISR module generates weights by using the same input features of its block as three independent channels.

[0170] In the preferred scheme, each DISR module generates weights by using the same input features as three independent channels, and the formula for the specific operation is:

[0171] (5.1),

[0172] (5.2),

[0173] (5.3),

[0174] (5.4),

[0175] (5.5),

[0176] (5.6).

[0177] wherein, is the output of dense connection, is the output of context attention module, is the output of statistical spatial channel attention module. The weight values of the three channels are calculated according to the input features by the dynamic weight module instead of setting fixed values.

[0178] In order to simplify the learning process of the DISR module, in the embodiment, is set according to the actual situation.

[0179] In the embodiment, the generator with a multi-channel fusion attention module is used, which is called MCFNet. The MCFNet is shown in Figure 6 .

[0180] In the preferred scheme, the generator (MCFNet) module includes a shallow feature extraction module, a multi-channel fusion attention module for deep feature extraction, and a reconstruction module, and the formula is:

[0181] The shallow feature extraction module uses a convolution layer with a 3x3 convolution kernel, and the formula is , is the extracted feature map;

[0182] The multi-channel fusion attention module, i.e. the MCF module, is used to extract deep features, and the formula is:

[0183] (6.1);

[0184] wherein, is the MCF block. The MCF block is composed of three DISR blocks connected by skip connection, and the DISR block includes a dense connection module branch, a context attention module branch, and a statistical spatial channel attention module branch, and the dynamic weights of the three branches are determined by the input features; then, a 3x3 convolution kernel is used to extract features from , , to complete the deep feature extraction part.

[0185] The equation of image reconstruction is:

[0186] (6.2);

[0187] wherein, represents interpolation up-sampling, is the final SR output.

[0188] In the preferred scheme, as shown in Figure 4 and Figure 5As shown, in the semantic discriminator, the semantic module is used to extract accurate and rich semantic features of the infrared image; then, the extracted semantic features are combined with the discriminator by using the MST module.

[0189] In the semantic discriminator, the semantic module is a Modified Resnet-50, and the semantic features S of the third layer are extracted h Input MST.

[0190] Obtain semantics The MST is used to guide the discriminator semantically, and the extracted image semantic information is embedded into the feature space of the discriminator, which provides key guidance for discriminating the authenticity of the texture, so as to force the discriminator to pay attention to the distribution of the semantic perception texture, specifically:

[0191] A1: semantics After being transmitted to the self-attention module, the query Q is obtained :

[0192] (7.1);

[0193] In the formula, LN, GN, SA and RA are layer normalization, group normalization, self-attention module and rearrangement respectively.

[0194] A2: the generator generates SR images and the HR images in the HR data set are transmitted into the convolutional layer to obtain the original enhanced features, and the original enhanced features are rearranged to obtain the keys and and the values and of the SR images and the HR images respectively and Then, the query Q, the keys and and the values and

[0195] of the SR images and the HR images are input into the Cross Attention module. A3: the semantic perception image features output by the Cross Attention are connected with the original enhanced features to obtain the features and

[0196] of the final SR images and the HR images, and the formula is:

[0197] (7.3);

[0198] (7.4);

[0199] (7.5);

[0200] (7.6);

[0201] (7.7);

[0202] (7.8);

[0203] (7.9);

[0204] In the formula, and is the output after the multi-head attention mechanism of the Cross Attention module, and is the output after the BasicTransformerBlock, is a scale factor, is a feedforward network, is the original enhanced feature.

[0205] In step A1, the relationship between each pixel and other pixels is calculated by self-attention mechanism, and the semantic features are processed by the cross-attention module. This module cross-attends the semantic features with other related features of the image, focuses on the relationship between texture and semantics, and obtains a super-resolution / high-quality image. In step A2, the convolutional layer processes the features and generates key-value pairs, and in step A3, the output semantic perception image features are connected with the original enhanced features. Through feature connection, the final feature map will contain information from semantic perception and enhanced image, so as to obtain high-quality image features and improve the discrimination accuracy of the discriminator.

[0206] As shown in Figure 2 , it is the structure diagram of the Semantic Model. In this embodiment, the semantic module Semantic Model is based on Modified Resnet-50. Resnet-50 originally consists of four layers, but as the number of layers increases, the resolution of the features is down-sampled, and the semantics become more abstract.

[0207] The semantic module Semantic Model extracts accurate and rich semantic features of the infrared image. Then, the extracted semantic features are combined with the discriminator by using the MST, and the structure of the semantic discriminator is as shown in Figure 5 .

[0208] In the preferred solution, in the DISR module, the input features are first compressed using global average pooling, then pass through a connection layer containing two fully connected and one ReLU activation layer, and finally calculate the weight of each branch through the Softmax function.

[0209] In this embodiment, the semantic module is denoted as ϕ, aiming to realize more fine-grained semantic-aware texture generation, and the target is to:

[0210] (8.1).

[0211] In the adversarial training of this embodiment, the loss function is composed of pixel-level supervision loss , perception loss and adversarial loss. Among them, the pixel-level supervision loss is used to constrain the pixel-level consistency and ensure the realism of the pixel value; the perception loss uses the features of VGG to provide coarse-grained perceptual constraints between the generated image and the real image; and the adversarial loss is introduced through the adversarial training process. In the adversarial training process, a discriminator D is used to distinguish whether the image is real or fake, and is optimized through the following loss function.

[0212] In the preferred solution, the HR image dataset and the LR image dataset are used to train the generator, and in the adversarial training process of the MCFGAN model, the discriminator is used to distinguish whether the image is true or false, and the discriminator is optimized by the loss function, and the formula is:

[0213] (9.1);

[0214] In the formula, and are the distributions of high-quality images and generated images respectively, is the expectation, and D is the discriminator;

[0215] The generator is optimized by the combination of three losses, and the loss function includes L1 loss, perception loss and adversarial loss, and the formula is:

[0216] (9.2);

[0217] In the formula, and are the weight coefficients of the perception loss and the adversarial loss respectively, is the adversarial loss, is the perception loss, is the pixel-level supervision loss;

[0218] The goal of the adversarial loss is to make the generator deceive the discriminator, and the optimization goal is The purpose of the present application is to provide a method and a system for generating high-quality images from low-quality images.

[0219] (9.3);

[0220] wherein, is a desired, and respectively a high-quality image and a generated image distribution.

[0221] In this embodiment, the generator generates high-quality images by optimizing a combination of three losses, minimizing these losses to optimize the image quality, making it closer to the real image in terms of pixels, perceptual characteristics and adversarial, and being able to "deceive" the discriminator. The goal of the discriminator is to maximize this loss, thus correctly identifying the difference between the generated image and the real image. Finally, the generator and the discriminator constantly play against each other in adversarial training, pushing the quality of the generated images to improve.

[0222] After the above steps, the optimized MCFGAN model is obtained, and it is used for testing.

[0223] ​The above steps are verified in combination with actual pictures: two pictures are selected from the thermal infrared data set to predict using the optimized MCFGAN model, and then compared with the current existing super-resolution network, the existing super-resolution network is respectively: Bicubic (Bicubic Interpolation, bicubic interpolation), BSRGAN (a generative adversarial network for image super-resolution, generative adversarial network for image super-resolution), ESRGAN (Enhanced Super-Resolution Generative Adversarial Network, enhanced super-resolution generative adversarial network), SwinIR (Swin Transformer for Image Restoration, Swin Transformer for Image Restoration), PSRGAN (progressive super-resolution generative adversarial network, progressive super-resolution generative adversarial network), Real-ESARGAN (Real-World Enhanced Super-Resolution Generative Adversarial Network, Real-World Enhanced Super-Resolution Generative Adversarial Network), SRDGRL (A blind super-resolution framework with degradation reconstruction loss, A blind super-resolution framework with degradation reconstruction loss), DARSR (Unsupervised Degradation Representation Learning for Blind Super-Resolution, Unsupervised Degradation Representation Learning for Blind Super-Resolution); The data sets are CVC-14 (Visible-FIR Day-Night Pedestrian Sequence Dataset, Visible-FIR Day-Night Pedestrian Sequence Dataset), CVC-09, OSU TPD (OSU Thermal Pedestrian Database, OSU Thermal Pedestrian Database), OSU CTD (OSU Color-Thermal Database, Ohio State University Color-Thermal Database) and TWIRD (Terravic Motion IR Database, Terravic Motion IR Database).

[0224] The scale factor is 4 asFigure 7 with a scale factor of 2 as shown in Figure 8

[0225] 1) The optimized MCFGAN model can effectively remove noise while better restoring the detail information of the thermal infrared image, and the effect is optimal.

[0226] 2) Two indexes, NIQE (Natural Image Quality Evaluator) and BRISQUE (Blind / Referenceless Image Spatial Quality Evaluator), are used for objective analysis, as shown in Tables 1 and 2, and the technical scheme of the embodiment obtains the lowest NIQE and BRISQUE values.

[0227] Table 1 NIQE / BRISQUE values of different super-resolution algorithms under five data sets with a scale factor of 4

[0228]

[0229] Table 2 NIQE / BRISQUE values of different super-resolution algorithms under five data sets with a scale factor of 2

[0230]

[0231] In use, in the process of super-resolution, the embodiment effectively eliminates the noise in the thermal infrared image; the multi-channel fusion structure improves the information mining depth and the utilization rate of information in attention, thereby improving the efficiency of information fusion; and compared with other super-resolution networks, it can be concluded that the embodiment can generate more realistic image textures and achieve better visual effects.

[0232] The above embodiments are only preferred technical solutions of the present application, and should not be regarded as limitations of the present application. The protection scope of the present application should be based on the technical solutions recited in the claims, including equivalent replacement schemes of the technical features recited in the claims. That is, equivalent replacement improvements within this scope are also within the protection scope of the present application.​

Claims

1. A thermal infrared image optimization method based on multi-channel fusion and semantic information, characterized in that, The method comprises the following steps: Obtaining a VIS and a corresponding TIR dataset, and obtaining an HR image dataset after data preprocessing; Building an MCFGAN model; wherein the MCFGAN model comprises a generator module and a semantic discriminator with a multi-channel fusion attention module; the generator comprises three skip-connection multi-channel fusion attention modules, i.e. DISR modules; the semantic discriminator comprises a semantic module and a semantic perception fusion module; wherein the semantic module is used for generating semantic information, and the semantic perception fusion module is used for integrating the semantic information into the discriminator; Inputting the HR image dataset into an infrared degradation model to obtain an LR image dataset; Training the generator by using the HR image dataset and the LR image dataset to generate an SR image dataset, and then inputting the generated SR image dataset into the semantic discriminator to determine the true or false; In the semantic discriminator, the semantic module generates semantic information according to the HR image dataset, and the semantic information is integrated into the discriminator through the semantic perception fusion module, so as to perform adversarial training on the MCFGAN model and obtain an optimized MCFGAN model; Inputting a to-be-tested TIR picture into the optimized MCFGAN model to obtain an optimized thermal infrared image.

2. The method of claim 1, wherein, The data preprocessing comprises the following steps: performing gray processing on the VIS and the TIR dataset to generate an HR image dataset, and then performing CLAHE processing on the HR image dataset; The core of the CLAHE contrast limited adaptive histogram equalization processing lies in local histogram equalization and contrast limitation, and the formula of the local histogram equalization is: (1.1)； wherein, is the number of pixels in the local region having a gray value is the cumulative sum of the number of all pixels having a gray value less than or equal to and is the pixel value of the image;​​ The local histogram equalization is transformed by linear transformation to make the pixel value distribution of the image as uniform as possible, and the formula is: (1.2)； wherein, is a pixel value of the original image at position , is a pixel value of the new image after the local histogram equalization processing at position , is a cumulative distribution function value of the minimum pixel value in the local histogram, is a cumulative distribution function value of the maximum pixel value in the local histogram, is a pixel value in the local histogram, is a maximum value of the new pixel value range; The formula of the contrast limitation is: (1.3)； In the formula, is the frequency value of the local histogram after contrast limited processing, is the histogram of the local region, and ClipLimit is the contrast limited parameter.

3. The method of claim 1, wherein, In the semantic discriminator, the semantic module Semantic Model is used for extracting accurate and rich semantic features of the infrared image; and then the extracted semantic features are combined with the discriminator by using the semantic perception fusion module; Among them, the semantic module is modified Resnet-50, and the semantic features S of the third layer are extracted h The input semantic perception fusion block discriminates the semantic guidance of the discriminator, embeds the extracted image semantic information into the feature space of the discriminator, provides key guidance for the discriminator to identify the authenticity of the texture, and thus forces the discriminator to pay attention to the distribution of the semantic perception texture. Specifically: A1: semantics After passing to the self-attention module, the query is obtained (7.1)； In the formula, LN, GN, SA and RA respectively represent layer normalization, group normalization, self-attention module and rearrangement; A2: the generator generates the SR image and the HR image in the HR dataset are passed into the convolutional layer to get the original enhanced feature, which is rearranged to get the key of the SR image and the HR image respectively and and the value and Then the query Q, the key and the value of the SR image and the HR image and and the value and are input into the Cross Attention module; A3: connect the semantic perception image features output by Cross Attention with the original enhanced features to obtain the features of the final SR image and the HR image and , the formula is: (7.2)； (7.3)； (7.4)； (7.5)； (7.6)； (7.7)； (7.8)； (7.9)； wherein, is the output after the multi-head attention mechanism of the Cross Attention module, and is the output after the BasicTransformerBlock, is a scale factor, is a feed-forward network, is the original augmented feature.

4. The method of claim 1, wherein, The generator module comprises a shallow feature extraction module, a multi-channel fusion attention module for deep feature extraction and a reconstruction module, and the formula is: The shallow feature extraction module uses a convolution layer with a 3x3 kernel, with the formula: , is the extracted feature map; The multi-channel fusion attention module, i.e. the MCF module, is used for extracting deep features, and the formula is: (6.1)； wherein is the MCF block, the MCF block consisting of three DISR blocks connected in series with three skips, is the output of the nth DISR block; The DISR block comprises a densely connected module branch, a context attention module branch and a statistical spatial channel attention module branch, and the dynamic weights of the three branches are determined by the input features; then, features are extracted from the using a 3x3 convolution kernel: , completing the depth feature extraction part; The equation of the image reconstruction is: (6.2)； wherein is the interpolated up-sampling, is the final SR output.

5. The method of claim 1, wherein, Each DISR module generates weights for three independent channels by using the same input features, and the formula of the specific operation is: (5.1)； (5.2)； (5.3)； (5.4)； (5.5)； (5.6)； wherein, is the dense connected output, is the output of the contextual attention module, is the output of the statistical spatial channel attention module; is the weight value of the three channels generated by the dynamic weight module according to the input features.

6. The method of claim 5, wherein, Each DISR module controls the balance by weighted summation, and the dynamic weighted contribution of the dense connection module, the context attention module and the statistical spatial channel attention module is included; The dense connection module Dense is adopted in the DISR module, and the formula is: (2.1)； (2.2)； (2.2)； (2.4)； (2.5)； (2.6)； (2.7)； (2.8)； (2.9)； (2.10)； In the formula, is a convolution layer, is an activation function, is a feature obtained by layer-by-layer convolution processing and splicing with the previous layer, Y is the final output, is a weighting coefficient.

7. The method of claim 5, wherein the method further comprises: The context attention module is adopted in the DISR module, and the formula is as follows: (3.1)； (3.2)； (3.3)； (3.4)； (3.5)； (3.6)； (3.7)； wherein, is a global feature, is a compressed feature, is a horizontal context information, is a vertical context information, is a length of a horizontal convolution kernel, is a height of a vertical convolution kernel, is a feature remapped to the original channel dimension, is an attention weight, is a final output, is an element-wise multiplication.

8. The method of claim 5, wherein, In the DISR module, the statistical spatial channel attention module has the following specific formula: (4.1)； (4.2)； (4.3)； (4.4)； (4.5)； (4.6)； wherein is the mean value within the channel, is the square of the deviation from the mean value, is the stable normalization factor with a regularization parameter added to the normalization factor, is the normalization factor, is a regularization parameter to prevent the denominator from being too small when calculating the normalization factor, is the attention weight.

9. The method of claim 5, wherein, In the DISR module, the input features are first compressed by using the global average pooling, then pass through the connection layer containing two fully connected layers and a ReLU activation layer, and finally the weights of each branch are calculated by the Softmax function.

10. The method of claim 1, wherein, The HR image dataset and the LR image dataset are used to train the generator, and in the MCFGAN model adversarial training process, the discriminator is used to distinguish whether the image is true or false, and the discriminator is optimized by using a loss function, and the formula is: (9.1)； wherein and are high-quality images and generated images are distributions, is a desire, and D is a discriminator. The generator is optimized by combining three losses, and the loss function includes L1 loss, perceptual loss and adversarial loss, and the formula is: (9.2)； wherein, and are the weight coefficients of perceptual loss and adversarial loss, respectively, is the adversarial loss, is the perceptual loss, is the pixel-level supervision loss; adversarial loss The goal of the generator is to fool the discriminator, the optimization objective is to The goal of the discriminator is the opposite: (9.3)； wherein is desired, and are high quality images and generated images respectively.

Images