Method for enhancing short-exposure images of inner wall of boiler drum based on transformer

Abstract

The present application relates to the technical field of image enhancement, in particular to a drum inner wall short exposure image enhancement method based on a transformer, which comprises the following steps: step 1, constructing a drum inner wall short exposure image dataset; step 2, constructing a short exposure image enhancement network based on a transformer, wherein the network comprises a preprocessing part, a feature extraction part, a transformer encoding-decoding structure part and an up-sampling reconstruction output part; the present application can improve the visual perception quality of captured data in the insufficient illumination scene of the drum inner wall, obtain more information, effectively improve the image brightness, restore the image details and remove the image noise.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image enhancement, in particular to a short-exposure image enhancement method for the inner wall of a kettle based on a Transformer. BACKGROUND

[0002] Image enhancement technology is widely used in industrial fields, especially in detection, recognition and analysis, to improve the understanding and interpretation of images in complex environments. In some application scenarios, such as pipe cylinder inner wall detection, due to the existence of low light conditions, the images collected often have problems such as detail loss, severe noise and color distortion, which seriously affect the visibility and recognizability of the images.

[0003] Short-exposure image enhancement technology is an important research direction in the field of computer vision, aiming to improve the quality of images acquired under insufficient lighting conditions. Traditional image enhancement methods mainly adjust the image to improve its visual quality through spatial filtering, sharpening techniques, histogram adjustment, wavelet transform, transform domain methods and contrast enhancement. Among them, spatial filtering uses mean or median filtering to smooth the image, sharpening methods highlight edge information, histogram adjustment enhances contrast through distribution adjustment, wavelet transform decomposes the image into different frequency wavelet coefficients, and transform domain methods such as Fourier transform operate in the frequency domain. Contrast enhancement methods adjust the gamma value or pixel mapping of the image through gamma correction and other means to improve the visual effect of the image. Under low light conditions in the pipe cylinder, traditional image enhancement methods have a series of problems. For example, the image may be disturbed by noise, details may be lost, and contrast may be reduced, and traditional methods often cannot effectively handle these problems, making it difficult to obtain clear and accurate pipe cylinder inner wall images in low light environments. SUMMARY

[0004] To solve the above technical problems, the present application provides a short-exposure image enhancement method for the inner wall of a kettle based on a Transformer, which can improve the visual perception quality of captured data in the insufficient lighting scenario of the inner wall of the kettle, to obtain more information, improve image brightness, restore image details and remove image noise.

[0005] The present application adopts the following technical solution: a short-exposure image enhancement method for the inner wall of a kettle based on a Transformer, the method being as follows:

[0006] Step 1, constructing a short-exposure image dataset for the inner wall of a pipe cylinder;

[0007] Step 2, constructing a short-exposure image enhancement network based on a Transformer, the network including a preprocessing part, a feature extraction part, a Transformer encoding-decoding structure part, and an up-sampling reconstruction output part; the following formula is used to express:

[0008] I1 = γ (F1 (I in ) - B)

[0009] I pre = F t [ F cnn (I1) + P]

[0010] In the formula, I1 is the image after the preprocessing part is processed, γ is a learnable magnification coefficient, I in is the input image data, F1 represents extracting four-channel data of the original raw image and performing normalization processing, B is a learnable parameter for removing some noise in the image and improving contrast, F cnn represents a feature extraction part, P represents a fixed position parameter, F t represents a Transformer encoding and decoding structure part and a last image upsampling and reconstruction output part, I pre represents the finally obtained enhanced image;

[0011] Step 3, a network loss function is constructed, and the learnable parameter B in the enhancement network in step 2 is constrained based on the loss function, so that the enhancement network can finally output a normal light image;

[0012] Step 4, input the data set in step 1 into the enhancement network in step 2 for training, the short exposure image in the data set is taken as the network input, and the long exposure image is taken as the reference image, so as to train the enhancement network and obtain the best model after training;

[0013] Step 5, the best model is imported into the enhancement network, the short exposure image is input, and the short exposure image is enhanced and output by using the best model.

[0014] Preferably, the step 1 is further specifically as follows:

[0015] Step 11, a camera capable of shooting raw format is selected to collect the images of the inner wall of the boiler drum, the collected images are divided into long exposure images and short exposure images, and the long exposure image and the short exposure image of a picture are a group;

[0016] Step 12, for the same group, only one long exposure image needs to be collected, and the exposure time is fixed according to the scene; the short exposure image is collected in the following way: starting from the shortest exposure time, the fixed exposure time is increased in turn, one short exposure image is collected each time, and a total of 10 short exposure images are collected, and the exposure time of each short exposure image is recorded in the file name;

[0017] In the same group of images, all the parameters are the same except the exposure time.

[0018] Preferably, in step 2, the network preprocessing part is further specifically:

[0019] Step 211, the original raw format image, i.e. the short exposure image, is extracted to obtain the data of its gbrg four channels, and normalized processing is performed, and the normalization formula is:

[0020]

[0021] x max is 65535, i.e. the maximum value in the short exposure image, x min is the minimum value in the short exposure image, x is the pixel value of the gbrg four channels of the short exposure image, and the size of the normalized image is h x w x 4, denoted as I s ; wherein h and w are the height and width of the normalized image, respectively;

[0022] Step 212, two learnable parameters γ and B are defined in the network, and the shape size is h x w x 4, γ represents the magnification coefficient of the short exposure image, and B is used to remove the micro noise in the image and increase the contrast of the image;

[0023] Step 213, γ(I s -B) operation is performed on the image data of the gbrg four channels, and a preprocessed image with a size of h x w x 4 is obtained, denoted as I1.

[0024] Preferably, in step S2, the feature extraction part is further specifically:

[0025] Step 221, I1 is sent to the feature extraction part;

[0026] Step 222, a first convolutional layer is built, the convolutional kernel size is 6 x 6, s = 2, p = 2, s represents the step length of the convolution operation, p represents the padding number of the feature map, and 2 represents padding two pixels in each of the up, down, left and right; the output channel is 64, and after the convolutional layer outputs, the output is subjected to batch normalization (BN) processing, and then the output is activated by a SiLU activation function, and the formula of the SiLU activation function is:

[0027]

[0028] wherein silu(x) is the output of the activation function, and x is the input value;

[0029] After the first convolutional layer, the feature map size is directly reduced by half, the dimension is changed to 64, and the feature map size is changed to (h / 2) x (w / 2) x 64;

[0030] Step 223, build the second layer of convolutional layer, the convolution kernel size is 3*3, s=2, p=2, the output channel is 128, the feature map size is reduced by half after the second layer of convolution, the dimension is changed to 128, and the feature map size is changed to (h / 4) * (w / 4) * 128;

[0031] Step 224, capture the global information in the image through Non-local Operation to better understand the contextual relationship and long-range dependency in the image, the specific form of Non-local Operation is expressed as:

[0032]

[0033] Wherein, x is the input feature map, j represents the output position, the response is calculated by enumerating i and then calculating, i represents all possible positions on the feature map, h(x i ,x j ) function calculates the similarity of i and j, G(x i ) function calculates the representation of the feature map at position i, C(x i ) is the standardization factor, and its calculation formula is:

[0034]

[0035] The specific implementation steps of channel attention are as follows:

[0036] a. Squeeze: first, do a GlobalPool operation on the input feature layer, that is, global average pooling, compress the two-dimensional feature (h*w) of each channel into a number, and get a 1*1*C feature strip, the length is the channel number C of the input feature map;

[0037] b. Calculate the attention weight: apply two fully connected layers to linearly transform the compressed feature data, and the first fully connected layer is followed by a ReLU activation function to increase nonlinearity;

[0038] c. Use the Sigmoid function to normalize each weight to 0-1, and its expression is:

[0039]

[0040] d. Apply the weight coefficient to each channel in the form of multiplication to get a new feature map, which remains unchanged in size and is still h*w*c;

[0041] The non-local operation captures global information in the image, and outputs the global information as input to the channel attention module. Finally, the original input and the output of the channel attention module are connected in residual connection, which alleviates the gradient vanishing problem. The hybrid attention module can guide the network to better learn color information and suppress noise, and obtain better enhancement effect in the polar light environment.

[0042] Step 225, build a feature extraction module C1. First, input the image through two convolutional layers with a kernel size of 1x1, s=1, p=0, and an output channel number of 64. Then, embed the MAB module. The output of one of the convolutional layers is used as the input of the MAB module. The output of the other convolutional layer is concatenated with the output of the MAB module. The output channel number is changed to 128 again. Finally, embed the image through a convolutional layer to increase the dimension. The output channel number is 128. After passing through the C1 module, the feature map size is still (h / 4) x (w / 4) x 128.

[0043] Step 226, embed 3 layers of MAB modules in the C1 module in a cascade manner to form the first feature extraction module, which extracts the shallow information of the image. The output feature map size is (h / 4) x (w / 4) x 128.

[0044] Step 227, build a feature extraction module C2. The structure of the C2 module is the same as that of the C1 module, except that the input and output channel numbers of the C2 module are changed to 256. The output feature map size is (h / 8) x (w / 8) x 256.

[0045] Step 228, embed 6 layers of MAB modules in the C2 module in a cascade manner to extract the deep information of the image. The output feature map of the C2 module is further processed through a convolutional layer with a kernel size of 3x3, s=2, p=1, and an output channel of 512. The final output feature map size of the feature extraction part is (h / 16) x (w / 16) x 512.

[0046] Preferably, in step 2, the Transformer encoding-decoding structure part is further specified as follows:

[0047] Step 231, perform position encoding on the feature map output by the feature extraction part. The two-dimensional absolute position encoding method is used for the feature map, and the specific formula is as follows:

[0048]

[0049]

[0050] wherein, PE (pos,2i) is an even dimension, PE(pos,2i+1) is odd; pos represents the position index of one channel information in the feature map, i represents the dimension index, the even dimension is calculated using the sin function, and the odd dimension is calculated using the cos function, d model represents the total dimension, d model = 512 / 2, that is, the first 256 dimensions represent row position information, and the last 256 dimensions represent column position information; the row position information and the column position information are spliced together to obtain position information with the same size as the input feature map, which is (h / 16)×(w / 16)×512, and is denoted as I p

[0051] Step 232, the calculated position information is embedded in the input feature map in the form of addition, at this time the feature map has position information, and the size is (h / 16)×(w / 16)×512;

[0052] Step 233, a Self-Attention structure is built, that is, a self-attention structure, and the calculation process of Self-Attention can be described as follows:

[0053]

[0054] wherein Q, K, and V all represent the input of Self-Attention, which are the same value, and the input of Self-Attention is I p , d k is the dimension of the feature map, and T is the transpose operation; Self-Attention can help the model better understand the context information in the sequence and establish global dependency relationships, so as to more accurately process sequence data;

[0055] Step 234, a Multi-Head Attention module is built, that is, a multi-head self-attention structure, which is composed of multiple Self-Attention, the input sequence is decomposed into multiple parts, and independent attention calculation is performed on each part, and then the results are linearly transformed to obtain the final output;

[0056] Step 235, a Feed Forward (FFN) layer is built, and the structure of the Feed Forward layer includes linear transformation, activation function and linear transformation; first, the input data is linearly transformed to map the data to a high-dimensional space; then, the activation function is used to increase the nonlinear expression ability of the model; finally, the data is linearly transformed again to map the features in the high-dimensional space back to the original space to obtain the output of the feed forward layer, and the calculation formula is as follows:

[0057] max(0,XW1+b1)W2+b2​

[0058] Where X is input data, W1, b1 are the weight and bias of the first fully connected layer, W2, b2 are the weight and bias of the second fully connected layer, the first fully connected layer uses ReLU function for activation, and the second fully connected layer does not activate;

[0059] Step 236, combine Multi-Head Attention module and FFN to build Encoder coding network;

[0060] Step 237, cascade 6 identical Encoder modules to form the encoding part;

[0061] Step 238, build Decoder module, Decoder contains two Multi-Head Attention components, the input Q, K, V of the first Multi-Head Attention is a learnable parameter, the K, V of the second Multi-Head Attention is the output of the Encoder coding information, and the Q is the output of the first Multi-Head Attention, which is sent into a FFN layer after two Multi-Head Attention to improve the expression ability of the model;

[0062] Step 239, cascade 3 identical Decoder modules to form the decoding part.

[0063] Preferably, in step 2, the Transformer encoding-decoding structure part is further specifically:

[0064] Step 241, the decoding network contains 3 Decoders, each Decoder module contains a group of outputs, and the three groups of outputs are spliced in the channel dimension to obtain the output I d ;

[0065] Step 242, input I d into a FFN layer to do linear transformation on the output;

[0066] Step 243, finally, the output is up-sampled by the PixelShuffle method to obtain the final reconstructed image.

[0067] Preferably, the step 3 is further specifically: the loss function expression is specifically:

[0068] L = (1-λ1-λ2)L1+λ1L ssim +λ2L perc

[0069] Where L represents the total loss function, Lssim L is a structural similarity loss ssim L is a perceptual loss; λ1, λ2 represent the weights of L ssim L is a perceptual loss; λ1, λ2 represent the weights of L perc L is a perceptual loss; λ1, λ2 represent the weights of L

[0070] Specifically, the function expression of L1 loss is:

[0071]

[0072] Wherein, I gt is the real image data, I h is the predicted image data, and δ is a constant not equal to 0.

[0073] Specifically, the function expression of L ssim loss is:

[0074]

[0075] Wherein, μ x , μ y represent the mean of the real image and the predicted image, σ represents the variance of the real image and the predicted image, and σ xy represents the covariance of the two, C1, C2 are constants not equal to 0; N is the sum of the number of all pixels of the image.

[0076] L perc represents the perceptual loss, which is calculated by the pre-trained network to calculate the difference between the two pictures. The perceptual loss is calculated by using the vgg16 pre-trained network. The calculation method of the perceptual loss is that the input image and the target image are respectively input into the pre-trained neural network to obtain their feature representations in the network; then the feature representations are input into the loss function to calculate the Euclidean distance between them, and the target is to minimize the distance between the input image and the target image in the feature space; the function expression is:

[0077]

[0078] Wherein, I gt , I h are the real image and the predicted image, The function calculates the feature output of the image after passing through the vgg16 network, H j , W j represent the height and width of the jth feature layer, and C j represents the channel.

[0079] Preferably, in step 4: in the process of training, the short exposure image and the long exposure image in the data set are randomly data enhanced, and the enhanced short exposure image is input into the network.

[0080] Preferably, the data enhancement is performed by random rotation and mirroring, the rotation includes 90, 180, 270 and 360 degree rotation of the image, and the probability of each random rotation is 0.25; the mirroring includes vertical mirroring and horizontal mirroring, and the probability of the mirroring is 0.5, and the probability of whether each data enhancement method is used in the training process is 0.5.

[0081] Preferably, in the step 4, the size of the short exposure image input into the network is the short exposure image after random cropping, the cropping size is 512*512, and the image size finally input into the network is 4*512*512.

[0082] The present application has the following beneficial effects:

[0083] The present application constructs a short exposure image enhancement network, which includes a preprocessing part, a feature extraction part, a Transformer encoding-decoding structure part, and an up-sampling reconstruction output part, the preprocessing part can adaptively process images under different light conditions by introducing two learnable parameters; the feature extraction part adopts a hybrid attention module combining channel attention and Non-local Operation, which can focus on local information and global information at the same time; the Transformer encoding-decoding structure part adopts a self-attention mechanism to effectively process the global dependence of sequences, and the position information is embedded on the input of the Transformer Encoder layer, so that the network can perceive the position relationship in the image; the network loss function introduces a joint loss function of L1 loss, SSIM loss and Perceptual loss, so that the network can fit the parameters from multiple aspects.

[0084] The present application can improve the visual perception quality of the captured data in the insufficient light scene of the inner wall of the pot barrel, obtain more information, and effectively improve the image brightness, restore the image details and remove the image noise. BRIEF DESCRIPTION OF DRAWINGS

[0085] Figure 1 is a schematic diagram of the overall structure of the enhancement network.

[0086] Figure 2 is a schematic diagram of the Non-local Operation structure.

[0087] Figure 3 is a MAB hybrid attention module.

[0088] Figure 4 is a schematic diagram of the feature extraction module.

[0089] Figure 5is a schematic diagram of a Transformer encoding-decoding structure. DETAILED DESCRIPTION

[0090] The application will be further described below with reference to the accompanying drawings.

[0091] As Figures 1 to 5 , the application provides a short-exposure image enhancement method for the inner wall of a boiler drum based on a Transformer, and the method is as follows:

[0092] Step 1, constructing a short-exposure image data set for the inner wall of a tube drum;

[0093] Step 2, constructing a short-exposure image enhancement network based on a Transformer, which is equivalent to a single-stage processing flow, taking a short-exposure image as input and directly outputting an enhanced image, and the network comprises a preprocessing part, a feature extraction part, a Transformer encoding-decoding structure part, and an up-sampling reconstruction output part; the following formula is used for expression:

[0094] I1=γ(F1(I in )-B)

[0095] I pre =F t [F cnn (I1)+P]

[0096] In the formula, I1 is an image processed by the preprocessing part, γ is a learnable amplification coefficient, which is equivalent to an illumination estimator, estimating the illumination distribution under different dark light conditions, and the low-light image is roughly enhanced after illumination estimation, I in is an input image data, F1 represents extracting four-channel data of an original raw image and performing normalization processing, B is a learnable parameter, which is used to remove some noise in the image and improve the contrast, F cnn represents the feature extraction part, P represents a fixed position parameter, F t represents the Transformer encoding-decoding structure part and the up-sampling reconstruction output part of the final image, and I pre represents the finally obtained enhanced image;

[0097] Step 3, constructing a network loss function, based on which the learnable parameter B in the enhancement network in step 2 is constrained, so that the enhancement network can finally output an image with normal illumination;

[0098] Step 4, inputting the data set in step 1 into the enhancement network in step 2 for training, taking the short-exposure image in the data set as the network input and the long-exposure image as the reference image, so as to train the enhancement network and obtain the best trained model;

[0099] Step 5, import the best model into the enhanced network, input the short exposure image, and use the best model to enhance the short exposure image and output.

[0100] The step 1 is further specifically as follows:

[0101] Step 11, a camera capable of shooting raw format is selected to collect the image of the inner wall of the boiler drum, the collected image is divided into a long exposure image and a short exposure image, and the long exposure image and the short exposure image of a picture are a group;

[0102] Step 12, for the same group, only one long exposure image needs to be collected, and the exposure time is fixed according to the scene; the short exposure image is collected in the following way: starting from the shortest exposure time, the fixed exposure time is increased in turn, one short exposure image is collected each time, and a total of 10 short exposure images are collected, and their exposure times are recorded in the file name;

[0103] In the same group of images, all other parameters are the same except the exposure time.

[0104] In the step 2, the network preprocessing part is further specifically as follows:

[0105] Step 211, the original raw format image, i.e. the short exposure image, is extracted to obtain the data of its gbrg four channels, and normalized processing is performed, and the normalization formula is:

[0106]

[0107] x max is 65535, i.e. the maximum value in the short exposure image, x min is the minimum value in the short exposure image, and x is the pixel value of the gbrg four channels of the short exposure image, and the size of the normalized image is h x w x 4, denoted as I s ; wherein h and w are the height and width of the normalized image, respectively;

[0108] Step 212, two learnable parameters γ and B are defined in the network, and the shape size is h x w x 4, γ represents the magnification coefficient of the short exposure image, and B is used to remove the small noise in the image and increase the contrast of the image;

[0109] Step 213, γ(I s -B) operation is performed on the image data of the gbrg four channels to obtain the preprocessed image, and the size is h x w x 4, denoted as I1.

[0110] In the step S2, the feature extraction part is further specifically as follows:

[0111] Step 221, I1 is sent to the feature extraction part;

[0112] Step 222, build the first layer of convolutional layer, the convolution kernel size is 6x6, s=2, p=2, s represents the step length of convolution operation, p represents the padding number of feature map, 2 indicates that each of up, down, left and right is padded with two pixels; the output channel is 64, after the output of the convolutional layer, the output is processed by batch normalization, that is, Batch Normalization (BN), and then the output is activated by SiLU activation function, the formula of SiLU activation function is:

[0113]

[0114] Wherein, silu(x) is the output of the activation function, x is the input value;

[0115] After the first layer of convolution, the feature map size is directly reduced by half, the dimension becomes 64, and the feature map size becomes (h / 2)x(w / 2)x64;

[0116] Step 223, build the second layer of convolutional layer, the convolution kernel size is 3x3, s=2, p=2, the output channel is 128, after the second layer of convolution, the feature map size is reduced by half, the dimension becomes 128, and the feature map size becomes (h / 4)x(w / 4)x128;

[0117] Step 224, capture global information in the image through Non-local Operation (i.e. non-local operation) to better understand the context relationship and long-range dependence in the image, the specific form of Non-local Operation is expressed as:

[0118]

[0119] Wherein, x is the input feature map, j represents the output position, its response is calculated by enumerating i and then calculating, i represents all possible positions on the feature map, h(x i ,x j ) function calculates the similarity of i and j, G(x i ) function calculates the representation of feature map (i.e. network feature map, representing the output after a certain layer of convolution in the network) at position i, C(x i ) is the standardization factor, and its calculation formula is:

[0120]

[0121] The specific implementation steps of channel attention are as follows:

[0122] a. Squeeze (i.e. global compression, compressing three-dimensional information into one dimension): first, a Global Pool operation is performed on the input feature layer, i.e. global average pooling, which compresses the two-dimensional features (h*w) of each channel into a number, obtaining a 1x1xC feature strip, the length of which is the channel number C of the input feature map;

[0123] b. Calculate attention weight: apply two fully connected layers to linearly transform the compressed feature data, and a ReLU activation function after the first fully connected layer to increase nonlinearity;

[0124] c. Normalize each weight to 0-1 using the Sigmoid function, and the expression is:

[0125]

[0126] d. Apply the weight coefficient to each channel in the form of multiplication to obtain a new feature map, which remains unchanged in size and is still h*w*c;

[0127] The Non-local operation captures global information in the image, and then outputs it as input to the channel attention module. Finally, the original input and the output of the channel attention module are connected in residual, which alleviates the gradient vanishing problem. By using the hybrid attention module, the network can better learn color information and suppress noise, and better enhancement effect can be obtained in the polar light environment.

[0128] Step 225, build a feature extraction module C1, first pass the input through two convolutional layers, the convolutional kernel size is 1x1, s=1, p=0, and the output channel number is 64, which reduces the dimension of the feature, then embed the MAB module, take the output of one of the convolutional layers as the input of the MAB module, and then concatenate the output of the other convolutional layer with the output of the MAB module, the output channel number becomes 128 again, and finally pass the image through a convolutional layer to increase the dimension, the output channel number is 128. After passing through a C1 module, the feature map size is still (h / 4) x (w / 4) x 128;

[0129] Step 226, embed 3 layers of MAB modules in the C1 module in a cascade manner to form the first feature extraction module, which extracts the shallow information of the image, and the output feature map size is (h / 4) x (w / 4) x 128;

[0130] Step 227, build a feature extraction module C2, the structure of the C2 module is the same as that of the C1 module, the difference is that the input and output channel number of the C2 module becomes 256, and the output feature map size is (h / 8) x (w / 8) x 256;

[0131] Step 228, embed the 6-layer MAB module in the C2 module in a cascade manner to extract deep information in the image, and the C2 module outputs a feature map which is then subjected to a convolution layer with a convolution kernel size of 3*3, s=2, p=1, and an output channel of 512; the size of the feature map finally output by the feature extraction part is (h / 16)*(w / 16)*512;

[0132] In step 2, the Transformer encoding-decoding structure part is further specifically as follows:

[0133] Step 231, position encoding is performed on the feature map output by the feature extraction part, and the two-dimensional absolute position encoding method is used for the feature map, and the specific formula is as follows:

[0134]

[0135] wherein, PE (pos,2i) is an even dimension, and PE (pos,2i+1) is an odd dimension; pos represents the position index of one channel information in the feature map, i represents the dimension index, the even dimension is calculated by using a sin function, the odd dimension is calculated by using a cos function, d model represents the total dimension, d model of the application = 512 / 2, that is, the first 256 dimensions represent row position information, and the last 256 dimensions represent column position information; the row position information and the column position information are spliced together to obtain position information with the same size as the input feature map, that is, (h / 16)*(w / 16)*512, which is denoted as I p ;

[0136] Step 232, the calculated position information is embedded into the input feature map in the form of addition, and at this time, the feature map has position information, and the size is (h / 16)*(w / 16)*512;

[0137] Step 233, build a Self-Attention structure, that is, a self-attention structure, and the calculation process of Self-Attention can be described as follows:

[0138]

[0139] wherein, Q, K, and V all represent the input of Self-Attention, which are the same value, the input of Self-Attention is I p , and d k is the dimension of the feature map, and T is a transpose operation; Self-Attention can help the model better understand the context information in the sequence, establish global dependency relationship, and thus more accurately process sequence data;

[0140] Step 234: Build a Multi-Head Attention module, which is a multi-head self-attention structure composed of multiple Self-Attention components. It decomposes the input sequence into multiple heads, performs independent attention calculations on each part, and then performs a linear transformation on the results to obtain the final output.

[0141] Step 235: Construct the Feed Forward (FFN) layer. The structure of the Feed Forward layer includes a linear transformation, an activation function, and another linear transformation. First, the input data undergoes a linear transformation, mapping the data to a high-dimensional space. Then, it passes through an activation function to increase the model's non-linear expressive power. Finally, another linear transformation is performed to map the features from the high-dimensional space back to the original space, obtaining the output of the feedforward layer. Its calculation formula is as follows:

[0142] max(0,XW1+b1)W2+b2

[0143] Where X is the input data, W1 and b1 are the weight biases of the first fully connected layer, W2 and b2 are the weights and biases of the second fully connected layer, the first fully connected layer is activated using the ReLU function, and the second fully connected layer is not activated;

[0144] Step 236: Combine the Multi-Head Attention module and FFN to build the Encoder network;

[0145] Step 237: Cascade six identical Encoder modules to form the encoding part;

[0146] Step 238: Build the Decoder module. The Decoder consists of two Multi-Head Attention layers. The inputs Q, K, and V of the first Multi-Head Attention are learnable parameters. The K and V of the second Multi-Head Attention are the outputs of the Encoder, and Q is the output of the first Multi-Head Attention. After passing through two Multi-Head Attention layers, the data is fed into an FFN layer to improve the model's expressive power.

[0147] Step 239: Cascade three identical Decoder modules to form the decoding section.

[0148] In step 2, the Transformer encoding-decoding structure is further specified as follows:

[0149] Step 241, decoding network contains 3 Decoders, each Decoder module contains a set of outputs, and the three sets of outputs are spliced in the channel dimension to obtain output I d ;

[0150] Step 242, I d is input into an FFN layer, and a linear transformation is performed on the output;

[0151] Step 243, finally, the output is up-sampled by the PixelShuffle method to obtain the final reconstructed image.

[0152] The step 3 is further specifically that the loss function expression is specifically:

[0153] L = (1-λ1-λ2)L1+λ1L ssim +λ2L perc

[0154] Wherein L represents the total loss function, L ssim is the structural similarity loss, L ssim is the perceptual loss; λ1, λ2 represent the weight coefficients of L ssim loss and L perc loss respectively.

[0155] Specifically, the function expression of the L1 loss is:

[0156]

[0157] Wherein I gt is the real image data, I h is the predicted image data, and δ is a constant not equal to 0;

[0158] Specifically, the function expression of the L ssim loss is:

[0159]

[0160] Wherein, μ x , μ y represent the mean of the real image and the predicted image respectively, represent the variance of the real image and the predicted image, σ xy represents the covariance of the two, C1, C2 are constants not equal to 0; N is the sum of all pixel quantities of the image.

[0161] L percThe perceptual loss represents a perceptual loss, which is calculated by a pre-trained network to calculate the difference between two images, and a VGG16 pre-trained network is used, and the calculation method of the perceptual loss is that the input image and the target image are respectively input into the pre-trained neural network to obtain their feature representations in the network; then the feature representations are input into a loss function to calculate the Euclidean distance between them, and the target is to minimize the distance between the input image and the target image in the feature space; and the function expression is:

[0162]

[0163] Wherein, I gt , I h are a real image and a predicted image, The function calculates the feature output of the image after the VGG16 network, H j , W j represent the height and width of the jth feature layer, and C j represents the channel.

[0164] In step 4, the short exposure images and the long exposure images in the data set are randomly data-augmented in the training process, and the short exposure images and the long exposure images are paired in the training stage and need to be augmented at the same time, and the augmented short exposure images are input into the network.

[0165] The data augmentation method is random rotation and mirroring, the rotation method includes rotating the image by 90, 180, 270 and 360 degrees, and the probability value of each random rotation method is 0.25; the mirroring method includes vertical mirroring and horizontal mirroring, and the probability value of the mirroring method is 0.5, and the probability value of whether each data augmentation method is used in the training process is 0.5.

[0166] In step 4, the size of the short exposure image input into the network is the short exposure image after random cropping, and the cropping size is 512*512, and the final image size input into the network is 4*512*512.

[0167] The application will be further described below in combination with a specific embodiment:

[0168] A short exposure image enhancement method for the inner wall of a boiler drum based on a Transformer,

[0169] 1. Constructing a short exposure image data set for the inner wall of a boiler drum

[0170] In a specific environment, a camera capable of shooting raw format is selected to collect images of the inner wall of a boiler drum, and the collected images are divided into long exposure images and short exposure images, and for a specific boiler drum, the long exposure and short exposure images of a picture are a group;

[0171] For the same group, only one long exposure time image needs to be collected, and the exposure time is fixed according to the scene. The short exposure time image collection method is to start from the shortest exposure time, and then increase the fixed exposure time in turn. Each time, a short exposure time image is collected, and a total of 10 short exposure time images are collected. The exposure time is recorded respectively and recorded in the file name;

[0172] The collection of the same group of images needs to control the variable, in addition to the different exposure time, other parameters such as contrast, ISO and other parameters are the same;

[0173] A total of 5000 groups of images are collected, and the data set is randomly divided into training set, validation set and test set, and the proportion is 45:1:4;

[0174] 2, build a short exposure image enhancement network based on Transformer, the network includes preprocessing part, feature extraction part, Transformer encoding-decoding structure, up-sampling reconstruction output several parts, the principle of network reconstruction short exposure image can be described by the following formula:

[0175] I1=γ(F1(I in )-B)

[0176] I pre =F t [F cnn (I1)+P]

[0177] In the formula, I1 is the preprocessed image, γ is the learnable amplification coefficient, I in is the input image data, F1 represents extracting four-channel data of the original raw image and doing normalization processing, B is a learnable parameter, which is used to remove some noise in the image and improve the contrast, F cnn represents the feature extraction part of the network, P represents the fixed position parameter, and F t represents the Transformer encoding-decoding structure and the final image reconstruction output part;

[0178] Step 2 is specifically:

[0179] 1) Network preprocessing part, including the following steps:

[0180] 11) Extract the gbrg four-channel data of the original raw format image, and do normalization processing, and the normalization formula is:

[0181]

[0182] Generally, x max is 65535, that is, the image bit depth is 16 bits, and the normalized image size becomes h x w x 4, denoted as I s ;

[0183] 12) As shown in the accompanying drawings, the left part is a specific schematic diagram of preprocessing, first define two learnable parameters γ and B in the network, both of which are h x w x 4 in shape and size, γ represents the magnification coefficient of the short exposure image, and B is used to remove the tiny noise in the image and increase the contrast of the image; Figure 1

[0184] 13) Define two fully connected modules, since the γ value and B value required by each picture are not the same, fine-tuning is needed, and the output of the fully connected is added to γ and B respectively to obtain the adjusted γ and B values;

[0185] 14) Perform γ(I s -B) operation on four-channel gbrg image data to obtain preprocessed image with size h x w x 4, denoted as I1;

[0186] 2) Send I1 into the feature extraction network, the overall structure of the feature extraction network is shown in the accompanying drawings, including the following steps: Figure 4

[0187] 21) First, build the first convolutional layer, the convolutional kernel size is 6 x 6, s = 2, p = 2, and the output channel is 64, after the convolutional layer output, the output is processed by batch normalization, i.e. Batch Normalization (BN), and then the output is activated by SiLU activation function, the formula of SiLU activation function is:

[0188]

[0189] After the first layer of convolution, the feature map size is directly reduced by half, the dimension becomes 64, and the feature map size becomes (h / 2) x (w / 2) x 64; If not specially mentioned, the output of the convolutional layer described in the following steps is by default connected with a BN layer and a SiLU activation function;

[0190] 22) Build the second convolutional layer, the convolutional kernel size is 3 x 3, s = 2, p = 2, and the output channel is 128, after the second convolutional layer, the feature map size is reduced by half, the dimension becomes 128, and the feature map size becomes (h / 4) x (w / 4) x 128;

[0191] ​​23) build a mixed attention mechanism module MAB, which contains a channel attention module and a Non-local Operation, the channel attention mainly focuses on the dependence between different channels, and emphasizes or suppresses the feature information of a specific channel by assigning weight information on each channel, which pays more attention to local information, while the Non-local Operation can capture global information in the image to better understand the context relationship and long-range dependence in the image, in order to better extract feature information in the image, the two modules are combined to form a mixed attention mechanism, and a schematic diagram of the MAB module is shown in FIG. 2; Figure 3

[0192] The Non-local Operation is a special attention mechanism, which aims to capture information between long-distance pixels and realize global receptive field of each pixel, and the network structure is shown in FIG. 3. Figure 2 It directly captures long-range dependencies by computing interactions between any two locations without being limited to neighboring points. When computing the response of a certain location, the Non-local Operation considers the weighted features of all positions, which can be spatial, temporal or spatio-temporal. The specific form of the Non-local Operation can be expressed as:

[0193]

[0194] where x is the input feature map, j represents the output position, such as spatial, temporal or spatio-temporal index, the response is obtained by enumerating i and calculating, i represents all possible positions on the feature map, the h function calculates the similarity of i and j, the g function calculates the representation of the feature map at the i position, C(x i ) is a normalization factor, and its calculation formula is:

[0195]

[0196] The specific implementation steps of the channel attention are as follows:

[0197] a. Squeeze: first, perform a Global Pool operation on the input feature layer, that is, a global average pooling, which compresses the two-dimensional feature (h*w) of each channel into a number, obtaining a 1×1×C feature strip, and the length is the channel number C of the input feature map;

[0198] b. Calculate the attention weight: apply two fully connected layers to linearly transform the compressed feature data, and a ReLU activation function is followed after the first fully connected layer to increase nonlinearity;

[0199] ​c. Normalize each weight value to 0-1 using the Sigmoid function, whose expression is:

[0200]

[0201] d. Apply the weight coefficient to each channel in the form of multiplication to obtain a new feature map, which is still h*w*c in size;

[0202] The Non-local Operation captures global information in the image, and outputs it as input to the channel attention module. Finally, the original input and the channel attention module output are connected in residual, which alleviates the gradient vanishing problem. The hybrid attention module can guide the network to better learn color information and suppress noise, and obtain better enhancement effect in the polar light environment;

[0203] 24) Build the feature extraction module C1. First, input the image through two convolutional layers with a kernel size of 1x1, s=1, p=0, and an output channel number of 64 to reduce the dimension of the feature. Then, embed the MAB module, and use the output of one of the convolutional layers as the input of the MAB module. Finally, concatenate the output of the other convolutional layer with the output of the MAB module, and change the output channel number to 128 again. After that, pass the image through a convolutional layer to increase the dimension, and the output channel number becomes 128. After passing through the C1 module, the size of the feature map is still (h / 4) x (w / 4) x 128;

[0204] 25) Embed 3 layers of MAB modules in the C1 module in a cascade manner to form the first feature extraction module, which extracts the shallow information of the image, and the output feature map size is (h / 4) x (w / 4) x 128;

[0205] 26) Build the feature extraction module C2. The structure of the C2 module is the same as that of the C1 module, except that the input and output channel numbers of the C2 module are changed to 256. The output feature map size is (h / 8) x (w / 8) x 256;

[0206] 27) Embed 6 layers of MAB modules in the C2 module in a cascade manner to extract the deep information of the image. The output feature map of the C2 module is then passed through a convolutional layer with a kernel size of 3x3, s=2, p=1, and an output channel of 512. The final output feature map size of the CNN feature extraction network is (h / 16) x (w / 16) x 512;

[0207] 3) Build the Transformer encoding-decoding network. The Transformer network uses a self-attention mechanism to effectively capture global information in the image. The specific steps are as follows:

[0208] 31) The feature map output by the CNN network is positionally encoded, and the application adopts a two-dimensional absolute position encoding method for the feature map, and the specific formula is as follows:

[0209]

[0210]

[0211] Wherein, pos represents the position index of a channel information in the feature map, i represents the dimension index, even dimensions are calculated using the sin function, and odd dimensions are calculated using the cos function, d model represents the total dimension, d model = 512 / 2, that is, the first 256 dimensions represent row position information, and the last 256 dimensions represent column position information, the row position information and the column position information are spliced together to obtain position information with the same size as the input feature map, the size is (h / 16)×(w / 16)×512, and it is recorded as I p ;

[0212] 32) The calculated position information is embedded in the input feature map in the form of addition, at this time the feature map has position information, and the size is (h / 16)×(w / 16)×512;

[0213] 33) Build Self-Attention structure, that is, self-attention structure, the calculation process of Self-Attention can be described as follows:

[0214]

[0215] Wherein, Q, K, V all represent the input of Self-Attention, generally all the same value, in the application, the input of Self-Attention is I p , d k is the dimension of the feature map, Self-Attention pays attention to the input sequence itself, rather than the corresponding relationship between the input and the target. In Self-Attention, the model calculates the correlation degree of each element in the sequence with all other elements, and these correlation degrees or weights reflect the mutual relationship between elements. In this way, Self-Attention can help the model better understand the context information in the sequence, establish global dependencies, and thus more accurately process sequence data.

[0216] 34) Build Multi-Head Attention module, that is, a multi-head self-attention structure, which is composed of multiple Self-Attention, by multi-head decomposition of the input sequence, and respectively for each part of the attention calculation, and then the results are linearly transformed to obtain the final output;

[0217] 35) Build FeedForward (FFN) layer, the structure of the FeedForward layer includes linear transformation, activation function and linear transformation. First, the input data is linearly transformed to map the data to a high-dimensional space. Then, after the activation function, the non-linear expression ability of the model is increased. Finally, a linear transformation is performed again to map the features in the high-dimensional space back to the original space to obtain the output of the feedforward layer, and the calculation formula is as follows:

[0218] max(0,XW1+b1)W2+b2

[0219] Where X is the input data, W1 and b1 are the weight and bias of the first fully connected layer, W2 and b2 are the weight and bias of the second fully connected layer, the first fully connected layer uses the ReLU function for activation, and the second fully connected layer does not perform activation;

[0220] 36) As shown in the left side of the Encoder in the accompanying Figure 5 , the Multi-Head Attention module and the FFN are combined to build the Encoder encoding network, where Add represents the residual connection of the output and the original input, and Norm represents the batch normalization processing;

[0221] 37) Cascade 6 identical Encoder modules to form the encoding network;

[0222] 38) Build Decoder module, as shown in Figure 5 , the Decoder contains two Multi-Head Attention inside, the input Q, K, V of the first Multi-Head Attention is a learnable parameter, the K, V of the second Multi-Head Attention is the output of the Encoder encoding information, and the Q is the output of the first Multi-Head Attention, after two Multi-Head Attention, it is sent to a FFN layer to improve the expression ability of the model;

[0223] 39) Cascade 3 identical Decoder modules to form the decoding network;

[0224] 4) The output of the encoding-decoding structure is resampled to reconstruct the output to obtain the enhanced image, which includes the following steps:

[0225] 41) Decode network contains 3 Decoders, each Decoder module contains a set of outputs, and the three sets of outputs are spliced in the channel dimension to obtain output I d ;

[0226] 42) I d is input into a FFN layer, and a linear transformation is performed on the output;

[0227] 43) Finally, the output is up-sampled by the PixelShuffle method to obtain the final reconstructed image;

[0228] 3, construct the network loss function, and the loss function expression is specifically:

[0229] L = (1 - λ1 - λ2) L1 + λ1L ssim + λ2L perc

[0230] Wherein λ1, λ2 represent the weight coefficients of L ssim loss and L perc loss respectively.

[0231] Specifically, the function expression of L1 loss is:

[0232]

[0233] Wherein I gt is the real image data, I h is the predicted image data, and δ is a constant not equal to 0;

[0234] Specifically, the function expression of L ssim loss is:

[0235]

[0236] Wherein, μ x , μ y represent the mean of the real image and the predicted image respectively, represent the variance of the real image and the predicted image, σ xy represent the covariance of the two, C1, C2 are constants not equal to 0;

[0237] Specifically, L percRepresent the perceptual loss, the perceptual loss is calculated by the pre-trained network between two pictures, the present application adopts vgg16 pre-training network, the calculation mode of perceptual loss is that input image and target image are respectively through the pre-trained neural network, the feature representation of them in the network is obtained.Then the feature representation is taken as the input of the loss function, the Euclidean distance between them is calculated, the target is to minimize the distance between input image and target image in the feature space.The function expression is:

[0238]

[0239] Wherein, I gt 、I h Respectively, the real image and the predicted image, The function calculates the feature output of the image after vgg16 network, H j 、W j Indicate the height, width of the jth feature layer, C j Indicate the channel;

[0240] 4, the data set prepared in advance is input into the network for training, the short exposure image is taken as the network input, and the corresponding long exposure image is taken as the real image, the input image is subjected to data enhancement in the training process, the data enhancement method is random rotation, mirror, the rotation includes rotating the image by 90, 180, 270, 360 degrees, the probability value of each random rotation method is 0.25, the mirror method includes vertical mirror and horizontal mirror, the probability value of the mirror method is 0.5, the probability value of whether each data enhancement method is adopted in the training process is 0.5;

[0241] 5, the best model of the training result is imported into the network, and the model is used for enhancing the output of the short exposure image.

[0242] The above only describes the preferred embodiment of the present application, and cannot be understood as limiting the application, any equivalent change and modification made according to the scope of the present application shall belong to the scope of the present application.

Claims

1. A method for enhancing short-exposure images of inner walls of a boiler drum based on a Transformer, characterized in that, The method is as follows: Step 1, constructing a short exposure image dataset of the inner wall of the boiler drum; Step 2, constructing a short exposure image enhancement network based on Transformer, the network comprising a preprocessing part, a feature extraction part, a Transformer encoding-decoding structure part, and an up-sampling reconstruction output part; the following formula is used to express: ； ； In the formula, is the image processed by the preprocessing part, is a learnable magnification coefficient, is the input image data, represents the four-channel data extracted from the original raw image, and normalization processing is performed, is a learnable parameter used to remove some noise in the image and improve the contrast, represents the feature extraction part, represents the fixed position parameter, represents the Transformer encoding and decoding structure part and the final image upsampling reconstruction output part, represents the enhanced image obtained finally; Step 3, constructing a network loss function based on which the learnable parameters in the enhancement network in step 2 are updated Constraints are made to facilitate the enhancement network to finally output an image with normal illumination; Step 4, inputting the dataset in step 1 into the enhancement network in step 2 for training, the short exposure images in the dataset serving as network inputs and the long exposure images serving as reference images, so as to train the enhancement network and obtain a best model after training; Step 5, importing the best model into the enhancement network, inputting a short exposure image, and using the best model to enhance the short exposure image and output; In the step 2, the network preprocessing part is further specifically: Step 211, extracting the data of four channels gbrg of the original raw format image, i.e. the short exposure image, and performing normalization processing, the normalization formula being: ； is 65535, i.e. the maximum value in the short-exposure image with a 16-bit image depth, is the minimum value in the short-exposure image, are the pixel values of the short-exposure image gbrg in four channels, and the normalized image size becomes , denoted as ; wherein h and w are the height and width of the normalized image, respectively. Step 212, defining two learnable parameters in the network and B, each of which has a shape of ; Step 213, performing operation on the image data of the four channels of gbrg , obtaining pre-processed image with size , denoted as .

2. The method of claim 1, wherein the method is a Transformer-based short-exposure image enhancement method for inner wall of a boiler drum. The step 1 is further specifically: Step 11, selecting a camera capable of shooting raw format to collect the images of the inner wall of the boiler drum, the collected images being divided into long exposure images and short exposure images, and the long exposure image and the short exposure image of a scene being a group; Step 12, for the same group, only one long exposure image needs to be collected, and the exposure time is fixed according to the scene; the short exposure image is collected in the following manner: starting from the shortest exposure time, the fixed exposure time is increased one by one, one short exposure image is collected each time, and a total of 10 short exposure images are collected, and the exposure time of each short exposure image is recorded in the file name; In the same group of images, all other parameters are the same except the exposure time.

3. The method of claim 1, wherein the method is a Transformer-based short-exposure image enhancement method for inner wall of a boiler drum. In the step 2, the feature extraction part is further specifically: Step 221, sending the feature extraction part the input feature extraction part; Step 222, build a first layer of convolutional layer, the convolution kernel size is , , , representing the step size of the convolution operation, representing the padding number of the feature map, 2 indicating padding two pixels in each of the top, bottom, left and right; the output channel is 64, and after the convolutional layer outputs, the output is subjected to batch normalization processing, that is, Batch Normalization, also called BN, and then the output is activated by a silu activation function, and the formula of the silu activation function is: ； ； wherein, is an activation function output, is an input value; The feature map size is directly reduced by half through the first layer of convolution, the dimension is changed to 64, and the feature map size is changed to ; Step 223, a second layer of convolutional layer is built, the convolution kernel size is , , , the output channel is 128, the feature map size is reduced by half through the second layer of convolution, the dimension becomes 128, and the feature map size becomes ; Step 224, capturing global information in the image through Non-local Operation, and the specific form of Non-local Operation is expressed as: ； wherein, is the input feature map, represents the output position, the response is to enumerate and then calculate, represent all possible positions on the feature map, functional calculation and the similarity of, function calculation feature map in position representation, is the standardization factor, the calculation formula is: ； The specific implementation steps of the channel attention are as follows: a、Squeeze: First, do a Global Pool operation on the input feature layer, that is, global average pooling, compress the two-dimensional features of each channel into a number, and get a feature strip with a length of C, the channel number of the input feature map. ​​ b, calculating the attention weight: applying two fully connected layers to perform linear transformation on the compressed feature data, and a ReLU activation function is followed after the first fully connected layer to increase nonlinearity; c, using a sigmoid function to normalize each weight value to 0-1, and the expression is: ； d. Apply the weight coefficient to each channel in the form of multiplication to get a new feature map, the size remains unchanged ; Non-local Operation captures global information in the image, and then outputs the global information as input to the channel attention module; finally, the original input and the channel attention module output are connected in residual, which alleviates the gradient disappearance problem; by using the hybrid attention module, the network can better learn color information and suppress noise, and better enhancement effect can be obtained in an extremely low light environment; Step 225, build a feature extraction module C3, first pass the input through two convolutional layers respectively, the convolution kernel size is , , , the output channel number is 64, reduce the dimension of the feature, then embed the MAB module, take the output of one of the convolutional layers as the input of the MAB module, and then concatenate the output of the other convolutional layer with the output of the MAB module, the output channel number becomes 128 again, finally, pass the image through a convolutional layer to increase the dimension, and the output channel number is 128, after passing through a C3 module, the feature map size is still ; Step 226, embed the 3-layer MAB module in the C3 module in a cascading manner to form a first feature extraction module to extract shallow information of the image, and output a feature map with a size of ; Step 227, build a feature extraction module C4, the structure of C4 module is the same as that of C3 module, except that the number of input and output channels of C4 module becomes 256, and the size of the output feature map is ; Step 228, embed the 6-layer MAB module in the C4 module in a cascading manner to extract deep information in the image, and the C4 module outputs a feature map which is further processed by a convolution layer with a convolution kernel size of , , , and an output channel of 512; the size of the feature map finally output by the feature extraction part is .

4. The method of claim 3, wherein the method is a Transformer-based short-exposure image enhancement method for inner wall of a boiler drum. In the step 2, the Transformer encoding-decoding structure part is further specifically: Step 231, performing position encoding on the feature map output by the feature extraction part, and using two-dimensional absolute position encoding for the feature map, and the specific formula is as follows: ； ； wherein, is even dimension, is odd dimension; pos represents the position index of one channel information in the feature map, represents the dimension index, the even dimension uses the sin function to calculate, and the odd dimension uses the cos function to calculate, represents the total dimension, wherein , that is, the first 256 dimensions represent the row position information, and the last 256 dimensions represent the column position information, and the row position information and the column position information are spliced together to obtain the position information same as the size of the input feature map, and the size is , which is denoted as ; Step 232, embedding the calculated position information into the input feature map in the form of addition, at this time the feature map with position information, size is ; Step 233, build a Self-Attention structure, that is, a self-attention structure, and the calculation process of Self-Attention is described as follows: ； Wherein Q, K, V all represent the input of Self-Attention, and the input of Self-Attention is , is the dimension of the feature map, and T is the transpose operation; Step 234, build a Multi-Head Attention module, that is, a multi-head self-attention structure, which is composed of multiple Self-Attention, by decomposing the input sequence into multiple heads, and performing independent attention calculation on each part, and then performing linear transformation on the results to obtain the final output; Step 235, build a Feed Forward layer, also known as FFN, the structure of the Feed Forward layer includes linear transformation, activation function and linear transformation; first, the input data is subjected to a linear transformation to map the data to a high-dimensional space; then, through the activation function, the non-linear expression ability of the model is increased; finally, a linear transformation is performed again to map the features in the high-dimensional space back to the original space to obtain the output of the feed forward layer, and the calculation formula is as follows: ； wherein is the input data, , are the bias of the weights of the first fully connected layer, respectively, , are the weights and bias of the second fully connected layer, the first fully connected layer is activated with a ReLU function and the second fully connected layer is not activated; Step 236, combine the Multi-Head Attention module and the FFN to build an Encoder encoding network; Step 237, cascade 6 identical Encoder modules to form the encoding part; Step 238, build a Decoder module, which contains two Multi-Head Attention modules inside, the input Q, K and V of the first Multi-Head Attention module are learnable parameters, the K and V of the second Multi-Head Attention module are the output of the Encoder encoding information, and the Q is the output of the first Multi-Head Attention module, which is sent into a FFN layer after two Multi-Head Attention to improve the expression ability of the model; Step 239, cascade 3 identical Decoder modules to form the decoding part.

5. The method of claim 4, wherein the method is a Transformer-based short-exposure inner wall image enhancement method. In step 2, the Transformer encoding-decoding structure part is further specifically: Step 241, decoding network contains 3 Decoders, each Decoder module contains a set of outputs, and the three sets of outputs are spliced in the channel dimension to obtain the output ; Step 242, to input a FFN layer, do linear transformation on the output; Step 243, finally, the output is up-sampled by the PixelShuffle method to obtain the final reconstructed image.

6. The method of claim 1, wherein the method is a Transformer-based short-exposure inner wall image enhancement method. The loss function expression is specifically: ； wherein L represents a total loss function, is a structural similarity loss, is a perceptual loss; , respectively represent losses, weighting coefficients of the losses. The function expression of L1 loss is specifically: ； wherein is a real image, is a predicted image, is a constant not equal to 0; In particular, The function expression of the loss is: ； wherein, respectively represent the mean of the real and predicted images, represent the variance of the real and predicted images, represent the covariance of both, , is a constant not equal to 0; is the sum of the number of pixels of the image; representative of perceptual loss, which is calculated by a pre-trained network to measure the difference between two images. The VGG16 pre-trained network is used, and the calculation method of perceptual loss is that the input image and the target image are respectively input into the pre-trained neural network to obtain their feature representations in the network; then the feature representations are taken as the input of the loss function, and the Euclidean distance between them is calculated, and the goal is to minimize the distance between the input image and the target image in the feature space; the function expression is: ； wherein, , are a real image and a predicted image, respectively, a function calculates a feature output of the image after passing through the vgg16 network, denotes a height and a width of the th feature layer, denotes a channel.

7. The method of claim 1, wherein the method is a Transformer-based short-exposure inner wall image enhancement method. In step 4, during the training process, the short exposure images and long exposure images in the data set are subjected to random data enhancement, and the enhanced short exposure images are input into the network.

8. The method of claim 7, wherein the method is a Transformer-based short-exposure image enhancement method for inner wall of a boiler drum. The data enhancement method is random rotation and mirroring; the rotation method includes rotating the image by 90, 180, 270 and 360 degrees, and the probability value of each random rotation method is 0.25; the mirroring method includes vertical mirroring and horizontal mirroring, and the probability value of the mirroring method is 0.5, and the probability value of whether to use each data enhancement method in the training process is 0.

5.

9. The method of claim 1 or 7 or 8, wherein, In step 4, the size of the short-exposure image input into the network is the short-exposure image after random cropping, and the cropping size is The image size of the final input into the network is .

Images