Deep learning infrared image denoising method and system based on multi-head self-attention mechanism

Abstract

The present invention discloses a method and system for infrared image denoising using deep learning based on a multi-head self-attention mechanism. The method comprises the following steps: S1: collecting and preprocessing high-definition infrared images; S2: obtaining a dataset based on the high-definition infrared images and the preprocessed infrared images; S3: establishing an infrared denoising neural network based on the multi-head self-attention mechanism, and training the infrared denoising neural network using the dataset to obtain a trained infrared denoising neural network; S4: using the trained infrared denoising neural network to denoise noisy infrared images. The present invention, through deep learning infrared image denoising technology using a multi-head self-attention mechanism, can effectively overcome the high noise problem existing in classic infrared imaging systems. This invention is highly beneficial for the research of deep learning image denoising technology using a multi-head self-attention mechanism for infrared imaging.

Description

Technical Field

[0001] The present invention relates to the field of infrared imaging, and more specifically, to a deep learning infrared image denoising method and system based on a multi-head self-attention mechanism. Background Art

[0002] As a detection method that helps the human visual system become sensitive to infrared radiation, infrared sensors output infrared images that reflect temperature differences among objects in a scene. Infrared sensors utilize passive imaging and operate in all weather conditions. Consequently, they are widely used in scientific research, military detection, fire monitoring, fault diagnosis, medical analysis, and remote sensing. Generally speaking, due to factors including the principles of infrared imaging technology, external environmental distortion, and thermal motion of the sensor itself, infrared images can suffer from low resolution, blurred edges, loss of detail, low image contrast, and background noise. To prevent noise from becoming a limitation that hinders the development of infrared imaging technology, removing noise signals from infrared images is crucial.

[0003] Classic infrared image denoising methods often employ numerical methods, such as spatial domain filtering, transform domain filtering, sparse representation-based methods, and total variation-based methods. These methods typically extract features based on a single noise distribution. However, due to physical factors of the imaging element and background thermal radiation, infrared image noise is composed of both Gaussian and Poisson distributions. Consequently, traditional infrared image denoising methods are ineffective.

[0004] Prior art discloses an infrared image denoising algorithm based on wavelet information redundancy. This algorithm obtains multiple sets of wavelet coefficients containing similar redundant information using different downsampling methods during the discrete wavelet transform (DWT) process. The algorithm then uses noise estimation parameters as the basis for the threshold of a nonlinear transformation function to perform a nonlinear transformation on the high-frequency wavelet coefficients, suppressing high-frequency noise. The transformed wavelet coefficients are then used to reconstruct multiple images (IDWT). The weighting of the multiple images containing similar redundant information further removes high-frequency noise, resulting in a high-quality infrared image. This method is ineffective for denoising infrared images with Gaussian and Poisson distributed noise. Summary of the Invention

[0005] The primary purpose of this invention is to provide a deep learning infrared image denoising method based on a multi-head self-attention mechanism. This method addresses the problems of poor imaging quality and high noise in infrared thermal imaging systems, improves imaging quality, and reduces noise in infrared images. It is expected to be widely used in scientific research, military detection, fire monitoring, fault diagnosis, medical analysis, remote sensing and other fields.

[0006] A further object of the present invention is to provide a deep learning infrared image denoising system based on a multi-head self-attention mechanism.

[0007] In order to solve the above technical problems, the technical solutions of the present invention are as follows:

[0008] A deep learning infrared image denoising method based on a multi-head self-attention mechanism includes the following steps:

[0009] S1: collect high-definition infrared images and perform preprocessing;

[0010] S2: obtaining a data set based on the high-definition infrared image and the preprocessed infrared image;

[0011] S3: establishing an infrared denoising neural network based on a multi-head self-attention mechanism, and training the infrared denoising neural network using the data set to obtain a trained infrared denoising neural network;

[0012] S4: Use the trained infrared denoising neural network to denoise the noisy infrared image.

[0013] Preferably, the collecting of high-definition infrared images in step S1 is specifically:

[0014] The high-definition infrared image set I is obtained by shooting in strong light and weak light respectively. HQ , contains M images in total, each image is recorded as Where i = 1, 2, ..., M.

[0015] Preferably, the preprocessing in step S1 is specifically as follows:

[0016] Add Gaussian noise and Poisson noise to the high-definition infrared image, set the Gaussian noise parameter σ and the Poisson noise parameter λ, and generate the infrared image set I with noise LQ , each image is recorded as Where i = 1, 2, ..., M.

[0017] Preferably, the data set in step S2 is specifically:

[0018] According to the high-definition infrared image collection I HQ and infrared image set I with noise LQ The corresponding image pair Construct a data set.

[0019] Preferably, the infrared denoising neural network based on the multi-head self-attention mechanism in step S3 is specifically:

[0020] The infrared denoising neural network based on the multi-head self-attention mechanism includes a local feature extraction module, a global feature extraction module and an image restoration module, wherein the local feature extraction module extracts local feature information I in the infrared image. localThe global feature extraction module extracts the global feature information I in the infrared image global The image restoration module converts the local feature information I local and global feature information I global After fusion, the noise-free infrared image I is restored rec .

[0021] Preferably, the local feature extraction module is specifically:

[0022] The local feature extraction module consists of two convolution layers with a convolution kernel size of K, denoted as F local , the dimension is C in Infrared image Processed into tensor information I with a dimension of 96 local ∈R H×W×C , where C = 96, H and W are the image height and length, respectively. The mathematical model is shown below:

[0023] I local =F local (I0).

[0024] Preferably, the global feature extraction module is specifically:

[0025] The mathematical model representation of the global feature extraction module is as follows:

[0026] I global =F global (I local )

[0027] In the above formula, F global Represents the global feature extraction module, which contains N multi-head self-attention calculation modules and a convolution operation with a convolution kernel size of K. Each module extracts features I1, I2, ..., I N And global feature I global It is calculated step by step by MSAB, and its mathematical model is expressed as:

[0028] I i =F MASB (I i-1 ),i=1,2,...,N

[0029] I global =F CONV (I N )

[0030] In the above formula, F MASB represents the multi-head self-attention calculation module, F CONVRepresents the convolution operation. To reduce the amount of calculation, the method adopts the sliding window multi-head self-attention mechanism. When the attention mechanism is calculated, the input I local ∈R H×W×C Divided into non-overlapping windows of size M×M, the feature size becomes in is the total number of windows, and then the standard multi-head self-attention calculation is performed in each window, and the features in a sliding window Its query, key, and value calculation process is as follows:

[0031] Q = XP Q ,K=XP K ,V=XP V

[0032] In the above formula, P Q , P K and P V Is the mapping matrix in the window, different windows share the same mapping matrix; after transformation The self-attention calculation method is as follows:

[0033]

[0034] In the above formula, B represents the learnable position encoding;

[0035] Use a multi-layer feature perceptron to further transform the features. The mathematical model is as follows:

[0036] X=WMSA(LN(X))+X

[0037] X=MLP(LN(X))+X

[0038] In the above formula, WMSA represents the multi-head self-attention calculation within the window, MLP represents the fully connected multi-layer feature perceptron, and LayerNormalization calculation is added. LN() represents LayerNormalization calculation; then the sliding window self-attention calculation is performed:

[0039] X=SWMSA(LN(X))+X

[0040] X=MLP(LN(X))+X

[0041] In the above formula, SWMA represents the calculation of the multi-head self-attention mechanism after the sliding window.

[0042] Preferably, the image restoration module is specifically:

[0043] The image restoration module includes two identical convolution operations. The mathematical model of this process is expressed as:

[0044] I rec =F rec (I local +I global )

[0045] In the above formula, F rec Represents the image restoration module. Finally, the output image I rec ∈R 640×640×3 The image restoration module is a learnable convolutional neural network, which fuses and reconstructs the feature information after noise removal from the first two modules and restores it to image information.

[0046] Preferably, in step S3, the infrared denoising neural network is trained using the data set, and the Charbonnier loss function L is optimized using the Adam function. The loss function is expressed as follows:

[0047]

[0048] In the above formula, I rec Represents the image after denoising by neural network, I HQ Represents the original high-definition image, parameter ε=10 -3 .

[0049] A deep learning infrared image denoising system based on a multi-head self-attention mechanism, including:

[0050] An image collection and processing module, which is used to collect high-definition infrared images and perform preprocessing;

[0051] A data set module, configured to obtain a data set based on the high-definition infrared image and the preprocessed infrared image;

[0052] A multi-head self-attention denoising neural network module, wherein the multi-head self-attention denoising neural network module is used to establish an infrared denoising neural network based on a multi-head self-attention mechanism, and train the infrared denoising neural network using the data set to obtain a trained infrared denoising neural network;

[0053] The denoising module uses a trained infrared denoising neural network to denoise the noisy infrared image.

[0054] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0055] This paper uses a multi-head self-attention mechanism to deeply learn infrared image denoising technology, which can effectively overcome the high noise problem existing in classic infrared imaging systems. This paper is conducive to the research of infrared imaging multi-head self-attention mechanism deep learning image denoising technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 Schematic diagram of the method of the present invention.

[0057] Figure 2 Schematic diagram of the infrared denoising neural network structure based on the multi-head self-attention mechanism in the embodiment.

[0058] Figure 3 Schematic diagram of a local feature extraction module in an embodiment.

[0059] Figure 4 Schematic diagram of the global feature extraction module in the embodiment.

[0060] Figure 5 This is the specific implementation process of multi-head self-attention calculation in the embodiment.

[0061] Figure 6 Schematic diagram of the image restoration module in the embodiment.

[0062] Figure 7 The infrared image denoising effect diagram provided in the embodiment, Figure (a) is a high-definition infrared image, Figure (b) is an infrared image with noise added, and Figure (c) is an infrared image after denoising.

[0063] Figure 8 Schematic diagram of the system module of the present invention.

[0064] In the figure, 101 is the local feature extraction module, 102 is the global feature extraction module, 103 is the image restoration module, 201 is the first convolution operation of the local feature extraction module, 202 is the second convolution operation of the local feature extraction module, 301 is the multi-head self-attention calculation in the global feature extraction module, 302 is the multi-head self-attention module of the global feature extraction module, 303 is the convolution operation of the global feature extraction module, 304 is the module end convolution operation of the global feature extraction module, 401 is layer regularization, 402 is window self-attention calculation, 403 is sliding window self-attention calculation, 404 is a multi-layer feature perceptron, and 501 is the convolution operation of the image restoration module. DETAILED DESCRIPTION

[0065] The accompanying drawings are for illustrative purposes only and are not to be construed as limiting this patent;

[0066] In order to better illustrate this embodiment, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product size;

[0067] It is understandable to those skilled in the art that some well-known structures and descriptions thereof may be omitted in the drawings.

[0068] The technical solution of the present invention is further described below with reference to the accompanying drawings and embodiments.

[0069] Example 1

[0070] This embodiment provides a deep learning infrared image denoising method based on a multi-head self-attention mechanism. Figure 1 As shown, the following steps are included:

[0071] S1: collect high-definition infrared images and perform preprocessing;

[0072] S2: obtaining a data set based on the high-definition infrared image and the preprocessed infrared image;

[0073] S3: establishing an infrared denoising neural network based on a multi-head self-attention mechanism, and training the infrared denoising neural network using the data set to obtain a trained infrared denoising neural network;

[0074] S4: Use the trained infrared denoising neural network to denoise the noisy infrared image.

[0075] The multi-head self-attention mechanism was originally used in natural language processing tasks. Due to its powerful ability to extract both global and local features, it has achieved excellent results in vision tasks, replacing convolutional neural networks. Leveraging the multi-head self-attention mechanism's global and local feature extraction capabilities can better fit the noise distribution in an image, thereby achieving superior denoising results.

[0076] The step S1 of collecting high-definition infrared images is specifically as follows:

[0077] The high-definition infrared image set I is obtained by shooting in strong light and weak light respectively. HQ , contains M images in total, each image is recorded as Where i = 1, 2, ..., M.

[0078] The preprocessing in step S1 is specifically as follows:

[0079] Add Gaussian noise and Poisson noise to the high-definition infrared image, set the Gaussian noise parameter σ and the Poisson noise parameter λ, and generate the infrared image set I with noise LQ , each image is recorded as Wherein, i=1, 2, ..., M. In this embodiment, the Gaussian noise parameter σ is set to 50, and the Poisson noise parameter λ is set to 0.5.

[0080] The data set in step S2 is specifically:

[0081] According to the high-definition infrared image collection I HQand infrared image set I with noise LQ The corresponding image pair Construct a data set.

[0082] Example 2

[0083] This embodiment, based on the first embodiment, specifically discloses:

[0084] The infrared denoising neural network based on the multi-head self-attention mechanism in step S3 is as follows: Figure 2 As shown, specifically:

[0085] The infrared denoising neural network based on the multi-head self-attention mechanism includes a local feature extraction module 101, a global feature extraction module 102 and an image restoration module 103, wherein the local feature extraction module 101 extracts local feature information I in the infrared image. local The global feature extraction module 102 extracts the global feature information I in the infrared image global The image restoration module 103 converts the local feature information I local and global feature information I global After fusion, the noise-free infrared image I is restored rec .

[0086] The local feature extraction module 101 is as follows: Figure 3 As shown, specifically:

[0087] The local feature extraction module 101 comprises two convolution layers 201 and 202 with a convolution kernel size of K, denoted as F local In this embodiment, K=3. in =3 infrared image I0∈R 640×640×3 Processed into tensor information I with a dimension of 96 local ∈R 640×640×96 , where C = 96, H and W are the image height and length, and the mathematical model is expressed as follows:

[0088] I local =F local (I0).

[0089] The global feature extraction module 102 is as follows: Figure 4 As shown, specifically:

[0090] The mathematical model of the global feature extraction module 102 is shown as follows:

[0091] I global =F global (I local )

[0092] In the above formula, F global It represents the global feature extraction module 102, which contains N multi-head self-attention calculation modules 302 and a convolution operation with a convolution kernel size of K=3, where each module extracts features I1, I2, ..., I N And global feature I global It is calculated step by step by MSAB, and its mathematical model is expressed as:

[0093] I i =F MASB (I i-1 ),i=1,2,...,N

[0094] I global =F CONV (I N )

[0095] In the above formula, F MASB represents the multi-head self-attention calculation module, F CONV Represents the convolution operation. To reduce the amount of calculation, the method adopts the sliding window multi-head self-attention mechanism. When the attention mechanism is calculated, the input I local ∈R 512×640×96 Divided into non-overlapping windows of size 8×8, the feature size becomes in is the total number of windows, and then a standard multi-head self-attention calculation is performed within each window, such as Figure 5 As shown, the features within a sliding window Its query, key, and value calculation process are as follows:

[0096] Q = XP Q ,K=XP K ,V=XP V

[0097] In the above formula, P Q , P K and P V Is the mapping matrix in the window, different windows share the same mapping matrix; after transformation The self-attention calculation method is as follows:

[0098]

[0099] In the above formula, B represents the learnable position encoding;

[0100] Use a multi-layer feature perceptron to further transform the features. The mathematical model is as follows:

[0101] X=WMSA(LN(X))+X

[0102] X=MLP(LN(X))+X

[0103] In the above formula, WMSA represents the multi-head self-attention calculation within the window, MLP represents the fully connected multi-layer feature perceptron, and LayerNormalization calculation is added. LN() represents LayerNormalization calculation; then the sliding window self-attention calculation is performed:

[0104] X=SWMSA(LN(X))+X

[0105] X=MLP(LN(X))+X

[0106] In the above formula, SWMA represents the calculation of the multi-head self-attention mechanism after the sliding window. Through the sliding window operation, the features between different windows can be extracted. The distance moved each time the sliding window is In this embodiment, the distance the sliding window moves each time is The self-attention mechanism and sliding window are used to extract features of global image pixel information, calculate the relationship between pixels, further retain image information, and remove noise information.

[0107] The image restoration module 103 is as follows: Figure 6 As shown, specifically:

[0108] The image restoration module 103 includes two identical convolution operations 501 to restore local features and global features into a noise-free image. The mathematical model of this process is expressed as:

[0109] I rec =F rec (I local +I global )

[0110] In the above formula, F rec Represents the image restoration module 103. Finally, the output image I rec ∈R 640×640×3 The image restoration module 103 is a learnable convolutional neural network, which fuses and reconstructs the feature information after noise removal from the first two modules, restores it to image information, and further filters out noise information.

[0111] In step S3, the infrared denoising neural network is trained using the data set, and the Charbonnier loss function L is optimized using the Adam function. The loss function is expressed as follows:

[0112]

[0113] In the above formula, I recRepresents the image after denoising by neural network, I HQ Represents the original high-definition image, parameter ε=10 -3 .

[0114] like Figure 7 As shown, a schematic diagram of the effect of using the infrared image denoising method proposed in this embodiment is shown, wherein Figure (a) is a high-definition infrared image, Figure (b) is an infrared image with noise added, and Figure (c) is an infrared image after denoising.

[0115] Example 3

[0116] A deep learning infrared image denoising system based on multi-head self-attention mechanism, such as Figure 8 Shown, including:

[0117] An image collection and processing module, which is used to collect high-definition infrared images and perform preprocessing;

[0118] A data set module, configured to obtain a data set based on the high-definition infrared image and the preprocessed infrared image;

[0119] A multi-head self-attention denoising neural network module, wherein the multi-head self-attention denoising neural network module is used to establish an infrared denoising neural network based on a multi-head self-attention mechanism, and train the infrared denoising neural network using the data set to obtain a trained infrared denoising neural network;

[0120] The denoising module uses a trained infrared denoising neural network to denoise the noisy infrared image.

[0121] The same or similar reference numerals correspond to the same or similar components;

[0122] The terms used in the drawings to describe positional relationships are for illustrative purposes only and should not be construed as limiting this patent;

[0123] Obviously, the above embodiments of the present invention are merely examples for the purpose of clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. A deep learning infrared image denoising method based on multi-head self-attention mechanism, characterized by: The following steps are involved: S1: collect high-definition infrared images and perform preprocessing; The preprocessing is specifically as follows: Add Gaussian noise and Poisson noise to the high-definition infrared image, set the Gaussian noise parameter σ and the Poisson noise parameter λ, and generate the infrared image set I with noise LQ , each image is recorded as Where i = 1, 2, ..., M; S2: obtaining a data set based on the high-definition infrared image and the preprocessed infrared image; S3: establishing an infrared denoising neural network based on a multi-head self-attention mechanism, and training the infrared denoising neural network using the data set to obtain a trained infrared denoising neural network; The infrared denoising neural network based on the multi-head self-attention mechanism is specifically: The infrared denoising neural network based on the multi-head self-attention mechanism includes a local feature extraction module, a global feature extraction module and an image restoration module, wherein the local feature extraction module extracts local feature information I in the infrared image. local The global feature extraction module extracts the global feature information I in the infrared image global The image restoration module converts the local feature information I local and global feature information I global After fusion, the noise-free infrared image I is restored rec ; The local feature extraction module consists of two convolution layers with a convolution kernel size of K, denoted as F local , the dimension is C in Infrared image Processed into tensor information I with a dimension of 96 local ∈R H×W×C , where C = 96, H and W are the image height and length, and the mathematical model is expressed as follows: I local =F local (I0); The global feature extraction module is specifically: The mathematical model representation of the global feature extraction module is as follows: I global =F global (I local ) In the above formula, F global Represents the global feature extraction module, which contains N multi-head self-attention calculation modules and a convolution operation with a convolution kernel size of K. Each module extracts features I1, I2, ..., I N And global feature I global It is calculated step by step by MSAB, and its mathematical model is expressed as: I i =F MASB (I i-1 ),i=1,2,...,N I global =F CONV (I N ) In the above formula, F MASB represents the multi-head self-attention calculation module, F CONV Represents the convolution operation. To reduce the amount of calculation, the method adopts the sliding window multi-head self-attention mechanism. When the attention mechanism is calculated, the input I local ∈R H×W×C Divided into non-overlapping windows of size M×M, the feature size becomes in is the total number of windows, and then the standard multi-head self-attention calculation is performed in each window, and the features in a sliding window Its query, key, and value calculation process is as follows: Q=XP Q ,K=XP K ,V=XP V In the above formula, P Q , P K and P V Is the mapping matrix in the window, different windows share the same mapping matrix; after transformation The self-attention calculation method is as follows: In the above formula, B represents the learnable position encoding; Use a multi-layer feature perceptron to further transform the features. The mathematical model is as follows: X=WMSA(LN(X))+X X=MLP(LN(X))+X In the above formula, WMSA represents the multi-head self-attention calculation within the window, MLP represents the fully connected multi-layer feature perceptron, and Layer Normalization calculation is added, LN() represents Layer Normalization calculation; then the sliding window self-attention calculation is performed: X=SWMSA(LN(X))+X X=MLP(LN(X))+X In the above formula, SWMA represents the calculation of the multi-head self-attention mechanism after the sliding window; S4: Use the trained infrared denoising neural network to denoise the noisy infrared image.

2. The deep learning infrared image denoising method based on multi-head self-attention mechanism according to claim 1 is characterized in that: The step S1 of collecting high-definition infrared images is specifically as follows: The high-definition infrared image set I is obtained by shooting in strong light and weak light respectively. HQ , contains M images in total, each image is recorded as Where i = 1, 2, ..., M.

3. The deep learning infrared image denoising method based on multi-head self-attention mechanism according to claim 2 is characterized in that: The data set in step S2 is specifically: According to the high-definition infrared image collection I HQ and infrared image set I with noise LQ The corresponding image pair Construct a data set.

4. The deep learning infrared image denoising method based on multi-head self-attention mechanism according to claim 1 is characterized in that The image restoration module is specifically: The image restoration module includes two identical convolution operations. The mathematical model of this process is expressed as: I rec =F rec (I local +I global ) In the above formula, F rec Represents the image restoration module; the final output image I rec ∈R 640×640×3 The image restoration module is a learnable convolutional neural network, which fuses and reconstructs the feature information after noise removal from the first two modules and restores it to image information.

5. The deep learning infrared image denoising method based on multi-head self-attention mechanism according to claim 4 is characterized in that: In step S3, the infrared denoising neural network is trained using the data set, and the Charbonnier loss function L is optimized using the Adam function. The loss function is expressed as follows: In the above formula, I rec Represents the image after denoising by neural network, I HQ Represents the original high-definition image, parameter ε=10 -3 .

6. A deep learning infrared image denoising system based on multi-head self-attention mechanism, characterized by: include: An image collection and processing module, which is used to collect high-definition infrared images and perform preprocessing; The preprocessing is specifically as follows: Add Gaussian noise and Poisson noise to the high-definition infrared image, set the Gaussian noise parameter σ and the Poisson noise parameter λ, and generate the infrared image set I with noise LQ , each image is recorded as Where i = 1, 2, ..., M; A data set module, configured to obtain a data set based on the high-definition infrared image and the preprocessed infrared image; A multi-head self-attention denoising neural network module, wherein the multi-head self-attention denoising neural network module is used to establish an infrared denoising neural network based on a multi-head self-attention mechanism, and train the infrared denoising neural network using the data set to obtain a trained infrared denoising neural network; The infrared denoising neural network based on the multi-head self-attention mechanism is specifically: The infrared denoising neural network based on the multi-head self-attention mechanism includes a local feature extraction module, a global feature extraction module and an image restoration module, wherein the local feature extraction module extracts local feature information I in the infrared image. local The global feature extraction module extracts the global feature information I in the infrared image global The image restoration module converts the local feature information I local and global feature information I global After fusion, the noise-free infrared image I is restored rec ; The local feature extraction module consists of two convolution layers with a convolution kernel size of K, denoted as F local , the dimension is C in Infrared image Processed into tensor information I with a dimension of 96 local ∈R H×W×C , where C = 96, H and W are the image height and length, and the mathematical model is expressed as follows: I local =F local (I0); The global feature extraction module is specifically: The mathematical model representation of the global feature extraction module is as follows: I global =F global (I local ) In the above formula, F global Represents the global feature extraction module, which contains N multi-head self-attention calculation modules and a convolution operation with a convolution kernel size of K. Each module extracts features I1, I2, ..., I N And global feature I global It is calculated step by step by MSAB, and its mathematical model is expressed as: I i =F MASB (I i-1 ),i=1,2,...,N I global =F CONV (I N ) In the above formula, F MASB represents the multi-head self-attention calculation module, F CONV Represents the convolution operation. To reduce the amount of calculation, the method adopts the sliding window multi-head self-attention mechanism. When the attention mechanism is calculated, the input I local ∈R H×W×C Divided into non-overlapping windows of size M×M, the feature size becomes in is the total number of windows, and then the standard multi-head self-attention calculation is performed in each window, and the features in a sliding window Its query, key, and value calculation process is as follows: Q=XP Q ,K=XP K ,V=XP V In the above formula, P Q , P K and P V Is the mapping matrix in the window, different windows share the same mapping matrix; after transformation The self-attention calculation method is as follows: In the above formula, B represents the learnable position encoding; Use a multi-layer feature perceptron to further transform the features. The mathematical model is as follows: X=WMSA(LN(X))+X X=MLP(LN(X))+X In the above formula, WMSA represents the multi-head self-attention calculation within the window, MLP represents the fully connected multi-layer feature perceptron, and Layer Normalization calculation is added, and LN() represents the Layer Normalization calculation; Then the sliding window self-attention calculation is performed: X=SWMSA(LN(X))+X X=MLP(LN(X))+X In the above formula, SWMA represents the calculation of the multi-head self-attention mechanism after the sliding window; The denoising module uses a trained infrared denoising neural network to denoise the noisy infrared image.

Images