Infrared image generation based on multi-modal self-attention mechanism generative adversarial network and device

Abstract

The application discloses an infrared image generation method and device based on a multi-modal self-attention mechanism generative adversarial network, and belongs to the field of infrared image generation. The method comprises the following steps: segmenting and linearly compressing an input visible light image; performing multi-layer window-based local self-attention calculation in the compressed image block and introducing an adaptive correlation mask to obtain a local correlation expression of the image; performing multi-layer fast global self-attention calculation between the compressed image blocks to obtain a global correlation expression of the image; fusing the local correlation expression and the global correlation expression of the image to obtain a feature map containing decoupled background features and target features; and decoding and up-sampling the feature map to output an infrared image with the same size as the input image. The generated infrared image has correct infrared features and better visual effects.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of infrared image generation, and more particularly relates to an infrared image generation method and device based on a multi-modal self-attention mechanism generative adversarial network. BACKGROUND

[0002] Infrared imaging is to obtain image information of an object and an environment by using energy difference of thermal radiation of the object and the environment. In an infrared imaging system, an infrared detector receives thermal radiation, converts the thermal radiation into an electrical signal, and finally presents an image of the photographed object through signal processing and imaging technology. Thanks to the imaging mechanism, infrared images can be used at any time and under any weather conditions, because the energy of thermal radiation of the object and the environment is not restricted by natural conditions such as day and night, sunny and rainy days, etc. Therefore, it has great significance to use infrared images as training data sets for various deep learning tasks.

[0003] A traditional infrared simulation system needs to consider many factors such as a target geometric model, a target physical parameter, target temperature calculation, target self-infrared radiation, target surface material infrared characteristics, target reflection to a background, and environmental radiation when performing infrared simulation, which brings great inconvenience to simulation. Moreover, after considering factors such as the above, the result of simulation under ideal conditions is often too ideal and does not fit the actual situation, especially the infrared background deviates greatly from the actual background.

[0004] Compared with the traditional infrared simulation system, a deep learning method such as a generative adversarial network does not need to consider any factor or perform complex theoretical calculation from the simulation process, and only needs to input a visible light image to output a corresponding infrared image; from the result, the work of the deep learning is to migrate the target image from the visible light domain to the infrared domain, and the quality of the generated infrared image is higher. However, due to the different imaging mechanisms of visible light and infrared images, directly converting the visible light to the infrared image under the condition that the target and the background are not decoupled will cause the background and the target in the generated infrared image to have the same infrared characteristics, which is inconsistent with the characteristics of the real infrared image. SUMMARY

[0005] In view of the defects of the related art, the present application aims to provide an infrared image generation method and device based on a multi-modal self-attention mechanism generative adversarial network, which aims to solve the problem of incorrect infrared characteristics when a complex visible light image containing a target is generated into an infrared image by a generative adversarial network.

[0006] To achieve the above-mentioned purpose, the present application provides an infrared image generation method based on a multi-modal self-attention mechanism generative adversarial network, comprising:

[0007] S1, segmenting the input visible light image to obtain image blocks of the same size, and performing linear compression;

[0008] S2, performing multi-layer window-based local self-attention calculation in the compressed image blocks and introducing adaptive correlation masks to obtain local correlation expression of the image;

[0009] S3, performing multi-layer fast global self-attention calculation between the compressed image blocks to obtain global correlation expression of the image;

[0010] S4, fusing the local correlation expression and the global correlation expression of the image to obtain a feature map containing decoupled background features and target features;

[0011] S5, decoding and up-sampling the feature map to output an infrared image consistent with the size of the input image.

[0012] Optionally, in step S2, W-MSA and SW-MSA in Swin Transformer are used to perform self-attention calculation within the window, and an adaptive correlation mask is introduced in SW-MSA.

[0013] Optionally, the step S3 comprises:

[0014] In the compressed image blocks, a fast global self-attention mechanism FG-MSA is used to compress each window into a vector that can be used for dot product calculation of the self-attention mechanism, and multi-head self-attention calculation is performed between the windows to obtain global correlation expression of the image.

[0015] Optionally, the step S4 comprises:

[0016] S41, splicing and fusing the output of W-MSA and the output of the first FG-MSA to obtain a first result, and performing residual calculation on the first result and the output of the first result after passing through a layer normalization layer and a multi-layer perception;

[0017] S42, taking the residual result as the input of SW-MSA and the second FG-MSA, splicing and fusing the output of SW-MSA and the output of the second FG-MSA to obtain a second result, and performing residual calculation on the second result and the output of the second result after passing through a layer normalization layer and a multi-layer perception;

[0018] S43, performing linear embedding on the residual result calculated in S42, taking it as the input of W-MSA and the first FG-MSA in S41, and performing 6 iterations in turn to obtain a feature map containing decoupled background features and target features.

[0019] Optionally, the step S5 comprises:

[0020] The feature map is decoded by adopting transposed convolution to obtain a high-resolution feature map, and is up-sampled to obtain an infrared image consistent with the size of the input image.

[0021] In a second aspect, the present application further provides an infrared image generation device based on a multi-modal self-attention mechanism generative adversarial network, comprising:

[0022] An image segmentation module is configured to segment an input visible light image to obtain image blocks of the same size and perform linear compression on the image blocks.

[0023] A local processing module is configured to perform multi-layer window-based local self-attention calculation and introduce adaptive correlation masks in the compressed image blocks to obtain a local correlation expression of the image.

[0024] A global processing module is configured to perform multi-layer fast global self-attention calculation between the compressed image blocks to obtain a global correlation expression of the image.

[0025] A fusion module is configured to fuse the local correlation expression and the global correlation expression of the image to obtain a feature map containing decoupled background features and target features.

[0026] A decoupling module is configured to decode and up-sample the feature map to output an infrared image consistent with the size of the input image.

[0027] In a third aspect, the present application further provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method according to any one of the first aspect when executing the computer program.

[0028] In a fourth aspect, the present application further provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the method according to any one of the first aspect.

[0029] Compared with the prior art, the above technical scheme of the present application can achieve the following beneficial effects:

[0030] 1. The application provides an infrared image generation method based on a multi-modal self-attention mechanism generative adversarial network, which constructs a multi-modal self-attention mechanism generative adversarial network model (MMSA-GAN), constructs local correlation expression and global correlation expression of an image respectively, and fuses the two, decouples the target and the background in the image block in the feature extraction stage, and generates an infrared image with visual effect and correct infrared features. Compared with the traditional infrared image generation algorithm, the generated infrared image has more correct infrared features and better visual effect.

[0031] 2. The infrared image generation method based on the multi-modal self-attention mechanism generative adversarial network provided by the application fuses the local correlation expression and the global correlation expression of the image, obtains the expression on the potential feature space of the background and the target, introduces a decoupled feature extraction method, and the output infrared image result contains better infrared image evaluation indexes; a fast global self-attention mechanism is constructed, the MMSA-GAN model obtains a global receptive field at the beginning while reducing the amount of calculation; in the local self-attention calculation, the adaptive correlation mask is introduced, and the robustness of the MMSA-GAN model on different data sets is improved. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 is a flowchart of the infrared image generation method based on the multi-modal self-attention mechanism generative adversarial network provided by the embodiment of the application;

[0033] Figure 2 is a model framework diagram of the infrared image generation method based on the multi-modal self-attention mechanism generative adversarial network provided by the embodiment of the application, wherein (a) is an image preprocessing module, (b) is a multi-modal self-attention module, (c) is a decoding module, and (d) is an adaptive correlation mask module;

[0034] Figure 3 is a multi-modal self-attention module model diagram in the infrared image generation method based on the multi-modal self-attention mechanism generative adversarial network provided by the embodiment of the application; wherein (a) is the overall architecture of the multi-modal self-attention module, and (b) is the internal details of the multi-modal self-attention block;

[0035] Figure 4 is a fast self-attention module model diagram in the infrared image generation method based on the multi-modal self-attention mechanism generative adversarial network provided by the embodiment of the application;

[0036] Figure 5The application provides a training process of an infrared image generation method based on a multi-modal self-attention mechanism generative adversarial network.

[0037] Figure 6 The application provides an infrared image generation method based on a multi-modal self-attention mechanism generative adversarial network, and a comparison chart of the method and other methods. DETAILED DESCRIPTION

[0038] In order to make the objects, technical solutions and advantages of the application clearer, the application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the application and not to limit the application. In addition, the technical features involved in each embodiment of the application described below can be combined with each other as long as they do not conflict with each other.

[0039] The content involved in the above embodiments will be described below in combination with a preferred embodiment.

[0040] Embodiment one

[0041] An infrared image generation method based on a multi-modal self-attention mechanism generative adversarial network, comprising:

[0042] S1, segmenting an input visible light image to obtain image blocks of the same size, and performing linear compression;

[0043] S2, performing multi-layer window-based local self-attention calculation in the compressed image blocks and introducing an adaptive correlation mask to obtain a local correlation expression of the image;

[0044] S3, performing multi-layer fast global self-attention calculation between the compressed image blocks to obtain a global correlation expression of the image;

[0045] S4, fusing the local correlation expression and the global correlation expression of the image to obtain a feature map containing decoupled background features and target features;

[0046] S5, decoding and up-sampling the feature map to output an infrared image consistent in size with the input image.

[0047] The calculation process of the self-attention mechanism represents the input features as a matrix X, and then performs three linear transformations on the matrix X through three learnable linear transformation matrices W Q , W K and W VQuery, key and value matrices Q, K and V are generated respectively. Similarity of Q and K is calculated by dot product, and the result is scaled by dividing by the square root of the vector dimension to obtain attention score matrix A, and then A is normalized by Softmax to generate attention weight matrix a, which represents the relevance of each input element to other elements. Then, the value matrix V is weighted and summed using the attention weight a to generate the final output, and each output vector captures the global dependency in the input sequence. Multi-head self-attention mechanism calculates multiple different attention heads in parallel, and finally the outputs of all heads are spliced and fused through a linear layer.

[0048] As shown in (a) of Figure 2 , a visible light image of HxWx3 is obtained, and the multi-modal self-attention mechanism based generative adversarial network (MMSA-GAN) provided in the scheme first divides the input visible light image into visible light image blocks of size n H x n W , and then linearly compresses the image blocks to compress n H x n W three-channel pixel points into a 1x1xN tensor. Subsequent processing only needs to process the compressed tensor, which greatly reduces the training time required.

[0049] Optionally, W-MSA and SW-MSA in Swin Transformer are used in step S2 to perform self-attention calculation within the window, and an adaptive correlation mask is introduced in SW-MSA.

[0050] W-MSA is a multi-head self-attention mechanism based on local windows. The input features are divided into multiple fixed-size non-overlapping windows, and then self-attention is calculated independently within each window. This method reduces the computational complexity from the usual O(N 2 ) to O(Mxw 2 ), where M is the number of windows and w is the size of the window. This localized calculation is very efficient, but because the windows do not overlap, it limits the exchange of information between windows and makes it difficult to capture global context.

[0051] To solve the problem of lack of interaction between windows in W-MSA, Swin Transformer introduces SW-MSA. The core idea is that when the input feature map is windowed, the window position is shifted relative to the previous layer by half the window size. SW-MSA will shift the entire window division to the right and down by half the window size. This shift causes overlapping areas between windows, allowing information to be exchanged between windows through the overlapping areas.

[0052] In Swin Transformer, only SW-MSA uses mask when calculating self-attention in window, and the parameters of the mask are fixed according to the position. The role of this mask is only to provide an efficient way for the calculation of SW-MSA, and it does not consider the possible correlation between non-adjacent tokens in the splicing window and the possible different strengths of the correlation between adjacent tokens in the ordinary window. It is not efficient to rely solely on the self-attention matrix to determine the correlation between different tokens when optimizing. In order to enable the mask to provide a certain guiding role when W-MSA and SW-MSA perform self-attention calculation, we propose an adaptive correlation mask. The size of the mask is consistent with the size of the self-attention matrix, and the parameters of the mask remain independent between different stages. After calculating the self-attention matrix, the mask is fused with the self-attention matrix, and then through softmax, the strength of the correlation between different tokens is determined. The adaptive correlation mask will be optimized for different datasets, and will efficiently complete the correlation guidance when performing self-attention calculation.

[0053] As shown in (a) of Figure 3 MMSA-GAN will perform four-stage processing on the input, and each stage contains a pair of multi-modal self-attention module and a linear embedding calculation. As shown in (b) of Figure 3 The multi-modal self-attention module will perform Figure 2The output of (a) is subjected to continuous self-attention calculation. The multi-modal self-attention module uses W-MSA and SW-MSA in Swin Transformer to perform self-attention calculation within the window; the calculation method of W-MSA is consistent with Swin Transformer. Through the multi-head self-attention mechanism of the local window, the input features are divided into multiple non-overlapping windows of fixed size, and the self-attention calculation is performed independently within each window. By limiting the calculation range of attention, the calculation efficiency of the MMSA-GAN model is improved, but due to the non-overlapping between windows, the information exchange between windows is limited, making it difficult to capture the global context. After calculating W-MSA, its output will be fused with the output of FG-MSA as the input of the second part of the multi-modal self-attention Block. When SW-MSA divides the input feature map into windows, the window position is shifted relative to W-MSA. SW-MSA will shift the entire window division to the right and down by half the window size. This shift causes a window of SW-MSA to contain multiple parts of the W-MSA divided window, thereby enabling information exchange between different windows of W-MSA. As the depth of the MMSA-GAN model increases, the receptive field obtained by the local self-attention calculation based on the window will continuously increase. During the calculation of SW-MSA, the adaptive correlation mask is fine-tuned according to the current training dataset, as shown in (d) of Figure 2 . This allows the self-attention calculation of SW-MSA to have a mask value that is more suitable for the current dataset, providing better robustness when generating infrared images during subsequent inference.

[0054] Optionally, the step S3 includes:

[0055] A fast global self-attention mechanism FG-MSA is used between the compressed image blocks to compress each window into a vector that can be used for self-attention mechanism dot product calculation, and multi-head self-attention calculation is performed between windows to obtain a global correlation expression of the image.

[0056] In order to allow the MMSA-GAN model to have a global receptive field from the beginning, the MMSA-GAN model is allowed to obtain a global receptive field from the beginning. Considering the resource consumption of global self-attention calculation, FG-MSA is proposed. As shown in Figure 4 , FG-MSA divides the MMSA-GAN model into multiple windows and obtains a tensor representing the window through an encoder. By allowing each feature vector to calculate multi-head self-attention with all window representation tensors, fast global self-attention calculation is achieved.

[0057] The scheme proposes a fast global self-attention mechanism FG-MSA. FG-MSA performs multi-head self-attention calculation between windows by compressing each window into a vector that can be used for self-attention mechanism dot product calculation. Assuming there are h x w windows, each window has a size of M x M, and the depth of the feature vector within the window is C, then the time complexity of global self-attention calculation and FG-MSA is respectively:

[0058] O(G-MSA)=4hwC 2 +2(hw) 2 C

[0059]

[0060] In the case of fixed size, O(FG-MSA) will be linear complexity of hw for any size of input image.

[0061] Optionally, the step S4 comprises:

[0062] S41, splicing and fusing the output of W-MSA and the output of the first FG-MSA to obtain a first result, and performing residual calculation on the first result and the output of the first result after passing through a layer normalization layer and a multi-layer perception;

[0063] S42, taking the residual result as the input of SW-MSA and the second FG-MSA, splicing and fusing the output of SW-MSA and the output of the second FG-MSA to obtain a second result, and performing residual calculation on the second result and the output of the second result after passing through a layer normalization layer and a multi-layer perception;

[0064] S43, performing linear embedding on the residual result calculated in S42, taking the linear embedding result as the input of W-MSA and the first FG-MSA in S41, and performing 6 iterations in sequence to obtain a feature map containing decoupled background features and target features.

[0065] Referring to Figure 3 (b) in the description, the multi-modal self-attention Block takes the output of the previous stage as input, splices the output of W-MSA and the output of the first FG-MSA through a contact module to obtain a first result, and performs a residual calculation on the first result and the output of the first result after passing through a layer normalization layer and a multi-layer perception to obtain a self-attention calculation fusion result of W-MSA and the first FG-MSA. And taking the self-attention calculation fusion result of W-MSA and the first FG-MSA as the input of SW-MSA and the second FG-MSA, a new round of fusion self-attention calculation is performed to realize the deep fusion of local self-attention and global self-attention.

[0066] The fusion result of the self-attention calculation of the W-MSA and the second FG-MSA is taken as the input of the SW-MSA and the first FG-MSA in the next level module, and the MMSA-GAN model can complete the decoupling of the target and the background in the visible light image through 6 iterations in turn by deepening the depth.

[0067] Optionally, the step S5 comprises:

[0068] The feature map is decoded by using the transpose convolution to obtain a high-resolution feature map, and is up-sampled to obtain an infrared image consistent with the size of the input image.

[0069] By using the transpose convolution technology, the latent space representation of the background and the target can be decoded and up-sampled into an infrared image with the same size as the input. The transpose convolution expands the spatial dimension of the feature map by applying a convolution kernel to the feature map. The latent space representation of the background and the target is initially regarded as the output generated by the conventional convolution, and then is up-sampled to obtain an infrared image with the same size as the input image. The MMSA-GAN model adopts the same training strategy as CycleGAN, and is trained unsupervisedly under the condition of no paired data set by introducing the cycle consistency loss.

[0070] Unlike standard convolution, the purpose of transpose convolution is to expand the spatial dimension of the feature map, which is usually achieved by applying a convolution kernel to the feature map. The latent space representation of the background and the target is first regarded as the output generated by the conventional convolution, and then is “restored” to the original high-resolution feature map by the inverse operation. Specifically, the transpose convolution uses the same convolution kernel, but moves and fills in different ways during the calculation process, usually filling zeros around the input feature map and inserting new data between adjacent positions, which can effectively capture and reconstruct the details of the image.

[0071] The above ensures that the generated infrared image has correct infrared characteristics, so that the MMSA-GAN model can generate results consistent with the content of the input visible light vehicle image under the condition of unsupervised training. The whole training process is as shown in Figure 5 The training mainly consists of two parts. The first part generates infrared images from visible light images, and the second part generates visible light images from infrared images. The training processes of the two parts are completely consistent, and each part includes two generation models and one discrimination model. The parameters of the generators in each part are shared. In the first part, the generator G AB is a network for generating infrared images from visible light image outputs; the second generator G BA is a network for generating visible light images from infrared images. The discrimination model D BThe input of the discriminator model D is the infrared image generated by the generative model and the real infrared image, and the output is a scalar used to evaluate whether the input is a real infrared image. In the second part, the process of generating visible light from infrared is trained. The input of the discriminator model D A The input of the discriminator model D is the infrared image generated by the generative model and the real infrared image, and the output is a scalar used to evaluate whether the input is a real infrared image. In the second part, the process of generating visible light from infrared is trained. The input of the discriminator model D

[0072] To verify the effectiveness of the infrared image generation method based on the multi-modal self-attention mechanism generative adversarial network proposed in the present application, in a specific embodiment, model training and performance verification are combined with actual scenarios.

[0073] The specific steps are as follows:

[0074] (1) Construction of training data set

[0075] In order to comprehensively evaluate the advantages of the present scheme in generating infrared images containing targets, experiments will be carried out on the VisDrone-DroneVehicle data set. VisDrone-DroneVehicle contains 28439 pairs of RGB-infrared paired images, covering urban roads, residential areas, parking lots and other scenes, from daytime to night. Each scene contains a large number of vehicle targets. Considering the training and testing cost, the present scheme uses 8000 pairs of visible-infrared images for training and 1000 visible light images for testing.

[0076] (2) Evaluation index and training platform

[0077] For evaluation index, the present scheme selects SSIM, PSNR and LPIPS. These indexes have their own characteristics and comprehensively reflect different aspects of image quality. SSIM focuses on the structural information in the image, effectively evaluating human perception of image details. PSNR provides quantitative evaluation of overall brightness and contrast, and is commonly used to measure distortion in compressed images. LPIPS uses a deep learning model to calculate the perceptual difference between images, providing an approximate value closer to the human visual system experience. All experiments are carried out on NVIDIA GeForce RTX 3090 using the PyTorch framework.

[0078] (3) Experimental results

[0079] The proposed MMSA-GAN is compared with current visible-to-infrared image translation methods.

[0080] Reference Figure 6 The detailed differences between the generated features and results of each method are intuitively demonstrated. CycleGAN can preserve the original shape information of the input image as much as possible, thus achieving rough visible-to-infrared image translation. However, due to the lack of decoupling between the target and background in the feature extraction stage, both of them exhibit the same infrared features in the generated image. Although pix2pix can partially preserve the infrared features of the target by supervised training and adjusting the loss function on the body, it ignores the shape information of the input image during the generation process, resulting in severe distortion of the shape of the target and background in the generated infrared image. The results generated by MUNIT also face the problems of shape distortion and loss of infrared feature information. The proposed infrared image generation method based on multi-modal self-attention mechanism generative adversarial network not only preserves the original content information of the image, but also ensures that the generated infrared background and target contain accurate infrared features.

[0081] Table 1 shows the evaluation of the quality of the generated images using four indicators: SSIM, PSNR, and LPIPS. SSIM is mainly used to measure the similarity of two images in terms of brightness, contrast, and structure. Its calculation method takes into account the local features of the image, comparing brightness, contrast, and structural information. This method enables SSIM to more accurately reflect human perception of image quality, making it an important indicator for evaluating the fidelity of generated images. PSNR is an objective image quality evaluation method, whose basic concept is to evaluate image quality by comparing the mean square error between the real image and the generated image. LPIPS is a deep learning-based indicator that focuses on the perceptual features of images, aiming to more accurately measure their perceptual similarity. It uses a pre-trained convolutional neural network to extract deep features from images, and then calculates the distance between these features.

[0082] As can be clearly seen from Table 1, when performing visible-to-infrared translation in complex scenes, MMSA-GAN outperforms all other methods in terms of SSIM, PSNR, and LPIPS indicators. This indicates that MMSA-GAN effectively maintains image quality and perceptual similarity while enhancing data augmentation capabilities.

[0083] Table 1

[0084] SSIM PSNR LPIPS CycleGAN 0.5722 27.86 0.2444 pix2pix 0.6583 28.31 0.1841 munit 0.5539 28.25 0.2971 MMSA-GAN 0.6907 28.85 0.1835

[0085] The scheme is an infrared image generation method MMSA-GAN based on a multi-modal self-attention mechanism generative adversarial network. The MMSA-GAN model has a global receptive field at the beginning, and in the feature extraction stage, the local correlation expression and the global correlation expression of the image are fused to realize the decoupling of the background and the target, so that the generated infrared image has more correct infrared features. In the local self-attention calculation, an adaptive correlation mask is proposed, so that the mask will be different due to different training data sets, enhancing the stability of the MMSA-GAN model on different data sets. The technical problem of incorrect infrared features when generating infrared images through a generative adversarial network from complex visible light images containing targets is solved, high-quality infrared images with consistent output content are realized, and the content of the generated infrared vehicle image is consistent with the input under the condition of unsupervised training, without the need for paired training data.

[0086] Embodiment two

[0087] The application also provides an infrared image generation device based on a multi-modal self-attention mechanism generative adversarial network, comprising:

[0088] An image segmentation module is configured to segment the input visible light image to obtain image blocks of the same size and perform linear compression.

[0089] A local processing module is configured to perform multi-layer window-based local self-attention calculation on the compressed image blocks and introduce an adaptive correlation mask to obtain the local correlation expression of the image.

[0090] A global processing module is configured to perform multi-layer fast global self-attention calculation between the compressed image blocks to obtain the global correlation expression of the image.

[0091] A fusion module is configured to fuse the local correlation expression and the global correlation expression of the image to obtain a feature map containing decoupled background features and target features.

[0092] A decoupling module is configured to decode and upsample the feature map to output an infrared image with the same size as the input image.

[0093] The infrared image generation device based on the multi-modal self-attention mechanism generative adversarial network provided by the embodiment of the application is used to execute the infrared image generation method based on the multi-modal self-attention mechanism generative adversarial network provided by any embodiment of the application, and has the corresponding beneficial effects.

[0094] Embodiment three

[0095] The application further provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method according to any one of Embodiment One.

[0096] Embodiment Four

[0097] The application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the method according to any one of Embodiment One.

[0098] Those skilled in the art will easily understand that the above description is only the preferred embodiment of the application, and is not intended to limit the application, and any modification, equivalent replacement and improvement within the spirit and principle of the application should be included in the protection scope of the application.

Claims

1. An infrared image generation method based on a multi-modal self-attention mechanism generative adversarial network, characterized in that, The method comprises the following steps: S1, segmenting an input visible light image to obtain image blocks of the same size and performing linear compression on the image blocks; S2, performing multi-layer window-based local self-attention calculation in the compressed image blocks and introducing an adaptive correlation mask to obtain a local correlation expression of the image; S3, performing multi-layer fast global self-attention calculation between the compressed image blocks to obtain a global correlation expression of the image; S4, fusing the local correlation expression and the global correlation expression of the image to obtain a feature map containing decoupled background features and target features; S5, decoding and up-sampling the feature map to output an infrared image consistent in size with the input image; In step S2, W-MSA and SW-MSA in Swin Transformer are used to perform self-attention calculation within a window; and a fast global self-attention mechanism FG-MSA is used between the compressed image blocks; The step S4 comprises: S41, splicing and fusing the output of W-MSA and the output of the first FG-MSA to obtain a first result, and performing residual calculation on the first result and the output of the first result after passing through a layer normalization layer and a multi-layer perception; S42, taking the residual result as the input of SW-MSA and the second FG-MSA, splicing and fusing the output of SW-MSA and the output of the second FG-MSA to obtain a second result, and performing residual calculation on the second result and the output of the second result after passing through a layer normalization layer and a multi-layer perception; S43, performing linear embedding on the residual result calculated in S42 to obtain a feature map containing decoupled background features and target features.

2. The method of claim 1, wherein, In step S2, W-MSA and SW-MSA in Swin Transformer are used to perform self-attention calculation within a window, and an adaptive correlation mask is introduced in SW-MSA.

3. The method of claim 2, wherein, The step S3 comprises: A fast global self-attention mechanism FG-MSA is used between the compressed image blocks, each window is compressed into a vector that can be used for self-attention mechanism dot product calculation, and multi-head self-attention calculation is performed between the windows to obtain a global correlation expression of the image.

4. The method of claim 1, wherein, The step S5 comprises: The feature map is decoded by using transpose convolution to obtain a high-resolution feature map, and up-sampling is performed to obtain an infrared image consistent in size with the input image.

5. An infrared image generation device based on a multi-modal self-attention mechanism generative adversarial network, applied to the infrared image generation method based on the multi-modal self-attention mechanism generative adversarial network according to any one of claims 1-4, characterized in that, The method comprises the following steps: An image segmentation module is configured to segment an input visible light image to obtain image blocks of the same size and perform linear compression on the image blocks; A local processing module is configured to perform multi-layer window-based local self-attention calculation in the compressed image blocks and introduce an adaptive correlation mask to obtain a local correlation expression of the image; A global processing module is configured to perform multi-layer fast global self-attention calculation between the compressed image blocks to obtain a global correlation expression of the image; and a fusion module configured to fuse the local correlation representation and the global correlation representation of the image to obtain a feature map containing decoupled background features and target features; a decoupling module configured to decode and up-sample the feature map to output an infrared image consistent with the size of the input image. 6.An electronic device comprising a memory and a processor, the memory storing a computer program, wherein, The processor executes the computer program to implement the steps of the method of any one of claims 1 to 4.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 4.

Images