Infrared image generation network based on GAN and deep attention mechanism

Abstract

The invention belongs to the technical field of computer vision and deep learning. The invention provides an infrared image generation network based on a GAN and a deep attention mechanism. The embodiment of the invention designs an infrared image generation network for target perception segmentation, an original visible light image is divided into a target area (human and vehicle) and a background area by adopting an image segmentation network, two generators are respectively adopted for processing, and a segmentation image obtained through the segmentation network is used as intermediate input to guide image generation. Meanwhile, image depth estimation is further fused into the model, an image depth map is used as an attention gating mechanism, generation of close shot and long shot can be effectively distinguished by the network, and for detail information, details of the close shot part are expected to be rich, and unimportant details of the long shot part are expected to be sparse. And the fidelity of the generated image is further improved, and the method has extremely high accuracy in the aspect of a generation task and has considerable practical value.

Description

Technical Field

[0001] This disclosure relates to the fields of computer vision and deep learning technology, and in particular to an infrared image generation network based on GAN and deep attention mechanism. Background Technology

[0002] In current research, the number of infrared image datasets is far less abundant than that of visible light image datasets, especially those publicly available and suitable for machine learning training. Because the collection of infrared data often involves proprietary technologies and sensitive information, sharing and public access to this data are difficult. Even when infrared image datasets exist, they typically cover only a limited number of scenes and objects, lacking the necessary diversity to meet the needs of various tasks such as face recognition, vehicle detection, and natural landscape analysis. Acquiring infrared image data of a scene is a major challenge when building infrared scene simulation systems. Currently, researchers commonly use infrared thermal imagers to capture infrared scene images in the field. While this method provides accurate data, it has significant limitations. First, field photography can only acquire data under specific environments; for special environments, the data volume is insufficient, leading to data scarcity. Furthermore, infrared imaging is easily affected by environmental conditions such as temperature, humidity, and background radiation; the acquisition process must be carried out under strict conditions to ensure data quality. In special application scenarios such as disaster monitoring, military reconnaissance, or nighttime security, infrared image acquisition often needs to be carried out at specific times or under specific conditions, which not only increases the difficulty of data acquisition but also increases time costs.

[0003] Traditional infrared image simulation methods suffer from drawbacks such as complex processes and difficulty in large-scale processing. To address this issue, image conversion algorithms become crucial. These algorithms convert visible light images into infrared images in specific environments, thereby expanding the infrared image database and providing sufficient data support for generating highly realistic infrared scene simulation technologies.

[0004] Generative Adversarial Networks (GANs), since their inception, have been continuously explored and improved over the years, and have been applied to many fields, such as image synthesis, style transfer, super-resolution problems, and image inpainting, all thanks to their excellent learning and generative capabilities. Compared to other generative models, GANs have been proven to have a better ability to generate high-quality, realistic images in most cases. Existing GANs typically use a single generator to process the entire image. Key targets (such as pedestrians or vehicles in surveillance footage) require high contrast to be accurately captured by the recognition system; while the background (like roads or green belts) needs to have its contrast reduced to avoid interference. However, traditional models train by mixing targets and backgrounds together, resulting in generated images with blurred targets and unremarkable features.

[0005] Therefore, it is necessary to improve one or more of the problems existing in the above-mentioned related technical solutions.

[0006] It should be noted that this section is intended to provide background or context for the technical solutions of this disclosure as set forth in the claims. The description herein does not constitute an admission that it is prior art simply because it is included in this section. Summary of the Invention

[0007] The purpose of this disclosure is to provide an infrared image generation network based on GAN and deep attention mechanism, thereby overcoming at least to some extent one or more problems caused by the limitations and defects of related technologies.

[0008] According to embodiments of this disclosure, an infrared image generation network based on GAN and deep attention mechanism is provided, comprising: Image segmentation network is used to extract target and background masks from visible light images to generate target and background images; A depth estimation network is used to extract depth information from visible light images and generate depth maps. The target generator is used to extract features from the target image to obtain a target fusion feature map, and then concatenates the target fusion feature map with the edge feature map along the channel dimension to obtain the target infrared image; The background generator maps the depth map to the feature space, concatenates it with the original feature map to generate a spatial attention heatmap, and performs feature reweighting and residual connection on the original feature map and the spatial attention heatmap to obtain the background infrared image. The image generation module is used to fuse the target infrared image and the background infrared image to obtain the final generated infrared image.

[0009] Furthermore, the target generator includes: The residual dense block includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a first feature concatenation module, an edge detection module, a local feature fusion layer, and a feature fusion module.

[0010] Furthermore, the specific process by which the target generator generates the infrared image of the target is as follows: The first convolutional layer extracts features from the target image to obtain the first feature map; The second convolutional layer extracts features from the target image and the first feature image to obtain the second feature image. The first convolutional layer extracts features from the target map and the second feature map to obtain the third feature map; The first convolutional layer extracts features from the target image and the third feature image to obtain the fourth feature image; The first feature stitching module stitches together the target image, the first feature image, the second feature image, the third feature image, and the fourth feature image. Then, the local feature fusion layer compresses the number of channels to obtain the target fused feature image. After performing edge detection processing on the target image, the edge detection module extracts features to obtain an edge feature map; The feature fusion module performs residual concatenation between the target fused feature maps along the channel dimension to obtain the target infrared image.

[0011] Furthermore, the background generator includes: Depth-gated attention module; among which, The deep gated attention module includes a feature extraction module, a deep feature extraction convolution, a reweighting module, and a second feature concatenation module.

[0012] Furthermore, the specific process by which the background generator generates the background infrared image is as follows: The background image is downsampled using the feature extraction module to extract the original feature image; The depth map is mapped to the feature space using a 3×3 depth feature extraction convolution. After the depth map and the original feature map are concatenated using the second feature concatenation module, a spatial attention heatmap is generated through two levels of 1×1 convolution. The reweighting module performs residual connections and feature reweighting on the spatial attention heatmap and the original feature map to obtain the reweighted features; The reweighted features are upsampled to obtain the infrared image of the target.

[0013] Furthermore, the specific process by which the image generation module generates the final infrared image is as follows: The target infrared image and the background infrared image are weighted and fused using target masks and background masks respectively to obtain the final generated infrared image.

[0014] Furthermore, a collaborative training strategy is employed to train the infrared image generation network, specifically including: Freeze the pre-trained depth estimation network and jointly optimize the segmentation network, target generator, and background generator; A spectral normalized discriminator is introduced and fine-tuned end-to-end through a joint loss function to complete the training; the joint loss function includes segmentation loss, perceptual loss, adversarial loss and depth consistency loss.

[0015] Furthermore, the segmentation loss is:

[0016] in, The value of the target mask predicted by the segmentation network at the i-th pixel. The value of the actual target mask at the i-th pixel; The perceived loss is:

[0017] in, To ultimately generate an infrared image, For real infrared images, For the i-th layer feature extractor, These are the layer weight coefficients. It is an L1 norm; The losses from the confrontation are:

[0018] in, For expectation operator, For discriminators; The deep consistency loss is:

[0019] in, Deep feature encoder.

[0020] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects: In the embodiments of this disclosure, the infrared image generation network based on GAN and deep attention mechanism described above, on the one hand, constructs a lightweight image segmentation network to extract target region masks and background masks from the input visible light image in real time. Based on the segmentation results, independent target generators and background generators are used in parallel: the target generator uses residual dense blocks to enhance thermal radiation features and outputs a high-contrast target infrared image; the background generator introduces a low-frequency constraint module to suppress high-frequency noise and reduce contrast requirements, and then achieves pixel-level accurate synthesis through a mask weighted fusion module to eliminate target edge blurring. On the other hand, firstly, depth information is extracted from the visible light image through a depth estimation network to generate a scene depth map; then, a depth information-guided attention gating module is designed to generate spatial attention weights using the depth map; finally, the attention weights are fused with the generator feature map to achieve differentiated generation of foreground targets and background regions. In addition, a staged adversarial training framework is adopted: the first stage freezes the pre-trained depth estimation network and jointly optimizes the segmentation network and the dual-branch generator; the second stage introduces a spectral normalization discriminator and performs end-to-end fine-tuning through a joint loss function to ensure the synergistic optimization of the saliency of target region thermal features and the fidelity of background structure. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0022] Figure 1 This diagram illustrates the structure of an infrared image generation network based on GAN and deep attention mechanism in an exemplary embodiment of this disclosure. Figure 2 This diagram illustrates the structure of the target generator in an exemplary embodiment of this disclosure. Figure 3 This diagram illustrates the structure of the background generator in an exemplary embodiment of this disclosure. Figure 4 This diagram shows a comparison of infrared images generated by the conventional method and the method of this application in exemplary embodiments of this disclosure. Figure 5 The total loss curve during training is shown in an exemplary embodiment of this disclosure; Figure 6 The loss curves of the target generator and the background generator during the training process are shown in the exemplary embodiments of this disclosure; Figure 7 The discriminator loss curve during training is shown in an exemplary embodiment of this disclosure; Figure 8 The peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) metric curves during the training process in an exemplary embodiment of this disclosure are shown. Detailed Implementation

[0023] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0024] Furthermore, the accompanying drawings are merely illustrative diagrams of embodiments of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities.

[0025] This example implementation provides an infrared image generation network based on GAN and deep attention mechanisms. (Reference) Figure 1As shown, the infrared image generation network based on GAN and deep attention mechanism may include: Image segmentation network is used to extract target and background masks from visible light images to generate target and background images; A depth estimation network is used to extract depth information from visible light images and generate depth maps. The target generator is used to extract features from the target image to obtain a target fusion feature map, and then concatenates the target fusion feature map with the edge feature map along the channel dimension to obtain the target infrared image; The background generator maps the depth map to the feature space, concatenates it with the original feature map to generate a spatial attention heatmap, and performs feature reweighting and residual connection on the original feature map and the spatial attention heatmap to obtain the background infrared image. The image generation module is used to fuse the target infrared image and the background infrared image to obtain the final generated infrared image.

[0026] The infrared image generation network based on GAN and deep attention mechanisms described above achieves the following: First, a lightweight image segmentation network is constructed to extract target region masks and background masks from the input visible light image in real time. Based on the segmentation results, independent target generators and background generators are used in parallel: the target generator uses residual dense blocks to enhance thermal radiation features and outputs a high-contrast target infrared image; the background generator introduces a low-frequency constraint module to suppress high-frequency noise and reduce contrast requirements, and then achieves pixel-level accurate synthesis through a mask weighted fusion module to eliminate target edge blurring. Second, a depth estimation network is first used to extract depth information from the visible light image to generate a scene depth map; then, a depth-information-guided attention gating module is designed to generate spatial attention weights using the depth map; finally, the attention weights are fused with the generator feature map to achieve differentiated generation of foreground targets and background regions. Furthermore, a phased adversarial training framework is adopted: the first phase freezes the pre-trained depth estimation network and jointly optimizes the segmentation network and the dual-branch generator; the second phase introduces a spectral normalization discriminator and performs end-to-end fine-tuning through a joint loss function to ensure the synergistic optimization of the saliency of target region thermal features and the fidelity of background structure.

[0027] Below, we will refer to Figures 1 to 8 The steps of the infrared image generation network based on GAN and deep attention mechanism described in this example embodiment will be explained in more detail.

[0028] In one embodiment, the process of the infrared image generation network based on a dual-branch GAN and a deep attention mechanism is as follows: Figure 1 As shown, the entire network comprises an image segmentation network, a depth estimation network, a target generator, a background generator, and an image generation module, and includes the following steps: Obtaining a Depth Map: A depth map is a special type of image where each pixel carries not only location information but also the distance between the corresponding object or surface and the camera or sensor, typically represented by grayscale values. Brighter pixels represent closer objects, while darker pixels represent farther objects. It helps the system extract 3D geometric information of a scene from a 2D image. Compared to RGB images, depth maps generally emphasize geometric information rather than visual texture. The depth map output is defined as: (1) Segmentation Network: Input a visible light image, output a segmentation map, i.e., a target mask and a background mask. The target segmentation map serves as guiding information, indicating the location of the target (people and vehicles). The segmentation network needs to be pre-trained, then the segmentation network is fixed, and then the generator and discriminator are trained. The output of the segmentation network is defined as: (2) Target Generator and Background Generator: In deep networks, gradients may gradually decay during backpropagation due to multiple nonlinear transformations, making it difficult to update deep parameters. Deep networks typically learn low-frequency global features (such as shape and structure), while shallow networks excel at capturing high-frequency details (such as texture and edges). Skip connections connect intermediate features of the encoder to corresponding layers of the decoder, preventing the loss of details during layer-by-layer downsampling / upsampling and significantly reducing blur and artifacts in the generated image. Residual dense modules enhance feature reuse, retaining more detailed information, suitable for small target detail reconstruction. The edge detection preprocessing module outputs a binary edge map. The feature fusion module takes the original feature map and edge feature map as input, concatenates the two features along the channel dimension, and outputs a fused feature map, organically fusing edge information with image features to improve the clarity of target boundaries. The target generator is defined as: (3) The background generator employs smoothing techniques: average pooling reduces high-frequency details; shallow feature extraction preserves global information; and a low-pass filter reduces background noise and artifacts. The outputs of the target generator and the background generator are summed to obtain the final infrared image. This design allows the target generator to focus on the target region, making the target more prominent and realistic. The background generator focuses on the background, reducing its contrast and preventing it from becoming overly complex.

[0029] Deep Gated Attention Mechanism: This module is the core innovation of this application. It guides the reweighting of the feature space through the depth map, solving a key problem in infrared image generation. This module is added to the generator's decoding stage, taking the current layer feature map of the generator and the depth map obtained from the visible light image through a depth estimation network as input. The attention map generation structure is as follows: (4) (5) (6) in It is the ReLU activation function. For the Sigmoid function, This is for channel splicing operations.

[0030] Discriminator: Target discriminator: The input is the target area of ​​a real infrared image. The discriminator needs to determine whether the infrared image of the target area is real. Global discriminator: The input is a complete real infrared image or a generated complete infrared image. It determines whether the entire image is real.

[0031] Model Training: Experiments were conducted on a rigorously registered dataset: This application uses paired visible and infrared image datasets, combined with existing publicly available infrared and visible light target detection registration datasets, and filters and cleans the data. This application constructs a multimodal benchmark dataset for complex all-weather scenarios, including image samples of pedestrians, vehicles, and other targets in various scenarios. The infrared and visible light images are rigorously registered, ensuring the success of subsequent model training and providing a data benchmark for the improvement and evaluation of subsequent algorithms. The training process employs a phased training strategy, where `train` is the training set and `test` is the test set, with 100 epochs of training.

[0032] Testing and verification: During the testing process, the visible light images of the test set are input into the trained generator, and its output is the corresponding infrared image. The performance evaluation metrics include peak signal-to-noise ratio (PSNR) and structural similarity (SSIM).

[0033] The technical solution of this application exhibits the following beneficial effects: (1) A dual-branch infrared image generation network based on target perception segmentation was designed. By segmenting the network, the target category (human / vehicle) is limited to achieve dual-branch decoupling. A differentiated generator structure is designed. High-contrast target generation and low-noise background generation require a differentiated feature processing mechanism, which solves the problem of target feature weakening caused by the coupling of target and background generation. (2) An attention gating mechanism network based on image depth estimation was designed. Infrared radiation characteristics are strongly correlated with the spatial position of objects, but existing methods lack effective utilization of depth information. The method in this application effectively solves the problem of image background distortion caused by the lack of depth information.

[0034] In one specific embodiment, the target generator and the background generator employ a differentiated structural design. The target generator takes a visible light image, a target mask, and a depth map as input. This generator is composed of dense residual blocks and outputs an infrared image of the target. The computation of the residual dense blocks satisfies:

[0035] The background generator takes a visible light image, a background mask, and a depth map as input. This generator consists of a low-frequency constraint module and outputs a background infrared image.

[0036] The infrared images generated by the target generator and the background generator are combined using a mask-weighted fusion operation to obtain the generated infrared image. The operation is as follows:

[0037] This method employs a differential feature processing mechanism to generate high-contrast features for the target region and low-noise features for the background region, thus solving the problem of insufficient target saliency in traditional methods.

[0038] The feature space is reweighted using depth maps. This allows the network to learn the depth information of the image, a feature not found in traditional generative adversarial networks (GANs). By guiding image generation with depth information, the problem of distorted thermal radiation distribution in generated images is solved. Both the target generator and the background generator embed depth-gated attention modules in the upsampling layer. The operation of this module satisfies the following:

[0039]

[0040] in Features extracted from the depth map using a 3×3 convolution. , This is the learnable parameter matrix.

[0041] In one specific embodiment Figure 1 This is a schematic diagram of the algorithm of this application, which consists of an image segmentation network, a depth estimation network, a target generator, and a background generator. The algorithm flow includes obtaining a depth map and a segmentation map, generating a target infrared image and a background infrared image through the target generator and background generator, respectively, and finally outputting an infrared image corresponding to the input visible light image. The specific implementation steps of this application are further explained below.

[0042] (1) Obtaining the depth map and segmentation map. The visible light image is processed through a pre-trained segmentation and depth estimation network to obtain the corresponding segmentation map (target mask and background mask) and depth map. These are used as intermediate inputs to the network, along with the original visible light image, and then fed into the image generation network. A schematic diagram of the segmentation map and depth map is shown below. Figure 1 As shown.

[0043] (2) Infrared images are generated using a target generator and a background generator. The target generator mainly consists of residual-dense modules, and its structure is as follows: Figure 2 As shown, the core structure is a dense connection mechanism (jump connection), mathematically expressed as: (7) The residual dense block in this application employs a feature extraction path consisting of four convolutional layers. Through the dense connections shown by the dashed lines, each convolutional layer receives the feature outputs of all preceding layers, mitigating the gradient vanishing problem. The local feature fusion layer compresses the number of channels using 1×1 convolutions (compression coefficient η=0.5), and finally preserves the original feature information through residual connections. The mathematical expressions for local feature fusion and residual connections are as follows: (8) (9) (3) Depth-gated attention mechanism. The depth-gated attention module adjusts feature generation through the following process: the depth map is mapped to the feature space through 3×3 convolution; after being concatenated with the generator feature map, it generates a spatial attention heatmap through two levels of 1×1 convolution; and the feature is reweighted through residual connection to enhance the feature representation ability of depth abrupt regions (such as target contours and occlusion boundaries), which significantly improves the structural distortion problem.

[0044] First, a depth map of the visible light image is obtained through a depth estimation module. Then, a depth-gated attention module is embedded into each upsampling layer of the background generator. Depth feature extraction maps the single-channel depth map to the same channel space as the feature map, expressed as: (10) in This represents the number of channels.

[0045] Next, feature concatenation and compression are performed, expressed as follows: (11) Following the generation of the attention map, the expression is: (12) Finally, feature reweighting and residual connections are used to ensure gradient propagation stability, outputting depth-aware weighted features: (13) The structure of the depth-gated attention module is as follows: Figure 3 As shown. The overall loss function system of the network is as follows: Segmentation loss: (14) Perceived loss: (15) Combat losses: (16) Deep consistency loss: (17) in This is a pre-trained deep feature encoder.

[0046] (4) Model Training. When training the discriminator, this paper uses the target regions of the real image and the generated image as inputs to the target discriminator, while simultaneously using the complete image as input to the global discriminator. The target generator needs to deceive both the target discriminator and the global discriminator, while the background generator needs to deceive the global discriminator. This design effectively separates the generation of the target and background, providing stronger supervision to the target region (with greater weights for the target discriminator and target reconstruction loss), thus generating more significant target regions, while maintaining lower contrast in the background region. To further improve the model's generalization ability and stability, random noise is added before the fully connected layer calculation to alleviate overfitting and enhance its performance in practical applications. The learning rate, batch size, and epochs are set to 1e-4, 8, and 100, respectively. The training curves are shown below. Figure 6 As shown.

[0047] Unless otherwise specified in this application, the operating system of the training and testing equipment is Ubuntu 20.04, equipped with an NVIDIA 3080 graphics card with 10GB of video memory, the algorithm compilation language is Python 3.9, and a GPU-based deep learning environment is built.

[0048] (5) Testing and verification. In order to verify the effectiveness of this application, the following experiments are conducted to illustrate the technical effects of this application.

[0049] The test algorithm compares the generated infrared images with those produced by the Pix2Pix algorithm (which establishes pixel-level mappings) and the CycleGAN algorithm (which removes pairing restrictions). For example, the generated image pairs... Figure 4 As shown.

[0050] pass Figure 4The comparison reveals that the infrared images generated by the improved generative adversarial network in this application are more realistic, more consistent with the imaging mechanism, and the targets are more prominent compared to those generated by the basic network. The generated images also perform better when applied to downstream tasks such as target detection. CycleGAN has an inherent drawback in infrared tasks—it lacks paired data supervision, making the generated results prone to deviating from physical laws. In terms of visual effects, the single generator of Pix2Pix / CycleGAN causes the target thermal features to be averaged by the background, while dual-branch generation has a significant optimization effect on the target area. For example, the pedestrian thermal outline generated by Pix2Pix is ​​blurry, while the method in this application can clearly preserve details such as helmets and backpacks, which is crucial for security scenarios.

[0051] Figures 5 to 8 The figures show the total loss curve during training, the generator loss curve, the discriminator loss curve, and the curves for two evaluation metrics (Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM)). From the overall convergence trend, the curves decrease rapidly before 2000 steps (initial learning phase), enter a stable oscillation period between 4000 and 8000 steps (typical adversarial equilibrium state of GANs), and converge to a stable low value (≈0.15) after 10000 steps, indicating that the model has reached equilibrium. The generator loss curve shows an oscillation amplitude <0.1, proving that the dual-branch architecture stabilizes the generation process, and the continuous decreasing trend of g_l1 (perceptual loss) verifies the effect of the deep attention mechanism on improving structural consistency. The convergence trend of the discriminator loss curve shows that the generated quality is close to real data. Finally, both PSNR and SSIM evaluation metrics show varying degrees of increase, verifying the effectiveness of this network design for image generation.

[0052] CycleGAN's cycle consistency loss amplifies high-frequency noise, while the low-frequency constraint module of the background generator in this application effectively solves the background noise problem. By comparison, it can be concluded that this application has significant improvements in thermal feature fidelity, structural integrity, and noise control.

[0053] In summary, comparative experiments demonstrate that the proposed infrared image generation network based on GAN and deep attention mechanism can reasonably and effectively address the shortcomings of traditional generation algorithms and generate images that better conform to the laws of infrared imaging. These advantages make this application of significant application value and broad market prospects in the field of infrared image generation.

[0054] The infrared image generation network based on GAN and deep attention mechanisms described above achieves the following: First, a lightweight image segmentation network is constructed to extract target region masks and background masks from the input visible light image in real time. Based on the segmentation results, independent target generators and background generators are used in parallel: the target generator uses residual dense blocks to enhance thermal radiation features and outputs a high-contrast target infrared image; the background generator introduces a low-frequency constraint module to suppress high-frequency noise and reduce contrast requirements, and then achieves pixel-level accurate synthesis through a mask weighted fusion module to eliminate target edge blurring. Second, a depth estimation network is first used to extract depth information from the visible light image to generate a scene depth map; then, a depth-information-guided attention gating module is designed to generate spatial attention weights using the depth map; finally, the attention weights are fused with the generator feature map to achieve differentiated generation of foreground targets and background regions. Furthermore, a phased adversarial training framework is adopted: the first phase freezes the pre-trained depth estimation network and jointly optimizes the segmentation network and the dual-branch generator; the second phase introduces a spectral normalization discriminator and performs end-to-end fine-tuning through a joint loss function to ensure the synergistic optimization of the saliency of target region thermal features and the fidelity of background structure.

[0055] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this disclosure, "a plurality of" means two or more, unless otherwise explicitly specified.

[0056] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0057] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. An infrared image generation network based on GAN and deep attention mechanism, characterized in that, include: Image segmentation network is used to extract target and background masks from visible light images to generate target and background images; A depth estimation network is used to extract depth information from visible light images and generate depth maps. The target generator is used to extract features from the target image to obtain a target fusion feature map, and then concatenates the target fusion feature map with the edge feature map along the channel dimension to obtain the target infrared image; The background generator maps the depth map to the feature space, concatenates it with the original feature map to generate a spatial attention heatmap, and performs feature reweighting and residual connection on the original feature map and the spatial attention heatmap to obtain the background infrared image. The image generation module is used to fuse the target infrared image and the background infrared image to obtain the final generated infrared image.

2. The infrared image generation network based on GAN and deep attention mechanism according to claim 1, characterized in that, The target generator includes: The residual dense block includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a first feature concatenation module, an edge detection module, a local feature fusion layer, and a feature fusion module.

3. The infrared image generation network based on GAN and deep attention mechanism according to claim 2, characterized in that, The specific process by which the target generator generates the infrared image of the target is as follows: The first convolutional layer extracts features from the target image to obtain the first feature map; The second convolutional layer extracts features from the target image and the first feature image to obtain the second feature image. The first convolutional layer extracts features from the target map and the second feature map to obtain the third feature map; The first convolutional layer extracts features from the target image and the third feature image to obtain the fourth feature image; The first feature stitching module stitches together the target image, the first feature image, the second feature image, the third feature image, and the fourth feature image. Then, the local feature fusion layer compresses the number of channels to obtain the target fused feature image. After performing edge detection processing on the target image, the edge detection module extracts features to obtain an edge feature map; The feature fusion module performs residual concatenation between the target fused feature maps along the channel dimension to obtain the target infrared image.

4. The infrared image generation network based on GAN and deep attention mechanism according to claim 3, characterized in that, Background generators include: Depth-gated attention module; among which, The deep gated attention module includes a feature extraction module, a deep feature extraction convolution, a reweighting module, and a second feature concatenation module.

5. The infrared image generation network based on GAN and deep attention mechanism according to claim 4, characterized in that, The specific process by which the background generator generates a background infrared image is as follows: The background image is downsampled using the feature extraction module to extract the original feature image; The depth map is mapped to the feature space using a 3×3 depth feature extraction convolution. After the depth map and the original feature map are concatenated using the second feature concatenation module, a spatial attention heatmap is generated through two levels of 1×1 convolution. The reweighting module performs residual connections and feature reweighting on the spatial attention heatmap and the original feature map to obtain the reweighted features; The reweighted features are upsampled to obtain the infrared image of the target.

6. The infrared image generation network based on GAN and deep attention mechanism according to claim 5, characterized in that, The specific process by which the image generation module generates the final infrared image is as follows: The target infrared image and the background infrared image are weighted and fused using target masks and background masks respectively to obtain the final generated infrared image.

7. The infrared image generation network based on GAN and deep attention mechanism according to claim 6, characterized in that, The infrared image generation network is trained using a collaborative training strategy, specifically including: Freeze the pre-trained depth estimation network and jointly optimize the segmentation network, target generator, and background generator; A spectral normalized discriminator is introduced and fine-tuned end-to-end through a joint loss function to complete the training; the joint loss function includes segmentation loss, perceptual loss, adversarial loss and depth consistency loss.

8. The infrared image generation network based on GAN and deep attention mechanism according to claim 7, characterized in that, The segmentation loss is: in, The value of the target mask predicted by the segmentation network at the i-th pixel. The value of the actual target mask at the i-th pixel; The perceived loss is: in, To ultimately generate an infrared image, For real infrared images, For the i-th layer feature extractor, These are the layer weight coefficients. It is an L1 norm; The losses from the confrontation are: in, For expectation operator, For discriminators; The deep consistency loss is: in, Deep feature encoder.

Images