Infrared image enhancement method based on style migration and multistage fusion

Abstract

According to the infrared image enhancement method based on style migration and multi-level fusion, a style migration preprocessing step comprises five levels of coding and decoding modules, multi-scale feature fusion is realized through cross-level dense jump connection, an encoder compresses spatial dimensions step by step, and a decoder recovers resolution through deconvolution and cancels a batch normalization layer. Information loss of small target features in the standardization process is avoided, and a triple mixing loss function is adopted for training; in the super-resolution reconstruction step, curvelet decomposition is carried out on a style migration result, curve features such as an arc line are accurately captured by adopting a 36-direction wedge-shaped basis function, a local variance threshold mechanism is adopted for high-frequency sub-band fusion, and a style migration feature weight is given to a high-variance region; low-frequency sub-bands are dynamically weighted according to regional contrast, a residual dense block enhanced SRGAN architecture is adopted in a reconstruction stage, and training stability is improved in cooperation with a spectrum normalization discriminator. Through the mode, the technical problem of effectively improving the quality of the low-quality infrared image is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and infrared image processing, and more particularly to an infrared image enhancement method based on style transfer and multi-level fusion, which is suitable for the enhancement processing of infrared images with low illumination and small-scale targets. BACKGROUND

[0002] Infrared imaging technology has become a key means in the fields of military reconnaissance and key area monitoring due to its all-weather working ability, but the three inherent defects caused by its imaging physical principle seriously restrict its application effect. First, the resolution of mainstream infrared sensors is generally low (such as 640x320 pixels), and the diffraction limit of long-wave infrared spectrum (7.5-14 μm) further limits the detail capturing ability. When the target distance exceeds 2000 meters, the key targets such as unmanned aerial vehicles only occupy 16x16 pixels in the image, resulting in the complete loss of features such as rotor structure. Second, the thermal radiation signal is easily affected by atmospheric attenuation, and the signal-to-noise ratio is 40-60% lower than that of visible light. When the temperature difference between the target and the background is ≤5℃ in a room temperature environment, the image gray level is compressed to less than 30% of the 8-bit dynamic range, significantly reducing the contrast. More essentially, thermal radiation imaging does not rely on surface reflection characteristics, resulting in a lack of edge texture in the image, and the hub details of the same vehicle are only presented as a blurred thermal blob in infrared imaging. These defects form the dilemma of "blurring-low contrast-no texture", and the urgent demand for high-resolution infrared images in the fields of military, monitoring, etc. forms a sharp contradiction.

[0003] Existing image enhancement techniques face serious limitations due to the mismatch between algorithms and physical characteristics. When the traditional interpolation method enlarges a 16x16 pixel target to 64x64 pixels, the PSNR is only 14.74 dB, and it fails to reconstruct high-frequency components; deep learning solutions such as SRGAN have a large receptive field when processing small targets, resulting in feature confusion, with the typical manifestation being the fusion of truck tires and body heat spots into invalid thermal clusters. In multi-scale fusion technology, although wavelet transform (DWT) can improve the overall contrast, the lack of direction selectivity will cause the sawtooth of curve edges, and the unmanned aerial vehicle rotor will present an edge curvature error of 12.7% after processing, far exceeding the human visual tolerance threshold. Reference-dependent methods such as histogram matching require high-quality reference images, but the standard reference images such as Camerman / Lena have significant differences in distribution from real infrared scenes, and forced matching will lead to distortion of the thermal distribution, such as the abnormal diffusion of firefighter helmet heat spots. The core of these problems lies in the fundamental contradiction between the "physical information sparsity of small targets" and the "strong prior assumptions of algorithms".

[0004] The urgent demand for small target enhancement in military, surveillance and other fields is driving technology to break through in the direction of physical-algorithm collaborative optimization. The field of military reconnaissance is facing severe challenges: the recognition rate of traditional methods for 16x16 pixel targets 2000 meters away is only 23%, while increasing the PSNR to 25 dB can increase the recognition rate to 89% and reduce the false hit rate by 70%. The medical diagnosis field is also urgent, and early screening of breast cancer relies on sub-millimeter hot spot resolution. The existing equipment can only image 3x3 pixels for 0.5 mm lesions, and the gray level confusion leads to a false negative rate as high as 34%. Technology breakthrough needs to be parallel on two tracks: the physical layer uses mid-wave infrared (3-5 μm) sensors to improve intrinsic resolution, but the cost increases by 300%; the algorithm layer needs to establish a "fuzzy-clear" style transfer model to break through the training data distribution constraint, develop a Curvelet fusion solution to solve the directional limitations of DWT, and retain weak features through residual skip connection of the nested U-Net structure. This "physical property cognition + data driven" enhancement framework will become the key path to decouple and reconstruct small target features under the premise of controllable cost. SUMMARY

[0005] The main purpose of the present application is to provide an infrared image enhancement method based on style transfer and multi-level fusion, aiming to solve the high-fidelity enhancement problem of small-scale targets in infrared images through collaborative optimization of physical property cognition and data driving.

[0006] In order to achieve the above purpose, the infrared image enhancement method based on style transfer and multi-level fusion provided by the present application comprises the following steps: Style transfer preprocessing step: receiving the input of the infrared image to be enhanced, outputting the style transfer image through the nested U-Net generator; the nested U-Net generator comprises a five-level encoding-decoding symmetric design structure, the encoder gradually compresses the spatial dimension and expands the feature channel through multi-layer convolution operation, forming a pyramid type feature abstraction; the decoder recovers the spatial resolution through deconvolution operation, and each level is connected with the corresponding encoder feature through dense skip connection; the nested U-Net generator cancels the batch normalization layer and adopts the gating mechanism to dynamically adjust the feature fusion weight; Super-resolution reconstruction step: performing Curvelet multi-scale decomposition on the style transfer image, integrating high-frequency and low-frequency sub-band features through adaptive fusion rules, generating a high-resolution image through an improved SRGAN, thereby realizing infrared image enhancement; the improved SRGAN generator comprises 8 residual dense blocks, and each dense block contains 4 layers of convolution and implements dense connection.

[0007] Further, the nested U-Net generator adopts a loss function with a triple constraint collaborative optimization strategy: The pixel-level mean square error (MSE) is used as a basic constraint to force the generated image to be consistent with the clear label in structure; The perceptual loss is calculated by the distance between the relu3_3 layer feature map of the pre-trained VGG16 network to drive the network to learn the texture characteristics consistent with human visual perception; The relative least square generative adversarial network (RLSGAN) constructs a dynamic game between the generator and the discriminator to improve the training stability. The pixel-level mean square error (MSE), perceptual loss, and relative least square generative adversarial network (RLSGAN) form the final loss function through weighting coefficients.

[0008] Further, the nested U-Net generator implements a training process of a phased optimization strategy. In the initial round of training, only the pixel-level loss is enabled for network coarse tuning to establish a basic mapping relationship. In the subsequent rounds, the perceptual loss and the adversarial loss are gradually introduced for fine optimization. The Adam optimizer is used with a dynamic learning rate decay mechanism. The initial learning rate is set and then decayed at key training nodes to effectively avoid local optimal traps. The FLIR ADAS dataset is selected as the training data.

[0009] Further, the FLIR ADAS dataset contains strictly registered blurred-clear infrared image pairs covering environmental temperature variation scenarios from -40℃ to 150℃, ensuring the environmental adaptability of the model.

[0010] Further, the number of channels in the encoder of the nested U-Net generator is configured as [64, 128, 256, 512], and the decoder at each level is connected to the encoder through a GRU gating unit. The gating mechanism dynamically adjusts the gating weight in the feature fusion weight according to the input image temperature.

[0011] Further, the integration of high-frequency and low-frequency sub-band features through adaptive fusion rules specifically includes: The high-frequency sub-band uses a local variance weighting mechanism to calculate the variance value within a certain size window, and preferentially retains the strong features of high-variance areas. The low-frequency sub-band dynamically allocates weights according to the regional contrast. Higher weights are given to high-contrast areas to enhance the target subject, and lower weights are given to low-contrast areas to suppress background noise. After fusion, iterative thresholding is used for reconstruction optimization to suppress artifacts.

[0012] Further, the improved SRGAN introduces three improvements based on the standard structure: The generator adds a residual dense block module to improve the feature reuse rate through four layers of convolutional dense connection; The discriminator implements spectral normalization to constrain the Lipschitz constant to 1.0 to enhance the training stability; The loss function adopts a triple constraint system, including a pixel-level content loss, a VGG19 feature-based perception loss and an adversarial loss, and the triple losses are weighted and fused.

[0013] Further, the curved wave multi-scale decomposition adopts a 4-layer decomposition structure, the number of directional subbands is greater than or equal to 36, and the size of the finest scale subband is 1024*1024.

[0014] The present application obtains an infrared image to be enhanced, performs style transfer enhancement on the infrared image to obtain an enhanced infrared image, and performs super-resolution reconstruction on the enhanced infrared image to obtain a high-quality infrared image. Through the above method, the style transfer and super-resolution reconstruction are performed on the infrared image to be enhanced, which effectively enhances the low-quality infrared image and solves the technical problem of how to improve the quality of the low-quality infrared image. BRIEF DESCRIPTION OF DRAWINGS

[0015] Fig. 1 The style transfer enhancement flowchart based on the nested U-Net is provided for example 1 of the present application.

[0016] Fig. 2 The super-resolution reconstruction method flowchart based on the curved wave domain fusion is provided for example 1 of the present application.

[0017] Fig. 3 The overall processing flowchart of example 1 of the present application. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application.

[0019] REFERENCE Figs. 1-3 The infrared image enhancement method based on style transfer and multi-level fusion includes a style transfer preprocessing step and a super-resolution reconstruction step.

[0020] Style transfer preprocessing step: receiving an infrared image input to be enhanced, outputting a style transfer image through a nested U-Net generator; the nested U-Net generator comprises a five-level encoding-decoding symmetric design structure, the encoder gradually compresses the spatial dimension through four layers of convolution operation with a step size of 2: the first layer uses a 64-channel 3x3 convolution kernel (step size 2) to downsample the 16x16 input image to 8x8; the second layer upgrades to a 128-channel convolution of the same size, outputting a 4x4 feature map; the third layer and the fourth layer use 256-channel and 512-channel convolutions, respectively, and expand the feature channels to form a pyramid feature abstraction, finally generating a 1x1x512 bottleneck feature vector; the decoder gradually recovers the spatial resolution through deconvolution upsampling operation, and after each deconvolution, the corresponding encoder features are concatenated (concat operation) to realize the fusion of low-order edge information and high-order semantic features. The nested U-Net generator cancels the batch normalization layer (BN) and uses a gating mechanism to dynamically adjust the feature fusion weight, avoiding the loss of small target features in the standardization process.

[0021] The number of channels in the encoder of the nested U-Net generator is configured as [64, 128, 256, 512], and the decoder of each level is connected with the encoder through a GRU gating unit; the GRU gating unit is used to dynamically adjust the feature fusion weight through a gating mechanism, wherein the gating weight is dynamically adjusted according to the temperature of the input image. Training adopts a curriculum learning strategy: only the pixel-level content loss is enabled for the first 300 rounds, and the perceptual loss and adversarial loss based on VGG19 features are introduced for the subsequent 700 rounds. In particular, the GRU dynamic adjustment of the jump connection weight is designed: when processing low-temperature targets (-20℃), the low-temperature feature weight is increased to 0.7 to suppress background noise; and when processing high-temperature targets (>80℃), the weight is reduced to 0.3 to avoid hot spot oversaturation.

[0022] The nested U-Net generator adopts a loss function with a triple constraint collaborative optimization strategy: The pixel-level mean square error MSE is used as the basic constraint to force the generated image to be consistent with the clear label in structure; The perceptual loss is calculated by the relu3_3 layer feature map of the pre-trained VGG16 network to drive the network to learn the texture characteristics consistent with human visual perception; The relative least squares adversarial loss RLSGAN constructs a dynamic game between the generator and the discriminator, which significantly improves the training stability compared with the traditional GAN loss; The pixel-level mean square error MSE, the perceptual loss, and the relative least square generative adversarial loss RLSGAN form a final loss function through a weighting coefficient, wherein the weights of the pixel-level mean square error MSE and the relative least square generative adversarial loss RLSGAN are respectively set as 0.6 and 0.1, and the weight of the perceptual loss is set as 10, so as to strengthen the detail generation capability; a mixed precision strategy (FP16) is implemented for training, batch_size=32, and the initial learning rate 0.0001 is decayed to 1e-6 through cosine annealing. The parameters are verified through thousands of experiments to balance the contradiction between edge sharpness and structural fidelity.

[0023] The nested U-Net generator implements a training process of a phased optimization strategy. In the initial 300 rounds of training, only the pixel-level loss is enabled to perform network coarse adjustment and establish a basic mapping relationship. In the subsequent 700 rounds, the perceptual loss and the adversarial loss are gradually introduced for fine optimization. An Adam optimizer is adopted in combination with a dynamic learning rate decay mechanism, and the initial learning rate is set as 0.0001. The training data is selected from the FLIR ADAS dataset. The FLIR ADAS dataset contains strictly registered blurred-clear infrared image pairs, covering environmental temperature variation scenarios from -40°C to 150°C, ensuring that the model has environmental adaptability. In this embodiment, it contains 10,000 strictly registered image pairs. Each blurred image is generated by a three-axis motion platform: horizontal displacement simulates wind speed influence (maximum offset ±15 pixels), rotation angle simulates target pitch (±30°), and Gaussian blur kernel (size 7×7, σ=1.5) simulates atmospheric scattering. This course learning type training strategy enables the network to first establish global structure cognition and then learn detail texture features, which conforms to the cognitive development law of the human visual system.

[0024] As the core generator, the nested U-Net generator retains the target intrinsic thermal features in view of the thermal radiation distribution characteristics of infrared images. The nested connection mode of the nested U-Net generator breaks through the single-level jump limitation of the traditional U-Net, realizes cross-level feature fusion, and enables low-order edge information and high-order semantic information to interactively enhance at multiple scales. Notably, the batch normalization layer (BN) is completely abandoned in the generator to avoid the homogenization loss of small target features in the standardization process.

[0025] Further reference Fig. 2 , a super-resolution reconstruction step: performing curve wave multi-scale decomposition on the style transfer image, integrating high-frequency and low-frequency sub-band features through an adaptive fusion rule, generating a high-resolution image through an improved SRGAN, and thus realizing infrared image enhancement; the improved SRGAN generator contains 8 residual dense blocks, and each dense block contains 4 convolution layers and implements dense connection.

[0026] The curvelet transform is used to replace the traditional wavelet transform, and the multi-directional decomposition of the image in the frequency domain is realized through the wedgelet basis function. The curvelet multi-scale decomposition adopts a four-layer decomposition structure, the number of directional subbands is greater than or equal to 36, and the size of the finest scale subband is 1024*1024. This kind of transform breaks through the directional limitation of the traditional wavelet, and expands the decomposition direction from 3 to at least 36, and is especially good at capturing complex curve features such as rotor arcs. In the decomposition process, the image is separated into high-frequency subbands representing details and low-frequency subbands carrying the main body heat distribution, forming a multi-scale feature map.

[0027] The curvelet multi-scale decomposition adopts a four-layer decomposition structure, and the specific is: The first layer of the basic scale uses 36-direction wedgelet basis functions, and the directional resolution is accurate to 10°, which can effectively capture the rotor arc feature; the second to fourth layers are gradually refined, and the size of the smallest scale subband reaches 1024*1024, which can analyze 0.5 pixel level details.

[0028] The adaptive fusion rule is used to integrate the high-frequency and low-frequency subband features, which includes: the high-frequency subband adopts a local variance weighting mechanism, and the variance value is calculated in a 3*3 window, and the strong features of the high-variance area (such as the target edge) are preferentially retained; the low-frequency subband dynamically allocates weights according to the regional contrast, and the high-contrast area is given a weight of 0.7 to strengthen the target main body, and the low-contrast area takes a weight of 0.3 to suppress background noise; this dual-track fusion mechanism not only preserves the edge integrity, but also optimizes the heat distribution distortion problem. After fusion, the reconstruction optimization is carried out through the iterative threshold method, and the threshold is set to 1.2 times the median of the subband energy to suppress the generation of artifacts.

[0029] The high-frequency subband fusion adopts a local variance threshold mechanism: the variance value is calculated in a 3*3 sliding window, and when the local variance is greater than 0.2 times the global variance (σ² is the total variance of the image), the style transfer image features are given a weight of 0.8; the low-variance area takes the original image features (weight 0.6) to suppress noise. The low-frequency subband fusion introduces a dynamic contrast weight: ; Wherein is the local contrast, and are the minimum and maximum contrasts of the whole image respectively, which ensures that the high-contrast target area (engine hot spot) retains 70% of the feature intensity.

[0030] The improved SRGAN introduces three improvements on the basis of the standard structure: The generator adds a residual dense block module, which improves the feature reuse rate through four-layer convolution dense connection; The discriminator implements spectral normalization processing, and the Lipschitz constant is constrained to 1.0 to enhance the training stability; The loss function adopts a triple constraint system, including a pixel-level content loss, a VGG16 feature-based perception loss and an adversarial loss, and the triple losses are weighted and fused.

[0031] A dynamic histogram matching module is specially designed as an optional extension for the medical scene, intelligently selects a similar image of a hot distribution from a preset reference library (including Camerman / Lena / Mandrill, etc.), adjusts the gray scale distribution through a cumulative distribution function mapping, and adopts a gradient protection mechanism to ensure that the high-frequency edges are not disturbed.

[0032] In the embodiment, the improved SRGAN, the generator includes 8 residual dense blocks, each dense block includes 4 layers of convolution (channel number 64), and the dense connection mode is adopted between layers, that is, the input of the nth layer is the channel splicing of the outputs of the first n-1 layers, and the feature reuse rate is increased by 40%. The discriminator implements spectral normalization processing, and the Lipschitz constant is constrained to be 1.0, and the local feature discrimination ability is enhanced in cooperation with the 70*70 PatchGAN structure. The loss function adopts a triple constraint of content-perception-adversarial: ; Among them, content loss, perception loss, adversarial loss.

[0033] The infrared image enhancement method based on style transfer and multi-level fusion provided by the application includes a style transfer enhancement method based on a nested U-Net and a curve wave domain fusion super-resolution reconstruction method; the style transfer enhancement method based on the nested U-Net includes five encoding and decoding modules, multi-scale feature fusion is realized through cross-level dense skip connection, the spatial dimension is compressed by the encoder step by step, and the resolution is restored by the decoder through deconvolution. The batch normalization layer is cancelled to avoid information loss of small target features in the standardization process. Training adopts a triple mixed loss function: a pixel-level mean square error loss guarantees structural consistency, a perception loss optimizes texture details, and a relative least square adversarial loss improves generation stability; the curve wave domain fusion super-resolution reconstruction method decomposes the style transfer result into a curve wave, and adopts 36-direction wedge basis functions to accurately capture curved features such as arcs. The high-frequency subband fusion adopts a local variance threshold mechanism, and the style transfer feature weight is given to the high-variance area; the low-frequency subband is dynamically weighted according to the regional contrast. The reconstruction stage adopts an SRGAN architecture enhanced by residual dense blocks, and cooperates with a spectral normalization discriminator to improve the training stability. In the above way, the technical problem of effectively improving the quality of low-quality infrared images is solved.

[0034] Those skilled in the art can easily understand that the above description is only the preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An infrared image enhancement method based on style transfer and multi-level fusion, characterized in that, It includes the following steps: Style transfer preprocessing steps: Receive the infrared image input to be enhanced, and output the style-transferred image through a nested U-Net generator; The nested U-Net generator contains a five-level encoder-decoder symmetrical design structure. The encoder gradually compresses the spatial dimension and expands the feature channels through multi-layer convolution operations to form a pyramid-shaped feature abstraction; The decoder restores the spatial resolution step by step through deconvolution operations, and each level establishes dense skip connections with the corresponding encoder features. The nested U-Net generator eliminates the batch normalization layer and uses a gating mechanism to dynamically adjust the feature fusion weights. Super-resolution reconstruction steps: The style-transfer image is decomposed into curve wave multi-scale decomposition, and high-frequency and low-frequency sub-band features are integrated through adaptive fusion rules. A high-resolution image is generated by an improved SRGAN, thereby achieving infrared image enhancement. The improved SRGAN generator contains 8 residual dense blocks, each containing 4 layers of convolution and implementing dense connections.

2. The infrared image enhancement method based on style transfer and multi-level fusion according to claim 1, characterized in that, The loss function of the nested U-Net generator employing a triple-constraint collaborative optimization strategy is as follows: Pixel-level mean square error (MSE) is used as a basic constraint to force the generated image to maintain structural consistency with the clear label. Perceptual loss is calculated by measuring the distance through the feature maps of the ReLU3_3 layer of the pre-trained VGG16 network, which drives the network to learn texture characteristics that conform to human visual cognition. The relative least squares adversarial loss RLSGAN constructs a dynamic game between the generator and the discriminator to improve training stability; The pixel-level mean squared error (MSE), perceptual loss, and relative least squares adversarial loss (RLSGAN) are weighted to form the final loss function.

3. The infrared image enhancement method based on style transfer and multi-level fusion according to claim 1, characterized in that, The training process of nested U-Net generators implements a phased optimization strategy. In the initial training rounds, only pixel-level loss is used for coarse network tuning to establish basic mapping relationships. In subsequent rounds, perceptual loss and adversarial loss are gradually introduced for fine optimization. The Adam optimizer is used in conjunction with a dynamic learning rate decay mechanism to avoid local optima traps. The FLIR ADAS dataset is used as the training data.

4. The infrared image enhancement method based on style transfer and multi-level fusion according to claim 3, characterized in that, The FLIR ADAS dataset contains rigorously registered blurry-sharp infrared image pairs, covering environmental temperature variations from -40°C to 150°C, ensuring the model's environmental adaptability.

5. The infrared image enhancement method based on style transfer and multi-level fusion according to claim 3, characterized in that, The number of encoder channels in the nested U-Net generator is configured as [64, 128, 256, 512]. Each stage of the decoder is connected to the encoder via a GRU gating unit. The gating mechanism dynamically adjusts the gating weights in the feature fusion according to the temperature of the input image: when processing low-temperature targets, the low-temperature feature weights are increased to suppress background noise, while the high-temperature targets are deweighted to avoid oversaturation of hot spots.

6. The infrared image enhancement method based on style transfer and multi-level fusion according to claim 1, characterized in that, The integration of high-frequency and low-frequency sub-band features through adaptive fusion rules specifically includes: The high-frequency subband adopts a local variance weighting mechanism, which calculates the variance value within a certain window size and prioritizes the retention of strong features in high-variance regions. The low-frequency subband dynamically allocates weights based on regional contrast, assigning higher weights to high-contrast regions to enhance the target subject and lower weights to low-contrast regions to suppress background noise. After fusion, reconstruction optimization is performed using an iterative thresholding method to suppress artifact generation.

7. The infrared image enhancement method based on style transfer and multi-level fusion according to claim 1, characterized in that, The improved SRGAN introduces three improvements based on the standard structure: The generator adds a residual dense block module, which improves feature reuse rate through dense connections of four convolutional layers; The discriminator undergoes spectral normalization and the Lipschitz constant is constrained to 1.0 to enhance training stability. The loss function adopts a triple constraint system, including pixel-level content loss, perceptual loss based on VGG19 features, and adversarial loss, and the three losses are weighted and fused.

8. The infrared image enhancement method based on style transfer and multi-level fusion according to claim 1, characterized in that, The curve wave multi-scale decomposition adopts a 4-layer decomposition structure, with ≥36 directional sub-bands and the smallest scale sub-band size is 1024×1024.

Images