Intelligent image enhancement system and method for multispectral night vision device

Abstract

This invention discloses an intelligent image enhancement system and method for multispectral night vision devices, belonging to the field of image enhancement technology. The method includes: firstly, decomposing the input image containing artifacts from a generative adversarial network (GAN) fusion image into different frequency sub-bands using multi-level discrete wavelet transform, and then applying a two-dimensional fast Fourier transform and peak detection algorithm to generate an artifact signature map that accurately represents the spatial location of artifacts; next, the artifact signature map is refined into a smooth spatial attention map by a lightweight guiding sub-network; this spatial attention map guides a lightweight correction network based on a U-Net architecture, focusing its correction capabilities on artifact regions; this correction network follows a residual learning paradigm, outputting only an artifact residual map; finally, the artifact residual map is added pixel-by-pixel to the artifact-containing GAN fusion image to obtain the final corrected image. This significantly improves the fidelity and reliability of the output results of existing complex GAN models.

Description

Technical Field

[0001] This invention relates to the field of image enhancement technology, and more specifically, to an intelligent image enhancement system and method for multispectral night vision devices. Background Technology

[0002] In the field of modern computer vision and image processing, Generative Adversarial Networks (GANs) have become a fundamental and powerful technology, achieving breakthroughs, particularly in tasks such as high-quality image synthesis, cross-domain image transformation, and multimodal data fusion. The core mechanism of GANs stems from their unique adversarial training framework, where the generator network and discriminator network compete and co-evolve in a zero-sum game, ultimately aiming to enable the generator to learn and reproduce the complex distributions of real-world data. However, the stability of this adversarial process is extremely challenging, especially when constructing deep, complex network architectures for generating high-resolution images. A widely recognized and deeply studied systematic challenge is the frequent presence of various "generative artifacts" in the output images of GANs. These artifacts are not simply random noise, but rather byproducts of structured features originating from specific structural components of the network, typically manifesting as unnatural periodic textures, checkerboard patterns, or subtle color distortions. For existing large and complex multispectral fusion GAN models that have already been trained with significant computational resources, eliminating these artifacts by directly modifying their core architecture is often impractical. Therefore, developing independent, efficient, and non-invasive post-processing techniques to improve and ensure the final fidelity and reliability of the generated images has become a crucial research direction.

[0003] In view of this, the present invention proposes an intelligent image enhancement system and method for multispectral night vision devices to solve the above problems. Summary of the Invention

[0004] To overcome the aforementioned deficiencies of the prior art and achieve the above objectives, the present invention provides the following technical solution: a multispectral night vision device intelligent image enhancement method, comprising:

[0005] Step F1: Based on the preset wavelet type and decomposition level of discrete wavelet transform, perform frequency domain joint analysis based on multi-level discrete wavelet transform and two-dimensional fast Fourier transform on the fusion image of generative adversarial network containing artifacts, and combine it with a peak detection algorithm based on fast Fourier transform peak detection threshold to locate the periodic frequency components caused by generative artifacts, and generate a single-channel artifact signature map with the same spatial size as the fusion image of generative adversarial network containing artifacts.

[0006] Step F2: Input the artifact signature image, along with the preset weights of the guiding subnetwork, the kernel size of the guiding subnetwork convolutional layer, and the kernel size of the output layer of the guiding subnetwork, into the guiding subnetwork and perform smoothing and nonlinear transformations to generate a spatial attention map with a value range between [0,1].

[0007] Step F3: Using a spatial attention map and based on the architecture defined by the kernel size of the correction network convolutional layer, the kernel size of the correction network pooling and upsampling layer, and the number of convolutional layers in the correction network module, guided weighting is applied to the decoder feature map of the lightweight correction network loaded with the weights of the lightweight correction network. The fused image of the generative adversarial network containing artifacts is then input into the lightweight correction network, and an artifact residual map is generated through the residual learning paradigm.

[0008] Step F4 involves adding the artifact residual image and the fused image from the generative adversarial network containing artifacts pixel by pixel to obtain the final corrected image.

[0009] Furthermore, the implementation method of step F1 includes:

[0010] Step F1F1: Perform a multi-level two-dimensional discrete wavelet transform based on wavelet type on the generative adversarial network fused image containing artifacts, with the number of layers specified by the decomposition level, to obtain the final low-frequency approximate subband and a set of multi-level high-frequency detail subbands.

[0011] Steps F1 and F2 involve traversing each high-frequency sub-band image in the multi-level high-frequency detail sub-band set, performing spectral analysis on it using two-dimensional fast Fourier transform, and identifying abnormal energy spikes using a peak detection algorithm based on the fast Fourier transform peak detection threshold, thereby generating a corresponding sub-band artifact fraction map for each high-frequency sub-band image.

[0012] Steps F1 and F3 involve upsampling all subband artifact score maps to the original size of the generative adversarial network fusion image containing artifacts, accumulating them, and then normalizing the accumulated results to generate a single-channel artifact signature map.

[0013] Furthermore, the implementation method of step F3 includes:

[0014] Step F3F1: The generative adversarial network fused image containing artifacts is input into the encoder path of a lightweight correction network based on the U-Net architecture and loaded with lightweight correction network weights. The encoder path generates multi-scale encoder feature maps and bottleneck layer feature maps by performing convolution and max pooling operations based on the correction network convolution kernel size, the correction network pooling and upsampling kernel size, and the number of convolutional layers in the correction network module.

[0015] In step F3F2, the bottleneck layer feature map and the multi-scale encoder feature map are input into the decoder path with the lightweight correction network weights loaded. Under the guidance of the spatial attention map, the decoder path generates the original network output by performing convolution and upsampling operations based on the correction network convolution kernel size, the correction network pooling and upsampling kernel size, and the number of convolutional layers in the correction network module.

[0016] Step F3F3 defines the original network output as an artifact residual map.

[0017] Further,

[0018] The implementation method of step F1F1 includes: Step F1F1F1, initializing the generative adversarial network fusion image containing artifacts as the current approximate subband; Step F1F1F2, initializing an empty multi-level high-frequency detail subband set; Step F1F1F3, when the number of iterations is less than the decomposition level, performing a single-level two-dimensional discrete wavelet transform on the current approximate subband based on the wavelet type to obtain a new next-level approximate subband and the current level high-frequency detail subband containing horizontal, vertical, and diagonal directions, updating the next-level approximate subband as the current approximate subband, and adding the current level high-frequency detail subband to the multi-level high-frequency detail subband set; Step F1F1F4, naming the current approximate subband at the end of the iteration as the final low-frequency approximate subband; The implementation method of step F1F1F3 includes: if the number of iterations is less than the decomposition level, then repeatedly executing steps F1F1F3F1 to F1F1F3F6. Step F1F1F3F1: Using the wavelet and scaling filter corresponding to the wavelet type, first perform a one-dimensional discrete wavelet transform along the rows of the current approximate subband to obtain the result, and then perform a one-dimensional discrete wavelet transform along the columns of the result to obtain the low-frequency approximate subband LL, the horizontal high-frequency detail subband LH, the vertical high-frequency detail subband HL, and the diagonal high-frequency detail subband HH; Step F1F1F3F2: Assign the LL subband to the next level approximate subband; Step F1F1F3F3: Combine the three subbands LH, HL, and HH into the current level high-frequency detail subband; Step F1F1F3F4: Update the next level approximate subband to the current approximate subband; Step F1F1F3F5: Add the current level high-frequency detail subband to the multi-level high-frequency detail subband set; Step F1F1F3F6: Increment the loop count by one.

[0019] Furthermore, the implementation methods for steps F1 and F2 include:

[0020] Step F1F2F1: Initialize an empty set of all subband artifact fractional maps;

[0021] Steps F1F2F2: For each high-frequency sub-band image in the multi-level high-frequency detail sub-band set, perform a two-dimensional fast Fourier transform to obtain a complex spectrum; calculate the amplitude spectrum of the complex spectrum to obtain the original amplitude spectrum; perform quadrant shifting on the original amplitude spectrum to move the zero-frequency component of the original amplitude spectrum to the center of the spectrum to obtain a centered amplitude spectrum.

[0022] Steps F1F2F3: On the centered amplitude spectrum, a two-dimensional local maximum detection algorithm is applied, and combined with the fast Fourier transform peak detection threshold for screening, to obtain a list of artifact energy peak coordinates.

[0023] Steps F1F2F4 involve generating a subband artifact fraction map for the high-frequency subband image based on the amplitude and position of each peak in the artifact energy peak coordinate list, and adding the subband artifact fraction map to the set of all subband artifact fraction maps.

[0024] Furthermore, the implementation methods for steps F1 and F3 include:

[0025] Step F1F3F1: Initialize a cumulative artifact score map with the same size as the fused image of the generative adversarial network containing artifacts and all elements being zero.

[0026] Steps F1F3F2 involve iterating through each sub-band artifact fraction map in the set of all sub-band artifact fraction maps.

[0027] Step F1F3F3: Based on the decomposition level corresponding to the current subband artifact score map, the subband artifact score map is upsampled to the original size of the generative adversarial network fusion image containing artifacts through bilinear interpolation to obtain the upsampled score map.

[0028] Steps F1F3F4 involve adding the upsampled score map to the cumulative artifact score map pixel by pixel, and then updating the cumulative artifact score map.

[0029] Steps F1F3F5: After the traversal is completed, the cumulative artifact score map is subjected to minimum-maximum normalization so that all pixel values ​​fall into the [0,1] interval, thus obtaining the artifact signature map.

[0030] Furthermore, the implementation method of step F3F1 includes:

[0031] Step F3F1F1: Construct the U-Net encoder path and load the corresponding weight parameters from the lightweight correction network weights into the U-Net encoder path; the overall architecture of the U-Net encoder path and each encoder module it contains are defined by the lightweight correction network weights.

[0032] Steps F3F1F2 take the image fused by the generative adversarial network containing artifacts as the initial input and pass through each module of the encoder path in sequence. In each module, the number of convolutional layers within the correction network module is used to perform convolution with the size of the correction network convolutional kernel, and one pooling operation with the size of the correction network pooling kernel and the upsampling kernel is performed. The output after convolution of each module is saved as an element of the multi-scale encoder feature map.

[0033] Step F3F1F3 defines the output of the deepest layer of the encoder path as the bottleneck layer feature map.

[0034] Furthermore, the implementation method of step F3F2 includes:

[0035] Step F3F2F1: Construct the U-Net decoder path and load the corresponding weight parameters from the lightweight correction network weights into the U-Net decoder path; the overall architecture of the U-Net decoder path and its various decoder modules are defined by the lightweight correction network weights.

[0036] Step F3F2F2: The bottleneck layer feature map is used as the initial input to the decoder path and passes through each decoder module in sequence. In each decoder module, the input feature map is upsampled, concatenated and convolved in combination with the corresponding level feature map from the multi-scale encoder feature map and the spatial attention map to generate a weighted feature map as the input to the next decoder module.

[0037] Steps F3F2F3 involve processing the weighted feature map output by the last decoder module through a 1×1 convolutional layer that corrects the network kernel size to obtain the original network output.

[0038] Furthermore, the specific implementation methods of steps F3F2F2 include:

[0039] Step F3F2F2F1: Perform a transposed convolution upsampling operation on the output of the previous decoder module, and concatenate it with the corresponding level feature map from the multi-scale encoder feature map in the channel dimension to obtain the concatenated feature map.

[0040] Step F3F2F2F2 involves performing convolution operations on the convolutional layers within the correction network module a number of times on the stitched feature map, and applying the ReLU activation function after each convolution operation to obtain the current decoder module output feature map.

[0041] Step F3F2F2F3 involves downsampling the spatial attention map to the same size as the output feature map of the current decoder module using a bilinear interpolation algorithm to obtain a scaled attention map. The scaled attention map is then multiplied element-wise with the output feature map of the current decoder module to obtain the weighted feature map.

[0042] Further,

[0043] The implementation method of step F1F2F2 includes: step F1F2F2F1, performing a two-dimensional fast Fourier transform on the high-frequency sub-band image to obtain a complex spectrum, wherein the calculation method of the transform includes: for each coordinate point in the frequency domain, traversing all pixels of the high-frequency sub-band image, multiplying the pixel value of each pixel by a complex exponential basis function, and then summing the results. The complex exponential basis function is generated by: calculating the product of the x-axis coordinate of the coordinate point in the frequency domain and the x-axis coordinate of the pixel of the high-frequency sub-band image, and the product of the x-axis coordinate of the coordinate point in the frequency domain and the product of the x-axis coordinate of the pixel of the high-frequency sub-band image in the frequency domain. The product of the y-axis coordinate of the coordinate point in the image and the y-axis coordinate of the pixel in the high-frequency sub-band image is obtained. The two products are normalized by the corresponding dimensions of the high-frequency sub-band image and then summed to obtain the sum value. The sum value is multiplied by the exponent of the result of a preset complex constant. Step F1F2F2F2 calculates the absolute value of each element in the complex spectrum to obtain the original amplitude spectrum. Step F1F2F2F3 performs a quadrant shift operation on the original amplitude spectrum by diagonally swapping the four quadrants of the original amplitude spectrum so that the zero-frequency component is located at the center of the image to obtain the centered amplitude spectrum.

[0044] The implementation method of steps F1F2F3 includes: Step F1F2F3F1, calling the standard two-dimensional peak finding algorithm to analyze the centered amplitude spectrum to identify all local maxima, and calculating the peak prominence of each maximum relative to its surrounding background to obtain a candidate peak list containing the coordinates of each peak and its corresponding prominence value; Step F1F2F3F2, filtering the candidate peak list according to the condition that the peak prominence is greater than the fast Fourier transform peak detection threshold to obtain a list of significant energy peaks; Step F1F2F3F3, extracting only the coordinates of all peaks from the list of significant energy peaks to form an artifact energy peak coordinate list.

[0045] A multispectral night vision intelligent image enhancement system implements the aforementioned multispectral night vision intelligent image enhancement method. The system includes:

[0046] The artifact signature module performs a joint frequency domain analysis based on multi-level discrete wavelet transform and two-dimensional fast Fourier transform on the generative adversarial network fused image containing artifacts, based on the preset wavelet type and decomposition level of discrete wavelet transform. It also combines a peak detection algorithm based on fast Fourier transform peak detection threshold to locate the periodic frequency components caused by generative artifacts and generate a single-channel artifact signature map with the same spatial size as the generative adversarial network fused image containing artifacts.

[0047] The attention module takes the artifact signature image and the preset weights, kernel size, and output kernel size of the guide subnetwork as input to the guide subnetwork, performs smoothing and nonlinear transformations, and generates a spatial attention map with a value range between [0,1].

[0048] The artifact residual module uses a spatial attention map and, based on the architecture defined by the kernel size of the correction network convolution, the kernel size of the correction network pooling and upsampling, and the number of convolutional layers within the correction network module, performs guided weighting on the decoder feature map of the lightweight correction network loaded with the weights of the lightweight correction network. The image fused by the generative adversarial network containing artifacts is then input into the lightweight correction network, and an artifact residual map is generated through the residual learning paradigm.

[0049] The correction module adds the artifact residual image and the fused image of the generative adversarial network containing artifacts pixel by pixel to obtain the final corrected image.

[0050] The beneficial effects of the intelligent image enhancement system and method for multispectral night vision devices of the present invention are as follows:

[0051] This invention generates artifact signature maps using a hybrid domain analysis method combining multi-level discrete wavelet transform (DWT) and two-dimensional fast Fourier transform (FFT). This approach allows for the effective separation of high-frequency details containing artifacts from the main image structure using the multi-resolution characteristics of DWT, and the precise identification and location of anomalous energy peaks caused by generative artifacts in the frequency domain using the high sensitivity of FFT to periodic signals. Therefore, this method generates a precise "signature map" characterizing the spatial location and intensity of artifacts, providing high-precision guidance for subsequent targeted correction and solving the problem of traditional methods failing to accurately distinguish artifacts from real image details.

[0052] This invention designs a lightweight guiding sub-network to perform smoothing and nonlinear transformations on the artifact signature image generated in the previous step to generate a spatial attention map. This feature optimizes the sparse and discontinuous artifact detection results that may exist in the original signature image into a spatially smooth and semantically coherent weight mask. This effectively avoids introducing new edge abrupt changes or visual defects due to the unsmoothness of the guiding signal during the correction process, ensuring a natural and smooth transition in the final correction effect.

[0053] This invention employs a residual learning paradigm and utilizes the aforementioned spatial attention map to guide a lightweight correction network based on the U-Net architecture. The effect of this feature is that by weighting the U-Net decoder feature map with the spatial attention map, the network's correction capability is surgically and precisely focused on artifact regions, avoiding unnecessary modifications to artifact-free areas. This design successfully achieves the complete elimination of artifacts while preserving all legitimate, high-frequency details and global structure in the original image with maximum fidelity.

[0054] The overall method of this invention is a non-intrusive post-processing module that obtains the final result by pixel-by-pixel adding the artifact residual map output by the network to the original image containing artifacts. This feature allows for the improvement of the quality of the output image without modifying or retraining any existing, complex GAN model. This makes it a modular, computationally efficient, and easily integrated quality assurance layer, significantly enhancing the reliability and credibility of generative AI outputs and reducing the cost and risk of deploying "high-potential" but flawed models to "production-grade" applications. Attached Figure Description

[0055] Figure 1 This is a schematic diagram of the process flow of the intelligent image enhancement method for multispectral night vision devices according to the present invention;

[0056] Figure 2 This is a schematic diagram of the system modules of the intelligent image enhancement system for multispectral night vision devices of the present invention;

[0057] Figure 3 This is a schematic diagram illustrating an application scenario of the intelligent image enhancement method for multispectral night vision devices according to the present invention. Detailed Implementation

[0058] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0059] Example 1

[0060] See Figure 1 The intelligent image enhancement method for multispectral night vision devices described in this embodiment includes steps F1 to F4.

[0061] This chapter provides a detailed, step-by-step technical implementation of the entire artifact correction process. Logically, the entire process is divided into two core stages. The core task of the first stage is to generate an artifact signature map that accurately represents the spatial distribution of artifacts by performing a hybrid domain processing technique on the input image, combining multi-resolution analysis and frequency domain analysis. The core task of the second stage is to use this signature map to guide a lightweight correction network based on residual learning to generate and apply artifact residual maps in a targeted manner, ultimately producing an artifact-free corrected image.

[0062] Step F1: Based on the preset wavelet type and decomposition level of discrete wavelet transform, perform frequency domain joint analysis based on multi-level discrete wavelet transform and two-dimensional fast Fourier transform on the fused image of the generative adversarial network containing artifacts. Combined with a peak detection algorithm based on the peak detection threshold of fast Fourier transform, locate the periodic frequency components caused by generative artifacts, and generate a single-channel artifact signature map with the same spatial size as the fused image of the generative adversarial network containing artifacts.

[0063] The fused image from the generative adversarial network (GAN) with artifacts is a three-channel RGB uint8 array with pixel values ​​ranging from 0 to 255. This data comes directly from the output of a pre-trained GAN, which exists as an independent third-party module specifically designed for multispectral image fusion tasks.

[0064] The wavelet type is a string whose value is hardcoded as 'db4'. This value was determined based on offline comparative experiments of the performance of various standard mother wavelets in image artifact analysis tasks. The choice of db4 was based on the fact that the db4 wavelet achieves the best balance between providing good time-frequency localization capabilities and maintaining computational efficiency. Its smoothing properties, compared to non-overlapping Haar wavelets, can more effectively capture the periodic high-frequency features unique to GAN artifacts, while avoiding the introduction of unnecessary block artifacts due to the non-smoothness of the analysis tool itself.

[0065] The decomposition level is an integer, hardcoded to 3. This value was determined based on an engineering trade-off analysis aimed at balancing frequency separation capability with spatial positioning accuracy. The choice of a 3-level decomposition is based on the fact that this decomposition depth is sufficient to effectively separate the image signal into frequency components of different scales, thereby separating artifact features from the macroscopic structure of the image, while ensuring that the highest frequency detail subbands still retain sufficient spatial resolution to accurately locate the spatial position of artifacts in the original image.

[0066] The Fast Fourier Transform (FFT) peak detection threshold is a floating-point number, determined through offline statistical analysis of a representative image sample set containing typical GAN-generated artifacts. This analysis includes: performing a two-dimensional FFT on the high-frequency subband of each image in the sample set to calculate its amplitude spectrum; then, calculating the prominence of all local energy peaks in the spectrum relative to local background noise; finally, by analyzing the statistical distribution of the prominence of these two types of peaks—artifact energy peaks and background spectral noise—a value capable of distinguishing between them with high confidence is set as the final FFT peak detection threshold.

[0067] Step F1 is designed based on a core physical insight: generative artifacts introduced by the upsampling operation in the GAN model often manifest as unnatural, periodic fine textures with specific frequencies and directions. According to Fourier theory, periodic signals in the spatial domain correspond to discrete, concentrated energy spikes in the frequency domain. Therefore, this step employs a dual analysis strategy: First, the multi-resolution analysis capability of Discrete Wavelet Transform (DWT) is utilized to effectively separate the macroscopic structure of the image from the microscopic details containing artifacts. Then, the high sensitivity of the Two-Dimensional Fast Fourier Transform (FFT) to global periodic patterns is leveraged to detect anomalous energy spikes caused by artifacts in the spectrum of each high-frequency sub-band, thereby achieving accurate artifact identification and spatial localization. This hybrid domain analysis method combining wavelet transform and Fourier transform integrates the former's localization capability in the time-frequency domain with the latter's detection capability in the pure frequency domain, achieving complementary advantages.

[0068] The implementation method of step F1 includes: Step F1F1, performing a multi-level two-dimensional discrete wavelet transform based on wavelet type with a specified number of decomposition levels on the fused image of the generative adversarial network containing artifacts, to obtain the final low-frequency approximate sub-band and a set of multi-level high-frequency detail sub-bands. Step F1F2, traversing each high-frequency sub-band image in the multi-level high-frequency detail sub-band set, performing spectral analysis on it using two-dimensional fast Fourier transform, and identifying abnormal energy spikes through a peak detection algorithm based on the peak detection threshold of fast Fourier transform, generating a corresponding sub-band artifact score map for each high-frequency sub-band image. Step F1F3, upsampling all sub-band artifact score maps to the original size of the fused image of the generative adversarial network containing artifacts, accumulating them, and normalizing the accumulation result to generate a single-channel artifact signature map.

[0069] The artifact signature map generated in step F1 is a single-channel, two-dimensional floating-point tensor with the same spatial dimensions as the input artifact-containing generative adversarial network fused image, and all pixel values ​​are normalized to the range of 0 to 1. Logically, the artifact signature map serves as a spatialized evidence map, where the intensity value of each pixel directly corresponds to the confidence level of the periodic high-frequency artifact detected at that location through DWT-FFT mixed-domain analysis. The purpose of the artifact signature map is to serve as the core input to the subsequent guiding sub-network, i.e., step F2, which smooths and refines this original evidence map to generate an attention mask more suitable for guiding correction.

[0070] The implementation method of step F1F1 includes: Step F1F1F1, initializing the generative adversarial network fusion image containing artifacts as the current approximate subband. Step F1F1F2, initializing an empty multi-level high-frequency detail subband set. Step F1F1F3, when the number of iterations is less than the decomposition level, performing a single-level two-dimensional discrete wavelet transform on the current approximate subband based on the wavelet type to obtain a new next-level approximate subband and the current level high-frequency detail subband containing horizontal, vertical, and diagonal directions, updating the next-level approximate subband as the current approximate subband, and adding the current level high-frequency detail subband to the multi-level high-frequency detail subband set. Step F1F1F4, naming the current approximate subband at the end of the iteration as the final low-frequency approximate subband.

[0071] The implementation method of step F1F1F3 includes: if the number of iterations is less than the decomposition level, then iteratively execute steps F1F1F3F1 to F1F1F3F6: Step F1F1F3F1: Using the wavelet and scaling filter corresponding to the wavelet type, first perform a one-dimensional discrete wavelet transform along the rows of the current approximate subband to obtain the result, then perform a one-dimensional discrete wavelet transform along the columns of the result to obtain the low-frequency approximate subband LL, the horizontal high-frequency detail subband LH, the vertical high-frequency detail subband HL, and the diagonal high-frequency detail subband HH. Step F1F1F3F2: Assign the LL subband to the next-level approximate subband. Step F1F1F3F3: Combine the three subbands LH, HL, and HH into the current-level high-frequency detail subband. Step F1F1F3F4: Update the next-level approximate subband to the current approximate subband. Step F1F1F3F5: Add the current-level high-frequency detail subband to the multi-level high-frequency detail subband set. Steps F1F1F3F6 increment the loop count by one.

[0072] The implementation method of step F1F2 includes: Step F1F2F1, initializing an empty set of all sub-band artifact fractional maps. Step F1F2F2, for each high-frequency sub-band image in the multi-level high-frequency detail sub-band set, performing a two-dimensional fast Fourier transform to obtain a complex spectrum; calculating the amplitude spectrum of the complex spectrum to obtain the original amplitude spectrum; performing quadrant shifting on the original amplitude spectrum, moving the zero-frequency component of the original amplitude spectrum to the center of the spectrum, to obtain a centered amplitude spectrum. Step F1F2F3, applying a two-dimensional local maximum detection algorithm to the centered amplitude spectrum, and combining it with a fast Fourier transform peak detection threshold for filtering, to obtain a list of artifact energy peak coordinates. Step F1F2F4, based on the amplitude and position of each peak in the artifact energy peak coordinate list, generating a sub-band artifact fractional map for the high-frequency sub-band image, and adding the sub-band artifact fractional map to the set of all sub-band artifact fractional maps.

[0073] The implementation method of step F1F2F2 includes: step F1F2F2F1, performing a two-dimensional fast Fourier transform on the high-frequency sub-band image to obtain a complex spectrum, wherein the calculation method of the transform includes: for each coordinate point in the frequency domain, traversing all pixels of the high-frequency sub-band image, multiplying the pixel value of each pixel by a complex exponential basis function and then summing the results, wherein the complex exponential basis function is generated in the following way: calculating the product of the x-axis coordinate of the coordinate point in the frequency domain and the x-axis coordinate of the pixel of the high-frequency sub-band image, and the product of the y-axis coordinate of the coordinate point in the frequency domain and the y-axis coordinate of the pixel of the high-frequency sub-band image; normalizing the two products by the corresponding dimensions of the high-frequency sub-band image and then summing them to obtain a sum value; taking the exponent of the sum value multiplied by a preset complex constant; the calculation formula is as follows: In the formula, Refers to the frequency coordinate The two-dimensional discrete Fourier transform result at the location, Refers to spatial coordinates The input image pixel values ​​are M and N, respectively, representing the height and width of the high-frequency sub-band image, and j is the imaginary unit. Step F1F2F2F2: Calculate the absolute value of each element in the complex spectrum to obtain the original amplitude spectrum. Step F1F2F2F3: Perform a quadrant shift operation on the original amplitude spectrum, diagonally swapping the four quadrants of the original amplitude spectrum so that the zero-frequency component is located at the center of the image, resulting in a centered amplitude spectrum.

[0074] The implementation method of steps F1F2F3 includes: Step F1F2F3F1, calling the standard two-dimensional peak finding algorithm to analyze the centered amplitude spectrum to identify all local maxima, and calculating the peak salience of each maximum relative to its surrounding background, obtaining a candidate peak list containing the coordinates of each peak and its corresponding salience value. Step F1F2F3F2, filtering the candidate peak list according to the condition that "peak salience is greater than the fast Fourier transform peak detection threshold", obtaining a list of significant energy spikes. Step F1F2F3F3, extracting only the coordinates of all peaks from the list of significant energy spikes to construct an artifact energy spike coordinate list.

[0075] The implementation method of step F1F3 includes: Step F1F3F1, initializing a cumulative artifact score map with the same size as the fused image of the generative adversarial network containing artifacts and all elements being zero. Step F1F3F2, traversing each sub-band artifact score map in the set of all sub-band artifact score maps. Step F1F3F3, based on the decomposition level corresponding to the current sub-band artifact score map, upsampling the sub-band artifact score map to the original size of the fused image of the generative adversarial network containing artifacts using bilinear interpolation to obtain the upsampled score map. Step F1F3F4, adding the upsampled score map and the cumulative artifact score map pixel by pixel, and updating the cumulative artifact score map. Step F1F3F5, after the traversal is completed, performing min-max normalization on the cumulative artifact score map so that all pixel values ​​fall within the range of the cumulative artifact score map. to The interval is used to obtain the artifact signature image.

[0076] The theoretical rationale for the hybrid analysis method employed in step F1 is based on a deep understanding of the artifact generation mechanism in generative adversarial networks (GANs) and the accurate application of the characteristics of classical signal processing transform tools. Existing research has clearly indicated that the upsampling module widely used in GANs systematically introduces periodic patterns into the generated images, the so-called "checkerboard artifacts." These artifacts manifest as isolated, high-energy spikes in the image's frequency domain. This scheme first employs the discrete wavelet transform, a standard multi-resolution analysis tool, to decompose the image signal into approximate subbands containing macroscopic structures and detail subbands containing microscopic details. This decomposition step is crucial because it effectively separates the high-frequency components, which are prone to concealing artifacts, from the main content of the image. Based on this, applying a two-dimensional fast Fourier transform to these separated high-frequency detail subbands significantly improves the detection sensitivity of characteristic artifact energy spikes and reduces background noise interference, thereby achieving a robust artifact localization method based on physical principles.

[0077] Before proceeding to step F2, the logical necessity of performing this step must be explained. While the artifact signature map generated in step F1 can accurately locate artifacts in the frequency domain, its direct conversion back to the spatial domain may result in a sparse point-like or line-like structure, potentially containing detection noise. Using such a raw, potentially spatially discontinuous map directly as the multiplicative attention mask for the subsequent correction network could introduce harsh edges or new visual discontinuities into the corrected image. Therefore, introducing a lightweight convolutional sub-network is necessary. This network functions as a data-driven, non-linearly smoothing filter, tasked with transforming the original "artifact evidence strength map" into a spatially smoother, semantically more coherent spatial attention map, thus ensuring a smooth and natural subsequent guided correction process.

[0078] Step F2 involves inputting the artifact signature image, along with the preset weights, kernel size, and output kernel size of the guiding subnetwork, into the guiding subnetwork for smoothing and nonlinear transformation, generating a value range within... to Spatial attention map between them.

[0079] The guiding subnetwork weights are a set of parameters stored in HDF5 file format. These parameters are obtained by optimizing a lightweight convolutional subnetwork in a separate training task. The goal of this training task is to learn a robust nonlinear mapping from potentially sparse and noisy artifact signature maps to spatially smooth and semantically coherent spatial attention maps.

[0080] The kernel size of the guiding subnetwork is a tuple whose value is hard-coded as (3, 3). This value is determined according to established standard practices in the field of deep learning for image processing tasks. A 3×3 kernel is the smallest size that can effectively capture the local information of a center pixel and its eight neighbors, providing a generally accepted optimal balance between ensuring an effective receptive field and controlling computational complexity.

[0081] The kernel size of the guide subnetwork's output layer convolutional layer is a tuple whose value is hard-coded as (1, 1). This value is determined based on its standard use in modern convolutional neural network architectures. 1×1 convolutional kernels are widely used to linearly combine and reduce the dimensionality of features along the channel dimension without altering the feature map space dimension. In this scenario, the kernel size of the guide subnetwork's output layer convolutional layer is used to efficiently aggregate multi-channel intermediate feature maps into a single-channel output map.

[0082] Artifact signature maps generated directly from frequency domain peak detection may suffer from sparsity and noise issues. Using them directly as attention maps could lead to overly harsh correction results. A lightweight convolutional sub-network is introduced, acting as a learnable nonlinear smoothing filter. This network learns a mapping from the "artifact evidence strength map" to the "optimal correction attention map," resulting in a smoother and more continuous spatial attention map. Finally, the use of the sigmoid activation function is necessary because it ensures that the output value strictly lies within the specified range. to Within this range, this is the standard requirement for attention masks in multiplication space, ensuring that subsequent operations are an effective "reweighting" of features.

[0083] The implementation method of step F2 includes: Step F2F1, constructing a guiding subnetwork consisting of two convolutional layers using the kernel size of the guiding subnetwork and a final convolutional layer using the kernel size of the guiding subnetwork output layer, and loading the guiding subnetwork weights. Step F2F2, passing the artifact signature image as input to the weighted guiding subnetwork to obtain the original attention output of the network. Step F2F3, applying the Sigmoid activation function to the original attention output of the network to normalize the value of each pixel to... to The interval is used to obtain the spatial attention map.

[0084] The spatial attention map generated in step F2 is a single-channel, two-dimensional floating-point tensor. Its spatial dimension is exactly the same as the input image fused by the generative adversarial network containing artifacts. Due to the final application of the sigmoid activation function, all pixel values ​​are strictly constrained to the range of 0 to 1. Logically, the spatial attention map represents a refined, spatially coherent weight mask, where regions with values ​​close to 1 represent artifact regions requiring the strongest correction, while regions with values ​​close to 0 represent artifact-free regions that need to be completely preserved. The purpose of the spatial attention map is to serve as a multiplicative gating signal in the subsequent main correction network, i.e., step F3, applied to the feature map of the decoder path, thereby achieving precise spatial guidance of the network's correction capabilities.

[0085] Before proceeding to step F3, the rationale behind the core architecture selection must be clarified. This scheme chooses U-Net as the core architecture for the correction network because U-Net has been widely validated for its superior performance in image-to-image conversion tasks. Its signature symmetric encoder-decoder structure, particularly the "skip connection" mechanism, allows the network to simultaneously utilize high-level semantic information from deep layers of the encoder and features from shallower layers that preserve high-frequency spatial details during the decoding phase. This fusion of multi-scale information is crucial for accurately eliminating artifacts while losslessly restoring the original image structure and texture. Furthermore, this scheme employs a residual learning paradigm, where the network does not directly learn to generate the final artifact-free image, but rather learns the difference between the predicted input and the ideal output—the artifact residual map. This design greatly simplifies the network's learning task, as it only needs to fit a typically sparser correction with a smaller numerical range, rather than the complex distribution of the entire image. This not only helps accelerate network convergence but also better protects most of the original image from unnecessary modification.

[0086] Step F3 uses a spatial attention map and, based on the architecture defined by the kernel size of the correction network convolutional network, the kernel size of the correction network pooling and upsampling, and the number of convolutional layers within the correction network module, performs guided weighting on the decoder feature map of the lightweight correction network loaded with the weights of the lightweight correction network. The fused image of the generative adversarial network containing artifacts is then input into the lightweight correction network, and an artifact residual map is generated through the residual learning paradigm.

[0087] The lightweight correction network weights are a set of parameters stored in HDF5 file format. These parameters were obtained through end-to-end optimization training of a U-Net architecture network on an offline supervised learning task on a dataset containing a large number of pairs of images with artifacts and their corresponding artifact-free ground truth images. The training process employed a composite loss function that combined L1 pixel reconstruction loss, adversarial loss, LPIPS perceptual loss, and feature matching loss. This function aimed to ensure that the artifact residual maps learned by the network, when combined with the original images, could produce a final corrected image that is pixel-accurate, perceptually realistic, and without introducing new blurring.

[0088] The size of the correction network convolutional kernel is a binary tuple, preferably hard-coded as (3, 3). This value is determined according to the same method used to determine the size of the guide network convolutional kernel, that is, following the standard design practice of convolutional neural networks in image processing applications, using 3×3 as the basic unit for capturing local spatial information.

[0089] The corrected network pooling kernel size and upsampling kernel size are a pair, preferably hard-coded as (2, 2). This value is determined based on the standard design principles of U-Net and its variant architectures. In the U-Net encoder path, using a 2×2 max-pooling layer with a stride of 2 is the standard method to halve the feature map space dimension; in the decoder path, using a 2×2 upsampling operation with a stride of 2 is the standard method to double the feature map space dimension.

[0090] The number of convolutional layers within the calibration network module is an integer, preferably hardcoded to 2. This value is determined based on the classic design pattern established in the original U-Net paper and its numerous subsequent successful variants. Stacking two consecutive convolutional layers at each scale level in each encoder or decoder module of U-Net has proven to be a robust and efficient balance between effectively increasing the network's nonlinear modeling capabilities and maintaining overall computational efficiency.

[0091] Step F3 employs U-Net as the core network architecture. U-Net, with its symmetrical encoder-decoder structure and signature "skip connections," is particularly well-suited for image-to-image transformation tasks, such as image restoration and correction. The encoder path captures contextual information and high-level semantic features of the image through layer-by-layer convolution and pooling; the decoder path gradually restores the spatial resolution of the image through upsampling. Crucially, the skip connections directly concatenate the shallow feature maps from the encoder (which retain high-frequency spatial details) with the corresponding feature maps from the decoder layers. This mechanism allows the network to utilize both deep semantic information and shallow detail information simultaneously during refined reconstruction, which is essential for accurately restoring the original image structure while eliminating artifacts.

[0092] The correction network is designed following the residual learning paradigm. Instead of directly learning to generate a complete "artifact-free image," the network learns the difference between the input and the target—the image fused by the generative adversarial network with artifacts—and the true artifact-free image, known as the artifact residual map. The final corrected image is obtained by adding the residual map output by the network to the original input image. This design greatly simplifies the learning task because the network only needs to focus on and fit a typically sparse, small-scale correction, rather than the complex distribution of the entire image. This helps accelerate network convergence and better preserve the overall structure and content of the image.

[0093] To meet the practical deployment needs of portable devices, the lightweight guidance subnetwork and lightweight correction network can be further optimized using existing artificial intelligence algorithms in their engineering implementation. For example, model size and computational complexity can be reduced through model quantization and pruning. The optimized model can be deployed on embedded chips, achieving low-power real-time processing while ensuring correction effectiveness, effectively avoiding image processing delays or stuttering caused by insufficient computing power of terminal devices. Furthermore, a system containing the aforementioned networks can also support offline operation. In environments without network connectivity, the final corrected image and analysis results, serving as crucial evidence, are temporarily stored locally and automatically synchronized with the command center once the network is restored.

[0094] The lightweight correction network weights are obtained through a multi-objective composite loss function. The loss function is obtained through joint optimization training. It is constructed by weighted summation of four core loss terms, aiming to balance four interrelated optimization objectives: pixel-level accuracy, perceptual realism, adversarial stability, and feature distribution matching. Its specific form is:

[0095]

[0096] In the formula, , , ,and These are the weight hyperparameters for each loss, which are set during the model training phase.

[0097] The following sections will introduce each loss term that constitutes this composite loss function:

[0098] L1 pixel reconstruction loss The construction method includes: calculating the L1 norm distance between the final corrected image and the true artifact-free image to ensure basic pixel-level fidelity. The formula is as follows: In the formula, Refers to the L1 pixel reconstruction loss value. Refers to a true, artifact-free image. Refers to the final corrected image. This means calculating the expectation of all samples in the dataset. This indicates the calculation of the L1 norm. This formula quantifies the average absolute difference at the pixel level between the final corrected image and the true artifact-free image, serving as the basis for optimizing network parameters to improve pixel-level fidelity. The calculation process is as follows: For each image pair in a batch, calculate the absolute value of the difference between the corresponding pixel values ​​of the final corrected image and the true artifact-free image. Then, sum the absolute differences of all pixels to obtain the sum of the absolute pixel differences of the image pair. Finally, calculate the average of the sum of the absolute pixel differences of all image pairs in the batch to obtain the final L1 loss.

[0099] Adversarial loss The construction method includes: The loss is based on an independent discriminator network, whose goal is to distinguish the final corrected image from the real artifact-free image. In this scheme, an adversarial loss of least-squares GAN is used to improve training stability. The adversarial loss of the correction network aims to make its generated images able to "fool" the discriminator, and its specific form is as follows:

[0100]

[0101] Where, This is the discriminator's output for the final corrected image. By minimizing this loss, the correction network is driven to generate images that are visually more difficult for the discriminator to distinguish as real or fake, thereby improving the perceptual quality and realism of the final result.

[0102] The LPIPS perceptual loss function is constructed by using a deep network pre-trained on a large-scale image classification task as a feature extractor. The final corrected image and the true artifact-free image are input into this network to extract feature activation maps from multiple intermediate layers. Then, the weighted L2 distance between the corresponding layer feature maps is calculated. Compared to traditional L1 / L2 loss, LPIPS (Learned Perceptual Image Patch Similarity) better simulates the perceptual judgment of the human visual system, effectively preventing image blurring and preserving rich texture details. Its calculation formula is as follows: In the formula, Refers to the LPIPS perceived loss value. It is the layer index in the pre-trained network. Indicates from the first Feature maps extracted from layers, and These are the real, artifact-free image and the final corrected image, respectively. and It is the first The height and width of the layer feature map It is used for calibration of the first The weight of layer importance, This indicates the calculation of the square of the L2 norm. This indicates element-wise multiplication, where h and w refer to the nth and wth elements, respectively. The height and width indices of pixels on the layer feature map. This formula quantifies the perceptual similarity between two images in the deep feature space, aiming to guide the network optimization direction so that the generated images are closer to the real images in human visual perception. The calculation process of this formula is as follows: First, the final corrected image and the real artifact-free image are input into a fixed pre-trained network; then, for each selected layer in the network... Extract the feature maps corresponding to the two images, calculate their difference, and then correlate the difference with the calibration weights of the layer. Perform element-wise multiplication; then, calculate the square of the L2 norm of this weighted difference map and normalize it by dividing by the size of the feature map; finally, sum the normalized results of all selected layers to obtain the final LPIPS loss.

[0103] Feature matching loss The construction method includes: this loss further utilizes a discriminator network; it requires that the feature activations generated by the final corrected image in multiple intermediate layers of the discriminator must match the feature activations generated by the real artifact-free image in the same layer. By forcing the generator to match the statistical distribution of real data in the feature space, this loss can greatly stabilize the adversarial training process and effectively prevent training problems such as mode collapse. Its calculation formula is as follows: In the formula, Refers to the feature matching loss value. It is an index of the intermediate layer of the discriminator network. It is the total number of selected intermediate layers. Indicates the discriminator's first... Feature maps output by the layer and These are the real, artifact-free image and the final corrected image, respectively. It is The total number of elements in the layer feature map. This indicates the calculation of the L1 norm. This formula is used to stabilize the adversarial training process by matching the intermediate feature representations of the generated and real images within the discriminator network, and to encourage the generator to learn the low-level feature distribution of the real data. The calculation process is as follows: First, the final corrected image and the real artifact-free image are input into the discriminator network; then, for each selected intermediate layer... The feature maps corresponding to the two images are extracted, the L1 norm between them is calculated, and normalized by dividing by the number of elements in the feature map of that layer. Finally, the normalization results of all selected layers are summed to obtain the final feature matching loss.

[0104] The implementation method of step F3 includes: Step F3F1, inputting the generative adversarial network fused image containing artifacts into the encoder path of a lightweight correction network based on the U-Net architecture and loaded with lightweight correction network weights. The encoder path generates multi-scale encoder feature maps and bottleneck layer feature maps by performing convolution and max pooling operations based on the correction network convolution kernel size, correction network pooling and upsampling kernel size, and the number of convolutional layers within the correction network module. Step F3F2, inputting the bottleneck layer feature map and multi-scale encoder feature map into the decoder path loaded with lightweight correction network weights. Guided by the spatial attention map, the decoder path generates the original network output by performing convolution and upsampling operations based on the correction network convolution kernel size, correction network pooling and upsampling kernel size, and the number of convolutional layers within the correction network module. Step F3F3, defining the original network output as an artifact residual map.

[0105] The artifact residual map generated in step F3 is a three-channel, two-dimensional floating-point tensor with the same spatial dimensions as the input GAN-fused image containing artifacts. Logically, the artifact residual map is a direct prediction and quantification of the generative artifacts present in the input image by the correction network. Numerically, it represents the amount of correction required to transform the GAN-fused image containing artifacts to an ideal artifact-free state. The purpose of the artifact residual map is to cancel out and remove artifacts in the final synthesis step, i.e., step F4, by performing pixel-by-pixel arithmetic addition with the original input image, thereby generating the final corrected image.

[0106] The implementation method of step F3F1 includes: Step F3F1F1, constructing a U-Net encoder path and loading the corresponding weight parameters from the lightweight correction network weights onto the U-Net encoder path; the overall architecture of the U-Net encoder path and its various encoder modules are defined by the lightweight correction network weights. Step F3F1F2, using the generative adversarial network fused image containing artifacts as the initial input, sequentially passing it through each module of the encoder path, performing convolution operations using the correction network kernel size a certain number of times within the correction network module, and a max pooling operation using the correction network pooling and upsampling kernel size once, and saving the output after convolution of each module as an element of the multi-scale encoder feature map. Step F3F1F3, defining the output of the deepest layer of the encoder path as the bottleneck layer feature map.

[0107] The implementation method of step F3F1F2 includes: Step F3F1F2F1, using the generative adversarial network fused image containing artifacts as the current feature map. Step F3F1F2F2, initializing an empty multi-scale encoder feature map list. Step F3F1F2F3, for each module in the encoder path, iteratively executing steps F3F1F2F3F1 to F3F1F2F3F3: Step F3F1F2F3F1, iteratively performing the number of operations within the correction network module on the current feature map, each operation involving applying a convolutional layer using the correction network kernel size, then applying the ReLU activation function, and naming the final result the module output feature map. Step F3F1F2F3F2, adding the module output feature map to the multi-scale encoder feature map list. Step F3F1F2F3F3, applying a max-pooling layer using the correction network pooling and upsampling kernel size to the module output feature map, and updating the result as the current feature map for use as input to the next module.

[0108] The implementation method of step F3F2 includes: Step F3F2F1, constructing the U-Net decoder path and loading the corresponding weight parameters from the lightweight correction network weights onto the U-Net decoder path; the overall architecture of the U-Net decoder path and its various decoder modules are defined by the lightweight correction network weights. Step F3F2F2, using the bottleneck layer feature map as the initial input of the decoder path, and sequentially passing it through each decoder module, in each decoder module, combined with the guidance of the corresponding level feature map from the multi-scale encoder feature map and the spatial attention map, performing upsampling, concatenation, and convolution processing on the input feature map to generate a weighted feature map as the input of the next decoder module. Step F3F2F3, processing the weighted feature map output by the last decoder module through a 1×1 convolutional layer using the kernel size of the correction network to obtain the original network output.

[0109] The specific implementation method of step F3F2F2 includes: Step F3F2F2F1, performing a transposed convolution upsampling operation on the output of the previous decoder module, and concatenating it with the corresponding layer feature map from the multi-scale encoder feature map in the channel dimension to obtain the concatenated feature map. Step F3F2F2F2, performing the number of convolution operations within the correction network module on the concatenated feature map, and applying the ReLU activation function after each convolution operation to obtain the output feature map of the current decoder module. Step F3F2F2F3, downsampling the spatial attention map to the same size as the output feature map of the current decoder module using a bilinear interpolation algorithm to obtain a same-scale attention map, and multiplying the same-scale attention map element-wise with the output feature map of the current decoder module to obtain the weighted feature map.

[0110] In step F3, the core mechanism for applying the spatial attention map to adjust the feature map of the U-Net decoder is based on the principle of "conditional computation," which has been successfully verified and applied in the feature-level affine transformation framework FiLM. The FiLM framework proves that by using information from one modality as a condition—in this case, the spatial attention map—it is possible to effectively perform dynamic, feature-wise linear modulation of the features of another modality—in this case, the decoder feature map. The element-wise multiplication operation used in this scheme is a specific implementation of feature modulation, playing a learnable spatial gating role and precisely guiding the network's repair capabilities. This design successfully applies the theoretical ideas of the FiLM framework to the artifact correction task, proving that its architectural choice has solid theoretical feasibility and rationality.

[0111] Step F4 involves adding the artifact residual image and the fused image from the generative adversarial network containing artifacts pixel by pixel to obtain the final corrected image.

[0112] The implementation process described in detail in this embodiment, at the data flow level, begins with a generative adversarial network fusion image containing artifacts. Through a hybrid domain analysis process combining discrete wavelet transform and fast Fourier transform, the image is first transformed into an artifact signature map that quantifies the artifact evidence. Next, the artifact signature map is smoothed and refined by a lightweight convolutional network to generate a spatial attention map as a guiding signal. Subsequently, the spatial attention map is used to guide a U-Net-based residual network, which processes the original input image and outputs an artifact residual map. Finally, by adding the artifact residual map to the original input, the final corrected image is produced as the final deliverable.

[0113] From a technical perspective, this implementation process efficiently and accurately removes various generative artifacts from GAN-fused images containing artifacts, including unnatural periodic textures and checkerboard patterns. While targeting and repairing artifacts, this process preserves the content details and global structure of artifact-free areas in the original image with maximum fidelity, avoiding the introduction of new blurring or information loss during the correction process. Ultimately, this solution, as a non-intrusive post-processing module, produces a high-fidelity corrected image that meets stringent application requirements in terms of both visual integrity and information accuracy without requiring any modification to the original GAN ​​model.

[0114] Example 2

[0115] See Figure 2 As shown, this embodiment provides an intelligent image enhancement system for multispectral night vision devices, the system including:

[0116] The artifact signature module performs a joint frequency domain analysis based on multi-level discrete wavelet transform and two-dimensional fast Fourier transform on the generative adversarial network fused image containing artifacts, based on the preset wavelet type and decomposition level of discrete wavelet transform. It also combines a peak detection algorithm based on fast Fourier transform peak detection threshold to locate the periodic frequency components caused by generative artifacts and generate a single-channel artifact signature map with the same spatial size as the generative adversarial network fused image containing artifacts.

[0117] The attention module takes the artifact signature image and the preset weights, kernel size, and output kernel size of the guide subnetwork as input to the guide subnetwork, performs smoothing and nonlinear transformations, and generates a spatial attention map with a value range between [0,1].

[0118] The artifact residual module uses a spatial attention map and, based on the architecture defined by the kernel size of the correction network convolution, the kernel size of the correction network pooling and upsampling, and the number of convolutional layers within the correction network module, performs guided weighting on the decoder feature map of the lightweight correction network loaded with the weights of the lightweight correction network. The image fused by the generative adversarial network containing artifacts is then input into the lightweight correction network, and an artifact residual map is generated through the residual learning paradigm.

[0119] The correction module adds the artifact residual image and the fused image of the generative adversarial network containing artifacts pixel by pixel to obtain the final corrected image.

[0120] Example 3

[0121] See Figure 3 This embodiment provides an application scenario for the intelligent image enhancement method for multispectral night vision devices according to the present invention. In this scenario, a UAV equipped with a multispectral night vision device first acquires raw multispectral image data when performing a mission. This raw data is transmitted to an onboard or edge-based GAN fusion module for processing. This module aims to fuse multimodal data into a more informative image, but this process may introduce generative artifacts, resulting in a "generative adversarial network fusion image containing artifacts." This artifact-containing GAN fusion image is then used as input and fed into the intelligent image enhancement method flow of the present invention for processing. The method flow of the present invention performs artifact correction on the image, outputting a clear, artifact-free "final corrected image." Finally, this high-quality corrected image is transmitted to the ground control and monitoring center for operators to use in applications with stringent requirements for image fidelity, such as real-time analysis, target recognition, or post-event analysis, significantly improving the practicality and reliability of the system.

[0122] Example 4

[0123] This embodiment provides an application scenario for the intelligent image enhancement method for multispectral night vision devices according to the present invention. During search and rescue operations at night or in low-visibility environments, a drone can be used as a platform to execute the intelligent image enhancement method described in this invention. The GAN fusion module on the drone outputs a generative adversarial network fused image containing artifacts. Subsequently, the aforementioned intelligent image enhancement method is applied in real time to correct artifacts in the fused image, resulting in a final corrected image. The final corrected image clearly highlights vital signs; even when the target is stationary, subtle changes in body temperature and respiration can be detected, thereby greatly improving the detection sensitivity and positioning accuracy for survivors.

[0124] Example 5

[0125] This embodiment provides an application scenario for the intelligent image enhancement method for multispectral night vision devices according to the present invention. In military reconnaissance or security monitoring, multispectral images acquired by front-end devices are fused using a GAN algorithm to enhance camouflage identification capabilities. The fusion process may produce artifacts. After the fused image containing artifacts is processed by the intelligent image enhancement method described in this invention, a final corrected image is output. This final corrected image has high fidelity, thus effectively combating interference such as infrared decoys or blinding effects from strong light, and can more accurately identify personnel wearing camouflage clothing or disguised equipment, improving battlefield awareness and security early warning capabilities.

[0126] Example 6

[0127] This embodiment provides an application scenario for the intelligent image enhancement method for multispectral night vision devices according to the present invention. During nighttime inspections of substations, chemical plants, etc., the thermal infrared and visible light images of the equipment are fused, and then processed by the intelligent image enhancement method described in this invention to eliminate artifacts, resulting in a final corrected image. Based on this final corrected image, operators or automated systems can accurately identify hot spots on electrical equipment, or combine visible light characteristics to observe damage to the equipment's exterior, achieving fault warnings and improving inspection efficiency and safety.

[0128] Example 7

[0129] This embodiment provides an extended application system based on the intelligent image enhancement system for multispectral night vision devices of the present invention. The extended application system uses the intelligent image enhancement system of the present invention as the core preprocessing unit and further integrates a live target contour reconstruction module. The live target contour reconstruction module receives the final corrected image output by the core preprocessing unit and inputs the final corrected image into a contour reconstruction network. The contour reconstruction network is a deep neural network whose structure may include an encoder path for capturing image context and a decoder path for generating completion information, and uses an attention mechanism to transfer effective features between the two. The contour reconstruction network is trained to learn prior knowledge of the target's structure, and can generate a highly realistic completed contour that conforms to the original structure of the target based on the contextual information and multispectral features in the image when receiving a final corrected image containing a partially occluded target.

[0130] Example 8

[0131] This embodiment provides an extended application system based on the intelligent image enhancement system for multispectral night vision devices of the present invention. The extended application system also uses the intelligent image enhancement system described in this invention as the core preprocessing unit, and further integrates a long-range weak target enhancement and recognition module. The long-range weak target enhancement and recognition module receives the final corrected image obtained by the core preprocessing unit. To enhance the details of long-range targets, the module can slice the regions containing potential targets in the final corrected image and input them into a super-resolution reconstruction network. The super-resolution reconstruction network is a deep convolutional neural network, whose network structure includes multiple residual modules for extracting deep features, and an upsampling module at the end of the network for reconstructing the feature map into a high-resolution image. The network enhances image details by learning an end-to-end mapping from low-resolution to high-resolution images, thereby supporting effective recognition of pixel-level targets.

[0132] For any parts not mentioned in this application, existing technologies may be used or referenced.

[0133] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0134] The above are merely embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A method for intelligent image enhancement in multispectral night vision devices, characterized in that, include: Step F1: Based on the preset wavelet type and decomposition level of discrete wavelet transform, perform frequency domain joint analysis based on multi-level discrete wavelet transform and two-dimensional fast Fourier transform on the fusion image of generative adversarial network containing artifacts, and combine it with a peak detection algorithm based on fast Fourier transform peak detection threshold to locate the periodic frequency components caused by generative artifacts, and generate a single-channel artifact signature map with the same spatial size as the fusion image of generative adversarial network containing artifacts. Step F2: Input the artifact signature image, along with the preset weights of the guiding subnetwork, the kernel size of the guiding subnetwork convolutional layer, and the kernel size of the output layer of the guiding subnetwork, into the guiding subnetwork and perform smoothing and nonlinear transformations to generate a spatial attention map with a value range between [0,1]. Step F3: Using a spatial attention map and based on the architecture defined by the kernel size of the correction network convolutional layer, the kernel size of the correction network pooling and upsampling layer, and the number of convolutional layers in the correction network module, guided weighting is applied to the decoder feature map of the lightweight correction network loaded with the weights of the lightweight correction network. The fused image of the generative adversarial network containing artifacts is then input into the lightweight correction network, and an artifact residual map is generated through the residual learning paradigm. Step F4 involves adding the artifact residual image and the fused image from the generative adversarial network containing artifacts pixel by pixel to obtain the final corrected image.

2. The intelligent image enhancement method for multispectral night vision devices according to claim 1, characterized in that, The implementation methods for step F1 include: Step F1F1: Perform a multi-level two-dimensional discrete wavelet transform on the generative adversarial network fused image containing artifacts, with the number of layers specified by the decomposition level and based on the wavelet type, to obtain the final low-frequency approximate subband and a set of multi-level high-frequency detail subbands. Steps F1 and F2 involve traversing each high-frequency sub-band image in the multi-level high-frequency detail sub-band set, performing spectral analysis on it using two-dimensional fast Fourier transform, and identifying abnormal energy spikes using a peak detection algorithm based on the fast Fourier transform peak detection threshold, thereby generating a corresponding sub-band artifact fraction map for each high-frequency sub-band image. Steps F1 and F3 involve upsampling all subband artifact score maps to the original size of the generative adversarial network fusion image containing artifacts, accumulating them, and then normalizing the accumulated results to generate a single-channel artifact signature map.

3. The intelligent image enhancement method for multispectral night vision devices according to claim 1, characterized in that, The implementation methods for step F3 include: Step F3F1: The generative adversarial network fused image containing artifacts is input into the encoder path of a lightweight correction network based on the U-Net architecture and loaded with lightweight correction network weights. The encoder path generates multi-scale encoder feature maps and bottleneck layer feature maps by performing convolution and max pooling operations based on the correction network convolution kernel size, the correction network pooling and upsampling kernel size, and the number of convolutional layers in the correction network module. In step F3F2, the bottleneck layer feature map and the multi-scale encoder feature map are input into the decoder path with the lightweight correction network weights loaded. Under the guidance of the spatial attention map, the decoder path generates the original network output by performing convolution and upsampling operations based on the correction network convolution kernel size, the correction network pooling and upsampling kernel size, and the number of convolutional layers in the correction network module. Step F3F3 defines the original network output as an artifact residual map.

4. The intelligent image enhancement method for multispectral night vision devices according to claim 2, characterized in that, The implementation method of step F1F1 includes: Step F1F1F1, initializing the generative adversarial network fusion image containing artifacts as the current approximate subband; Step F1F1F2, initializing an empty multi-level high-frequency detail subband set; Step F1F1F3, when the number of iterations is less than the decomposition level, performing a single-level two-dimensional discrete wavelet transform on the current approximate subband based on the wavelet type to obtain a new next-level approximate subband and the current level high-frequency detail subband containing horizontal, vertical, and diagonal directions, updating the next-level approximate subband as the current approximate subband, and adding the current level high-frequency detail subband to the multi-level high-frequency detail subband set; Step F1F1F4, naming the current approximate subband at the end of the iteration as the final low-frequency approximate subband; The implementation method of step F1F1F3 includes: if the number of iterations is less than the decomposition level, then iteratively execute steps F1F1F3F1 to F1F1F3F6: In step F1F1F3F1, using the wavelet and scaling filter corresponding to the wavelet type, first perform a one-dimensional discrete wavelet transform along the rows of the current approximate subband to obtain the result, and then perform a one-dimensional discrete wavelet transform along the columns of the result to obtain the low-frequency approximate subband LL, the horizontal high-frequency detail subband LH, the vertical high-frequency detail subband HL, and... Diagonal high-frequency detail subband HH; Step F1F1F3F2, assign the LL subband to the next level approximate subband; Step F1F1F3F3, combine the three subbands LH, HL, and HH into the current level high-frequency detail subband; Step F1F1F3F4, update the next level approximate subband to the current approximate subband; Step F1F1F3F5, add the current level high-frequency detail subband to the multi-level high-frequency detail subband set; Step F1F1F3F6, increment the loop count by one.

5. The intelligent image enhancement method for multispectral night vision devices according to claim 2, characterized in that, The implementation methods for steps F1 and F2 include: Step F1F2F1: Initialize an empty set of all subband artifact fractional maps; Steps F1F2F2: For each high-frequency sub-band image in the multi-level high-frequency detail sub-band set, perform a two-dimensional fast Fourier transform to obtain a complex spectrum; calculate the amplitude spectrum of the complex spectrum to obtain the original amplitude spectrum; perform quadrant shifting on the original amplitude spectrum to move the zero-frequency component of the original amplitude spectrum to the center of the spectrum to obtain a centered amplitude spectrum. Steps F1F2F3: On the centered amplitude spectrum, a two-dimensional local maximum detection algorithm is applied, and combined with the fast Fourier transform peak detection threshold for screening, to obtain a list of artifact energy peak coordinates. Steps F1F2F4 involve generating a subband artifact fraction map for the high-frequency subband image based on the amplitude and position of each peak in the artifact energy peak coordinate list, and adding the subband artifact fraction map to the set of all subband artifact fraction maps.

6. The intelligent image enhancement method for multispectral night vision devices according to claim 2, characterized in that, The implementation methods for steps F1 and F3 include: Step F1F3F1: Initialize a cumulative artifact score map with the same size as the fused image of the generative adversarial network containing artifacts and all elements being zero. Steps F1F3F2 involve iterating through each sub-band artifact fraction map in the set of all sub-band artifact fraction maps. Step F1F3F3: Based on the decomposition level corresponding to the current subband artifact score map, the subband artifact score map is upsampled to the original size of the generative adversarial network fusion image containing artifacts through bilinear interpolation to obtain the upsampled score map. Steps F1F3F4 involve adding the upsampled score map to the cumulative artifact score map pixel by pixel, and then updating the cumulative artifact score map. Steps F1F3F5: After the traversal is completed, the cumulative artifact score map is subjected to minimum-maximum normalization so that all pixel values ​​fall into the [0,1] interval, thus obtaining the artifact signature map.

7. The intelligent image enhancement method for multispectral night vision devices according to claim 3, characterized in that, The implementation methods for steps F3F1 include: Step F3F1F1: Construct the U-Net encoder path and load the corresponding weight parameters from the lightweight correction network weights into the U-Net encoder path; the overall architecture of the U-Net encoder path and each encoder module it contains are defined by the lightweight correction network weights. Steps F3F1F2 take the image fused by the generative adversarial network containing artifacts as the initial input and pass through each module of the encoder path in sequence. In each module, the number of convolutional layers within the correction network module is used to perform convolution with the size of the correction network convolutional kernel, and one pooling operation with the size of the correction network pooling kernel and the upsampling kernel is performed. The output after convolution of each module is saved as an element of the multi-scale encoder feature map. Step F3F1F3 defines the output of the deepest layer of the encoder path as the bottleneck layer feature map.

8. The intelligent image enhancement method for multispectral night vision devices according to claim 3, characterized in that, The implementation methods for steps F3 and F2 include: Step F3F2F1: Construct the U-Net decoder path and load the corresponding weight parameters from the lightweight correction network weights into the U-Net decoder path; the overall architecture of the U-Net decoder path and its various decoder modules are defined by the lightweight correction network weights. Step F3F2F2: The bottleneck layer feature map is used as the initial input to the decoder path and passes through each decoder module in sequence. In each decoder module, the input feature map is upsampled, concatenated and convolved in combination with the corresponding level feature map from the multi-scale encoder feature map and the spatial attention map to generate a weighted feature map as the input to the next decoder module. Steps F3F2F3 involve processing the weighted feature map output by the last decoder module through a 1×1 convolutional layer that corrects the network kernel size to obtain the original network output.

9. The intelligent image enhancement method for multispectral night vision devices according to claim 8, characterized in that, The specific implementation methods of steps F3F2F2 include: Step F3F2F2F1: Perform a transposed convolution upsampling operation on the output of the previous decoder module, and concatenate it with the corresponding level feature map from the multi-scale encoder feature map in the channel dimension to obtain the concatenated feature map. Step F3F2F2F2 involves performing convolution operations on the convolutional layers within the correction network module a number of times on the stitched feature map, and applying the ReLU activation function after each convolution operation to obtain the current decoder module output feature map. Step F3F2F2F3 involves downsampling the spatial attention map to the same size as the output feature map of the current decoder module using a bilinear interpolation algorithm to obtain a scaled attention map. The scaled attention map is then multiplied element-wise with the output feature map of the current decoder module to obtain the weighted feature map.

10. The intelligent image enhancement method for multispectral night vision devices according to claim 5, characterized in that, The implementation method of step F1F2F2 includes: step F1F2F2F1, performing a two-dimensional fast Fourier transform on the high-frequency sub-band image to obtain a complex spectrum, wherein the calculation method of the transform includes: for each coordinate point in the frequency domain, traversing all pixels of the high-frequency sub-band image, multiplying the pixel value of each pixel by a complex exponential basis function, and then summing the results. The complex exponential basis function is generated by: calculating the product of the x-axis coordinate of the coordinate point in the frequency domain and the x-axis coordinate of the pixel of the high-frequency sub-band image, and the product of the x-axis coordinate of the coordinate point in the frequency domain and the product of the x-axis coordinate of the pixel of the high-frequency sub-band image in the frequency domain. The product of the y-axis coordinate of the coordinate point in the image and the y-axis coordinate of the pixel in the high-frequency sub-band image is obtained. The two products are normalized by the corresponding dimensions of the high-frequency sub-band image and then summed to obtain the sum value. The sum value is multiplied by the exponent of the result of a preset complex constant. Step F1F2F2F2 calculates the absolute value of each element in the complex spectrum to obtain the original amplitude spectrum. Step F1F2F2F3 performs a quadrant shift operation on the original amplitude spectrum by diagonally swapping the four quadrants of the original amplitude spectrum so that the zero-frequency component is located at the center of the image to obtain the centered amplitude spectrum. The implementation method of steps F1F2F3 includes: Step F1F2F3F1, calling the standard two-dimensional peak finding algorithm to analyze the centered amplitude spectrum to identify all local maxima, and calculating the peak prominence of each maximum relative to its surrounding background to obtain a candidate peak list containing the coordinates of each peak and its corresponding prominence value; Step F1F2F3F2, filtering the candidate peak list according to the condition that the peak prominence is greater than the fast Fourier transform peak detection threshold to obtain a list of significant energy peaks; Step F1F2F3F3, extracting only the coordinates of all peaks from the list of significant energy peaks to form an artifact energy peak coordinate list.

11. A multispectral night vision device intelligent image enhancement system, characterized in that, The system for implementing the intelligent image enhancement method for multispectral night vision devices according to any one of claims 1-10 includes: The artifact signature module performs a joint frequency domain analysis based on multi-level discrete wavelet transform and two-dimensional fast Fourier transform on the generative adversarial network fused image containing artifacts, based on the preset wavelet type and decomposition level of discrete wavelet transform. It also combines a peak detection algorithm based on fast Fourier transform peak detection threshold to locate the periodic frequency components caused by generative artifacts and generate a single-channel artifact signature map with the same spatial size as the generative adversarial network fused image containing artifacts. The attention module takes the artifact signature image and the preset weights, kernel size, and output kernel size of the guide subnetwork as input to the guide subnetwork, performs smoothing and nonlinear transformations, and generates a spatial attention map with a value range between [0,1]. The artifact residual module uses a spatial attention map and, based on the architecture defined by the kernel size of the correction network convolution, the kernel size of the correction network pooling and upsampling, and the number of convolutional layers within the correction network module, performs guided weighting on the decoder feature map of the lightweight correction network loaded with the weights of the lightweight correction network. The image fused by the generative adversarial network containing artifacts is then input into the lightweight correction network, and an artifact residual map is generated through the residual learning paradigm. The correction module adds the artifact residual image and the fused image of the generative adversarial network containing artifacts pixel by pixel to obtain the final corrected image.

Images