Infrared spatio-temporal noise generation method based on hybrid neural representation

Abstract

This invention discloses an infrared spatiotemporal noise generation method based on hybrid neural representations. The method includes establishing an infrared denoising dataset containing paired indoor noise-clear video and unpaired outdoor noise-clear video; constructing an infrared spatiotemporal noise model based on hybrid neural representations and its training loss function; and building a spatiotemporal noise generator G and a spatiotemporal discriminator. This invention employs a divide-and-conquer approach, independently exploring the noise synthesis path from both spatial and temporal dimensions. By constructing a hybrid neural representation of noise, it deeply integrates the spatial and temporal embeddings of noise and implicitly models the complex spatiotemporal distribution of infrared noise through recurrent adversarial learning. This comprehensively and deeply characterizes the spatiotemporal properties of noise, providing new ideas and tools for noise modeling and analysis in the field of video processing.

Description

Technical Field

[0001] This invention relates to the field of noise modeling technology, specifically to a method for generating infrared spatiotemporal noise based on hybrid neural representations. Background Technology

[0002] Infrared imaging technology, with its ability to capture infrared radiation imperceptible to the human eye, has demonstrated broad application potential in various fields such as surveillance, medical diagnosis, and military reconnaissance. However, noise in imaging systems often leads to degradation of infrared video quality, thus limiting the further application of this technology. Recently, deep learning technology has made significant progress in video denoising, but its training process is highly dependent on noise-clean paired video data. In practical applications, collecting paired infrared video datasets is an extremely challenging task. Therefore, most learning-based video denoising methods instead use synthetic data for training, which is generated by applying noise models to clean videos. This means that the performance of these denoising methods in real-world scenarios is largely affected by the accuracy of the noise model.

[0003] Currently, a series of methods for noise modeling have been proposed, broadly categorized into two types: physics-based methods and learning-based methods. Physics-based noise models obtain the statistical distribution of different noise sources by analyzing the imaging process of infrared detectors. These methods typically classify the spatial noise of infrared imaging systems into random noise and fixed-pattern noise, and simulate it using a Gaussian distribution with specific parameters. However, due to the influence of circuit design and complex image signal processing, the spatial distribution of actual noise often exhibits non-Gaussian and nonlinear characteristics. Learning-based noise models simulate complex noise distributions by exploring the relationship between clean and noisy data. These models generate realistic noise samples by minimizing the domain difference between synthetic noise and actual observed noise. Compared with traditional physics-based noise models, these methods can extract noise characteristics that better match real-world scenarios from rich data. However, most existing learning-based noise models are limited to the single-frame image level, only simulating the spatial distribution characteristics of noise at a single moment.

[0004] However, regardless of whether the method is physics-based or learning-based, most noise models operate on a single frame of image, only representing the noise distribution at a single moment. Due to sensor hardware limitations or dynamic scene changes, many noise sources exhibit significant temporal correlations in practical applications. Although a few noise synthesis methods use simple Gaussian distributions or averaging methods to approximate the noise characteristics in the time dimension, the modeling accuracy is unsatisfactory. If the noise model cannot accurately reflect the spatiotemporal distribution characteristics of actual noise, it may limit the performance of video denoising algorithms in dynamic scenes.

[0005] Learning the spatiotemporal distribution of infrared noise is a challenging task. First, different types of infrared noise exhibit significantly different spatiotemporal statistical behaviors. Spatially, fixed-pattern noise and temporal stripe noise often appear as structured, large-area stripes, while random noise displays pixel-level random fluctuations. Temporally, fixed-pattern noise typically remains constant at fixed locations within the image, while temporal stripe noise and random noise exhibit high instability and rapid changes. Second, accurate modeling of the temporal correlation of infrared noise is crucial, especially in dynamic scenes. Due to the randomness and uncertainty of noise, traditional methods used to describe inter-frame correlation, such as optical flow algorithms, are not suitable for spatiotemporal modeling of noise. Summary of the Invention

[0006] The purpose of this invention is to provide an infrared spatiotemporal noise generation method based on hybrid neural representations to solve the technical problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] S1. Establish an infrared denoising dataset, which includes indoor paired noise-clear video dataset and outdoor unpaired noise-clear video dataset;

[0009] S2. Construct an infrared spatiotemporal noise model based on hybrid neural representations and the model's training loss function. The model includes a spatiotemporal noise generator G and a spatiotemporal discriminator;

[0010] The spatiotemporal noise generator G includes a spatial embedding generator SEG, a temporal embedding generator TEG, and a hybrid neural representation of noise HNeRN. The spatiotemporal discriminator consists of a spatial discriminator. and time discriminant It consists of two parts;

[0011] First, the spatial embedding generator SEG obtains the spatial embedding S by introducing a noise prior P with learnable parameters. Simultaneously, the temporal embedding generator TEG obtains the temporal embedding T through positional encoding of the frame index. Subsequently, the noisy hybrid neural representation HNeRN deeply fuses the spatial embedding S and the temporal embedding T to generate a noisy image. The spatiotemporal noise generator G employs a recurrent strategy based on noise maps during training, combining clear video... Degradation generates noisy video ;

[0012] S3. Input the infrared denoising dataset from S1 into the model in S2 for training to obtain a trained infrared spatiotemporal noise model based on hybrid neural representations, which will then be used to refine the clear video. Degraded to a noisy video by the spatiotemporal noise generator G .

[0013] Furthermore, in step S2, the spatial embedding generator SEG generates the spatial embedding S in the following specific way: the spatial embedding generator SEG consists of a 3x3 convolutional layer and three residual blocks. The spatial embedding generator SEG introduces a noise prior P with learnable parameters and modifies it with the noiseless image signal at time t. The input matrix of the SEG is concatenated and then subjected to preliminary feature extraction through a 3x3 convolutional layer. It is then mapped to a high-dimensional feature space through deep encoding of three residual blocks. Furthermore, the spatial embedding generator (SEG) also processes the noise signal at time t-1. Encoding was performed, and the final SEG generated a spatial embedding S containing context information; the temporal embedding generator TEG generated a temporal embedding T in the following specific implementation: the temporal embedding generator TEG consists of two parts: frequency position coding (FPE) and multilayer perceptron (MLP); first, FPE maps the frame index to a high-dimensional space, and then the MLP takes the encoded information as input to further mine deeper time-related information.

[0014] Furthermore, the deep fusion of the noise-mixed neural representation HNeRN in step S2 is specifically implemented as follows:

[0015] The hybrid neural representation of noise, HNeRN, is composed of spatial embedding S, temporal embedding T, and a spatiotemporal decoder STD. STD consists of three residual blocks and one convolutional layer. A feature element-wise multiplication strategy is used to deeply fuse the spatial embedding S and temporal embedding T. Subsequently, the spatiotemporal decoder STD is used to reconstruct the fused features, generating noise corresponding to the clear image at time t. .

[0016] Furthermore, the clear video Degradation generates noisy video The specific implementation method is as follows: a loop strategy based on the noise map is adopted to process the noise image at time t-1. After encoding, it is used as conditional information in the noise at time t. The generation process, and finally the noise With clean images The addition produces the generated noisy image. .

[0017] Furthermore, in step S2, the spatiotemporal discriminator analyzes the noisy video. The specific implementation method for authenticity verification is: time discriminator. With the generated noisy video sequence For input, spatial discriminator With a single noisy video frame generated As input data, the temporal embeddings T of multiple video frames are concatenated together and then fused with the first convolution of the temporal discriminator.

[0018] Furthermore, step S2 trains the loss function. The specific construction steps are as follows:

[0019] S21. First, analyze the noise image at time t. The code is cut into n 32x32 blocks. The mean and variance of each block are calculated. Then, a bidirectional consistency loss function based on noise intensity is introduced. For a time series, using The terms constrain the noise intensity of the forward and reverse loops at each time step;

[0020] S22. Train the noise generator G and the spatiotemporal discriminator network using relative average adversarial loss, with the adversarial loss function... Defined as:

[0021]

[0022] In the above formula, Indicates spatial countermeasures loss, This indicates that time counteracts loss;

[0023] S23, Stability loss during use To constrain the generated noisy image, Defined as:

[0024]

[0025] In the above formula, N is the number of pixels i in the mini-batch B. This represents the noise at the i-th pixel;

[0026] S24. Calculate the training loss function. The calculation formula is:

[0027]

[0028] in, and These represent the weights of each loss.

[0029] Furthermore, in step S21, the noise intensity bidirectional consistency loss function The calculation formula is:

[0030]

[0031] In the above formula, This indicates the operation of calculating the mean. This indicates the operation of calculating variance. This represents the i-th cut block in the forward loop. This represents the i-th cut block in the reverse loop.

[0032] Beneficial Effects: This invention employs a divide-and-conquer approach, independently exploring the synthesis path of noise from both spatial and temporal dimensions. By constructing a hybrid neural representation of noise, it deeply integrates the spatial and temporal embeddings of noise and implicitly models the complex spatiotemporal distribution of infrared noise through recurrent adversarial learning. This comprehensively and deeply characterizes the spatiotemporal properties of noise, providing new ideas and tools for noise modeling and analysis in the field of video processing. It demonstrates excellent performance in infrared spatiotemporal noise modeling and significantly improves the denoising performance of video denoising networks in the real world. In exploring spatiotemporal noise modeling at the video level, it can synthesize time-dependent video noise and simulate a modeling framework for real-world infrared spatiotemporal noise, which can alleviate the difficulty of paired video collection. Attached Figure Description

[0033] To more clearly illustrate the technical solutions of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a framework diagram of the infrared spatiotemporal noise model IRSTN of this invention;

[0035] Figure 2 This is a schematic diagram of the spatiotemporal noise generator and discriminator of the present invention;

[0036] Figure 3 This is a schematic diagram illustrating the bidirectional consistency loss of noise intensity in this invention.

[0037] Figure 4 These are sample frames of noisy-clean infrared video captured in an indoor scene by this invention;

[0038] Figure 5 These are sample frames of noisy-clean infrared video captured in an outdoor scene by this invention.

[0039] Figure 6 This is an evaluation plot of the average Kullback-Leibler divergence (AKLD) of different noise models in the indoor scene of this invention;

[0040] Figure 7 The graphs show the variation of noise variance over time generated by different modeling methods of this invention.

[0041] Figure 8 This is a comparison chart of the visual noise modeling results of this invention in indoor environments;

[0042] Figure 9 This is a comparison chart of the visual noise modeling results of this invention outdoors;

[0043] Figure 10 The figure shows the ablation experiment results of this invention on the contribution of different noise priors in the model framework from two aspects: PSNR and SSIM.

[0044] Figure 11 The figure shows the ablation experiment results of this invention, which examines the contributions of each component in IRSTN to denoising from both PSNR and SSIM perspectives.

[0045] Figure 12 This is a quantitative comparison of video denoising results for indoor and outdoor datasets in this invention;

[0046] Figure 13 These are visualizations of the denoising results of different methods of this invention on an indoor infrared dataset;

[0047] Figure 14 This is a visualization of the denoising results of different methods of the present invention on an outdoor infrared dataset. Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] The present invention addresses the physical priors of the noise involved. The specific explanation is as follows: The spatiotemporal degradation process from clean video to noisy video can be represented as:

[0050]

[0051]

[0052]

[0053]

[0054] in, The original, noise-free video signal. The signal is a noisy video signal. N represents the spatiotemporal noise sequence. , and Let represent the original image signal, the noisy image signal, and the noise at time t, respectively. For time t, the noisy image can be represented as:

[0055]

[0056] Infrared noise at time t It includes spatial noise and temporal noise. Known spatial noise can be divided into random noise and fixed-mode noise. Random noise mainly includes readout noise. shot noise Known time noise is mainly composed of time stripe noise. composition.

[0057] Readout noise When electrons undergo analog-to-digital conversion, amplification, and processing, readout noise is generated within the detector's electronic components. .

[0058] shot noise When a photon reaches the detector, shot noise is generated due to the fluctuation characteristics of photons. .

[0059] It is worth noting that readout noise and shot noise A heteroscedastic Gaussian random variable can usually be used to approximate this, with its mean equal to the original video signal. The variance is determined by the parameters of the readout noise. and shot noise parameters and the original video signal Joint decision:

[0060]

[0061] in, and These are learnable parameters.

[0062] Fixed-mode noise Due to limitations in manufacturing processes, the photoelectric responsivity of individual detector pixels on the infrared focal plane varies, which can lead to fixed-pattern noise in the image. This noise can be obtained from offset frames captured in a dark environment.

[0063] Time stripe noise During signal sampling, the analog-to-digital converter (ADC) is affected by electronic thermal noise, which in turn generates time stripe noise. This embodiment sets its appearance in the vertical direction. This embodiment uses one-dimensional Gaussian properties to address time stripe noise. Modeling:

[0064]

[0065] in, These are learnable parameters. These known spatiotemporal noises constitute the physical priors of the noise. :

[0066]

[0067] Infrared detectors may also encounter unknown noise that is difficult to parameterize during the imaging process. These known and unknown noises together constitute the infrared noise at time t. . It can be represented as:

[0068]

[0069] in, This represents unknown noise that is difficult to parameterize. Furthermore, it involves calibration noise priors. Parameterizing noise is a laborious and expensive task. This invention utilizes readily available unpaired noise and clear video data to implicitly learn noise priors through adversarial learning and neural representations. The parameters in the simulation and unknown noise It can synthesize infrared noise sequences that conform to the actual spatiotemporal distribution. Generate clear video Corresponding noise video .

[0070] like Figures 1-14 As shown, the present invention provides an infrared spatiotemporal noise generation method based on hybrid neural representation, the specific steps of which are as follows:

[0071] S1. Establish an infrared denoising dataset, which includes indoor paired noise-clear video dataset and outdoor unpaired noise-clear video dataset;

[0072] S2. Construct an infrared spatiotemporal noise model (IRSTN) based on hybrid neural representations and the model's training loss function. The model includes a spatiotemporal noise generator G and a spatiotemporal discriminator; wherein, the spatiotemporal noise generator G includes a spatial embedding generator SEG, a temporal embedding generator TEG, and a hybrid neural representation of noise HNeRN, and the spatiotemporal discriminator consists of a spatial discriminator... and time discriminant It consists of two parts;

[0073] like Figure 1As shown, the spatial embedding generator SEG first obtains the spatial embedding S by introducing a physics-based noise prior P to capture the spatial context of the noise. Simultaneously, the temporal embedding generator TEG obtains the temporal embedding T through positional encoding of the frame index to describe the temporal correlation of the noise. Subsequently, the hybrid neural representation of noise HNeRN deeply fuses the spatial embedding S and the temporal embedding T to generate a noisy image. The spatiotemporal noise generator G employs a recurrent strategy based on the noise map during training, replacing clear video with... Degradation generates noisy video By employing recurrent generative adversarial learning, IRSTN can implicitly model the complex spatiotemporal distribution characteristics of infrared noise using only unpaired data. Addressing the differences in statistical characteristics among different noise types, this scheme adopts a divide-and-conquer approach, independently exploring the noise synthesis path from both spatial and temporal dimensions. IRSTN accurately describes the spatial distribution and temporal evolution of noise through a compact and independent representation method.

[0074] like Figure 2 As shown in this embodiment, the implementation of IRSTN is further explained as follows:

[0075] Spatiotemporal embedding generation:

[0076] The Spatial Embedding Generator (SEG) consists of a 3x3 convolutional layer and three residual blocks. The SEG introduces a noisy prior P with learnable parameters and modifies it with the noiseless image signal at time t. The input matrix of the SEG is concatenated and formed by these concatenations. This input matrix then undergoes preliminary feature extraction through a convolutional layer and is mapped to a high-dimensional feature space through deep encoding of three residual blocks. This process aims to deeply explore the complex spatial interactions between noise and image signals. Furthermore, considering the correlation of noise between adjacent frames, the spatial embedding generator SEG also processes the noise signal at time t-1. After encoding, SEG ultimately generates a spatial embedding S containing contextual information. The generation process can be described as follows:

[0077]

[0078] Another challenge in spatiotemporal modeling of noise lies in how to effectively represent the time dimension to capture the dynamic characteristics of noise evolution over time. Directly using timestamps as input often fails to fully reflect temporal correlation, thus limiting the performance of deep models in spatiotemporal modeling.

[0079] Therefore, this invention constructs a Temporal Embedding Generator (TEG), comprising two parts: Frequency Position Encoding (FPE) and Multilayer Perceptron (MLP). First, FPE maps frame indices to a high-dimensional space, enabling the neural network to better fit frequently changing data. Then, the MLP uses the encoded information as input to further mine deeper time-related information. The FPE and temporal embedding generation processes are respectively represented as follows:

[0080]

[0081]

[0082] Where t represents the normalized frame index; b and l are two hyperparameters, which are set to 1.25 and 80 in this embodiment based on experience.

[0083] Hybrid neural representation of noise HNeRN:

[0084] The hybrid neural representation of noise, HNeRN, is composed of spatial embedding S, temporal embedding T, and a spatiotemporal decoder STD. It deeply integrates and collaboratively models the spatial and temporal distributions of noise. STD consists of three residual blocks and one convolutional layer. To effectively integrate and realize the spatiotemporal information of the noise, a strategy of element-wise multiplication of feature elements is adopted to deeply fuse the spatial embedding S and the temporal embedding T. This process promotes the interaction of spatiotemporal information and enhances the spatiotemporal representation capability of the features. Subsequently, the spatiotemporal decoder STD is used to reconstruct the fused features, generating noise corresponding to the clear image at time t. HNeRN can be represented as:

[0085]

[0086] Through close collaboration among these components, HNeRN achieves a unified representation and modeling of real-world noise in the spatiotemporal dimensions.

[0087] To fully extract inter-frame information, the spatiotemporal noise generator G employs a recurrent strategy based on noise maps during training. For example... Figure 2 As shown in (a), this strategy aims to construct a continuous noise sequence in the time dimension, transforming the noise image at time t-1. After encoding, it is used as conditional information in the noise at time t. The generation process, and finally the noise With clean images The addition produces the generated noisy image. This process can be represented as:

[0088]

[0089] During training, the length of the loop sequence is N, where N is set to 5 by default. The loop strategy effectively promotes the coherence of the noise over time, ensuring that the generated noisy video appears more natural and smooth. (Generated noisy video) It can be represented as:

[0090]

[0091] The spatiotemporal noise generator G integrates the spatial and temporal embeddings of noise to construct a hybrid neural representation of noise, thereby degrading a clear image into a noisy image.

[0092] like Figure 2 As shown in (b) and (c), the spatiotemporal discriminator consists of a spatial discriminator. and time discriminant It consists of two parts; specifically, the spatial discriminator. Focusing on the spatial distribution characteristics of noise to generate a single noisy video frame. As input data. Time discriminator. The focus is then placed on the temporal distribution of noise, and the generated noisy video sequence is used as the basis for this analysis. As input, the temporal discriminator and the spatial discriminator respectively determine the authenticity of a single frame image and an image sequence.

[0093] Given that implicit neural representations can provide high-quality temporal information, this solution cleverly incorporates this characteristic into the discriminator design. Specifically, the temporal embeddings T of multiple video frames are concatenated and then fused with the first convolutional layer of the temporal discriminator. Introducing the temporal embedding T into the discriminator can more sensitively capture the temporal correlation in dynamic scenes, effectively reduce temporal artifacts, and thus further improve the accuracy and robustness of the entire discrimination process.

[0094] Model training loss function The specific construction steps are as follows:

[0095] During the generation of long-term noise cycles, the accumulation of errors may lead to flickering as the time series grows. For natural videos, the noise intensity of the forward and reverse time sequences should be consistent at any given moment.

[0096] S21. Introduce a bidirectional consistency loss function for noise intensity. It aims to suppress error accumulation and ensure the continuity of long-term noise generation.

[0097] like Figure 3 As shown, firstly, the noise image at time t... The data is cut into n 32x32 blocks, and the mean and variance of each block are calculated to characterize the local noise intensity. Then, for a time series, the data is processed using... The term constrains the noise intensity of the forward and reverse loops at each time step. The calculation formula is:

[0098]

[0099] in, This indicates the operation of calculating the mean. This indicates the operation of calculating variance. This represents the i-th cut block in the forward loop. This represents the i-th crop block in the reverse loop. By constraining the consistency of noise intensity in the forward and reverse loops, temporal artifacts in the generated noisy video are effectively suppressed.

[0100] S22. To reduce the domain difference between the generated video and the target video, a relative average adversarial loss is used to train the noise generator G and the spatiotemporal discriminator network. The adversarial loss function is... Defined as:

[0101]

[0102] In the above formula, Indicates spatial countermeasures loss, This indicates that time counteracts loss;

[0103] S23. In addition, to suppress potential artifacts, stability loss is used. To constrain the generated noisy image, Defined as:

[0104]

[0105] In the formula, N is the number of pixels i in the mini-batch B. Let represent the noise of the i-th pixel. By minimizing the stabilization loss, the channel mean of the generated noise approaches zero, effectively reducing image artifacts.

[0106] S24. In summary, calculate the training loss function. The calculation formula is:

[0107]

[0108] in, and These represent the weights of each loss, which are set to 1e-1 and 1e-2 respectively in this embodiment based on experience.

[0109] S3. Input the infrared denoising dataset from S1 into the model in S2 for training to obtain a trained infrared spatiotemporal noise model based on hybrid neural representations, which will then be used to refine the clear video. Degraded to a noisy video by the spatiotemporal noise generator G And through a spatiotemporal discriminator, the noisy video is analyzed. Verify the authenticity.

[0110] This invention establishes a new infrared denoising dataset, which includes both paired indoor video and unpaired outdoor video. The video data was acquired by a First Light C-RED 2 near-infrared camera with a resolution of 640×512.

[0111] A frame-by-frame animation-like approach was employed to simulate object motion in videos, enabling the acquisition of paired noisy-clean video datasets within an indoor environment. Specifically, controllable objects (such as toys) were used, and their movement was manually manipulated to simulate the motion effects of objects in the video. For each object movement, bias calibration was first disabled, and a noisy image was captured; then, bias calibration was enabled, and M clean images were captured consecutively. The average of these clean images was used as a noise-free reference image. After a slight movement of the object, it was kept stationary to capture the next set of paired images. Finally, all single-frame images were merged chronologically to form the noisy video and its corresponding clean video. Ten different indoor scenes were captured, each containing 12 frames. Figure 4 It shows consecutive noisy frames of indoor video and their corresponding clear frames.

[0112] Considering the uncontrollable motion of objects in outdoor scenes, the traditional multi-frame averaging method for obtaining noise-free reference images is no longer applicable. Therefore, this invention adopts an alternative strategy, acquiring multiple noise-free videos by performing camera offset calibration and employing long exposure techniques. This strategy covers seven different outdoor scenes, with 200 consecutive clean, noise-free frames captured for each scene. In addition, 23 infrared noise videos with fixed-pattern noise and random noise were also captured, each scene also containing 200 consecutive frames. Figure 5 This demonstrates consecutive noisy and clear frames in outdoor video footage. It's important to note that the captured outdoor clear and noisy videos are unpaired.

[0113] To obtain the prior of fixed-pattern noise, a method was employed to capture static scenes using a camera in complete darkness, followed by multi-frame averaging to improve the signal-to-noise ratio. Specifically, a lens cap was used to completely block external light, ensuring the camera sensor was in complete darkness to eliminate the influence of ambient light. With bias calibration disabled, 500 frames of unprocessed video were continuously captured. This step aimed to capture the inherent noise characteristics of the sensor under no-light conditions. The prior of fixed-pattern noise was extracted by summing each frame and calculating the average. The multi-frame averaging method effectively reduced the impact of random noise on the final result, improving the accuracy of noise extraction.

[0114] This invention conducted spatiotemporal noise modeling and video denoising experiments on captured real-world infrared video datasets. For paired indoor datasets, this invention used eight scenes as the training set and the remaining two scenes as the test set. For unpaired outdoor datasets, clean frames from six scenes and noisy frames from 17 scenes were used as the training set. The remainder were used as the test set.

[0115] For training the IRSTN infrared spatiotemporal noise model based on hybrid neural representations, the batch size was 4 and the number of training epochs was 400. The Adam algorithm was chosen for optimization, with an initial learning rate of 0.0001, which remained constant for the first 1000 training epochs and then gradually decayed to 0 for the next 1000 training epochs. Video frames were randomly cropped to 128×128 pixels and randomly flipped horizontally or vertically to enhance the model's generalization ability. All experiments were conducted in a Python 3.8.0 environment using the PyTorch 1.12.0 framework and an NVIDIA GeForce RTX 3090 graphics card.

[0116] To verify the advancement of the proposed infrared spatiotemporal noise model IRSTN based on hybrid neural representations, it was compared with several state-of-the-art noise modeling algorithms, namely ELD, C2N, Starlight, and Cai et al. ELD is a physically-based image-level noise modeling method. C2N is a learning-based image-level noise modeling method. Starlight is a learning-based video noise modeling method. Cai et al. is a physically-based video noise modeling method. Considering the scarcity of learning-based noise modeling methods specifically for video, this invention was compared with UnsupRecycleGan, a state-of-the-art video translation method with similar tasks. To ensure fairness in the comparison, all learning-based methods incorporated noise priors. Furthermore, the denoising performance of IRSTN was evaluated and compared in detail with other advanced denoising methods such as V-BM4D.

[0117] The specific comparison results are as follows:

[0118] 1) Quantitative comparison:

[0119] The proposed infrared spatiotemporal noise model IRSTN, based on hybrid neural representations, was analyzed in depth on an indoor dataset and compared with other noise modeling methods. For quantitative evaluation, the widely adopted metric Average Kullback-Leibler Divergence (AKLD) was used to quantify the difference between real noise and the synthetic noise generated by each noise model. AKLD, as a statistical measure of probability distribution variability, indicates a higher consistency between generated noise and real noise as the value of the AKLD is lower. Figure 6 As shown, the average Kullback-Leibler divergence (AKLD) evaluation of different noise models in indoor scenes is presented. Thanks to fine-grained modeling of the spatiotemporal distribution of noise, IRSTN performs best in the AKLD evaluation metric, exhibiting the smallest AKLD value. This result demonstrates the state-of-the-art performance of IRSTN compared to other methods in noise modeling tasks.

[0120] To further explore the dynamic changes in noise characteristics, the Blind Image Quality Index (BIQI) was used to assess the noise level. For example... Figure 7 As shown, the BIQI curves of noise images generated by different modeling methods evolve over time. The noise images generated by IRSTN show a high degree of consistency with the real noise images actually captured by the infrared detector in terms of their changing trends, which strongly demonstrates that IRSTN has higher accuracy and reliability in simulating the spatiotemporal distribution characteristics of noise.

[0121] 2) Visual comparison

[0122] like Figure 8-9As shown, the modeling results of different noise modeling methods in indoor and outdoor scenes are presented. The noise generated by the low-light noise model ELD and the method proposed by Cai et al. show a significant deviation from the distribution of real noise. This deviation may stem from these methods' over-reliance on calibration processes or Gaussian distribution assumptions, failing to adequately capture the complexity of noise. C2N faces challenges in learning and simulating the spatiotemporal characteristics of infrared noise, possibly because this method primarily models random noise and lacks sufficient representation ability for fixed-pattern noise. While the Starlight algorithm incorporates factors such as fixed-pattern noise, random noise, and timeline noise, its network training is based solely on single-frame images, limiting its accurate capture of the temporal evolution characteristics of noise sequences. UnsupRecycleGAN demonstrates good performance in video conversion tasks, but its ability to model the detailed spatiotemporal distribution of noise remains insufficient. In contrast, IRSTN, by introducing noise physical priors and a spatiotemporal modeling framework based on hybrid neural representations, can effectively fit the spatiotemporal distribution of noise in infrared imaging systems, generating more realistic noisy videos. These advantages make IRSTN superior to other methods in terms of both accuracy and realism in noise modeling.

[0123] This embodiment conducts an ablation study on the contributions of different noise priors in the model framework from both PSNR and SSIM perspectives. The best results are shown in bold:

[0124] like Figure 10 As shown, an ablation study was conducted on each noise component in the infrared noise prior P. This invention calculates the AKLD between synthetic and real noise for an indoor dataset to evaluate the performance of the noise model. The results show that each component included in the noise model positively contributes to improving the AKLD value, thus demonstrating the importance of these components in infrared noise modeling.

[0125] This embodiment conducts an ablation study on the contribution of each component of IRSTN to denoising from both PSNR and SSIM perspectives. The best results are shown in bold:

[0126] This embodiment quantitatively evaluates key components in the infrared spatiotemporal noise model IRSTN based on hybrid neural representations, including the temporal embedding generator (TEG), the temporal discriminator, the bidirectional consistency loss function for noise intensity, and the recurrent strategy based on the noise graph. To verify the advantages of these components, ablation experiments were conducted on an indoor dataset.

[0127] This embodiment removes the aforementioned components one by one from the IRSTN framework and retrains the noise model while keeping other training parameters unchanged. Subsequently, the infrared noisy videos generated by these models are used as training data to train the corresponding denoising network. Figure 11 As shown, TEG, time discriminator, bidirectional consistency loss function for noise intensity, and recurrent strategy based on noise graph significantly improve the denoising performance of the denoising network.

[0128] To verify the reliability and sophistication of the proposed infrared spatiotemporal noise model IRSTN, this embodiment conducted video denoising experiments on indoor and outdoor datasets, and quantitatively compared the denoising results for the indoor and outdoor datasets. In each column, the best results are highlighted in bold:

[0129] Specifically, firstly, noisy samples synthesized using noise modeling methods are combined with corresponding real noise-free samples to form complete video data pairs. Based on this, FastDVDNet is selected as the video denoising algorithm, configured with default parameters to ensure optimal performance. For the indoor dataset, Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) are chosen as evaluation metrics for denoising performance. Considering the lack of ground truth benchmarks for outdoor datasets, Natural ImageQuality Evaluator (NIQE) is introduced to objectively evaluate the denoising performance of outdoor scenes. Experimental results are as follows: Figure 12 As shown in the figure, the proposed IRSTN model achieves the best denoising performance on both indoor and outdoor datasets.

[0130] like Figure 13-14 As shown, the denoising visualization results of different methods on indoor and outdoor infrared datasets are presented respectively. The IRSTN method proposed in this invention, due to its accuracy in spatiotemporal noise modeling, significantly improves the performance of denoising networks in practical applications compared to other state-of-the-art methods. The quantitative indicators of denoising performance and the intuitive visualization results both demonstrate the enormous application potential and value of IRSTN in the field of video denoising.

[0131] The proposed IRSTN introduces a physics-based noise prior to obtain a spatial embedding representing the spatial context of the noise. Secondly, positional encoding using frame indices is used to generate a temporal embedding describing the temporal correlation of the noise. The deep fusion of these two embeddings forms a hybrid neural representation of noise, HNeRN, which can uniformly represent the spatiotemporal characteristics of noise. To degrade clean video into realistic noisy video, a recurrent generative adversarial strategy is employed, enabling IRSTN to capture and reproduce the complex dynamic changes of noise. Furthermore, a bidirectional consistency loss function for noise intensity is used to constrain the local similarity of the noise's mean and variance in the forward and reverse loops, ensuring the temporal coherence of the generated noisy video.

[0132] This invention proposes an infrared spatiotemporal noise model, IRSTN, based on hybrid neural representations. It aims to overcome the limitations of existing noise modeling techniques in accurately depicting the complex spatiotemporal characteristics of real-world noise, synthesizing realistic noise in dynamic scenes from unpaired videos. Given the statistical differences in the spatiotemporal behavior of infrared noise, IRSTN employs a divide-and-conquer approach, independently exploring the noise synthesis path from both spatial and temporal dimensions. IRSTN constructs a hybrid neural representation of noise, deeply integrating spatial and temporal embeddings, and implicitly models the complex spatiotemporal distribution of infrared noise through recurrent adversarial learning. Extensive experiments demonstrate that IRSTN performs excellently in infrared spatiotemporal noise modeling, while also significantly improving the denoising performance of video denoising networks in the real world. In exploring spatiotemporal noise modeling at the video level, it can synthesize time-dependent video noise. The proposed hybrid neural representation of noise aims to comprehensively and deeply characterize the spatiotemporal properties of noise, providing new ideas and tools for noise modeling and analysis in the field of video processing. The modeling framework for simulating real-world infrared spatiotemporal noise can alleviate the difficulties of paired video collection.

[0133] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0134] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A method for generating infrared spatiotemporal noise based on hybrid neural representation, characterized in that, Includes the following steps: S1. Establish an infrared denoising dataset, which includes indoor paired noise-clear video dataset and outdoor unpaired noise-clear video dataset; S2. Construct an infrared spatiotemporal noise model based on hybrid neural representations and the model's training loss function. The model includes a spatiotemporal noise generator G and a spatiotemporal discriminator; The spatiotemporal noise generator G includes a spatial embedding generator SEG, a temporal embedding generator TEG, and a hybrid neural representation of noise HNeRN. The spatiotemporal discriminator consists of a spatial discriminator. and time discriminant It consists of two parts; First, the spatial embedding generator SEG obtains the spatial embedding S by introducing a noise prior P with learnable parameters. At the same time, the temporal embedding generator TEG obtains the temporal embedding T by positional encoding of the frame index. Subsequently, the hybrid neural representation of noise, HNeRN, is deeply fused with spatial embedding S and temporal embedding T to generate a noisy image. The spatiotemporal noise generator G employs a recurrent strategy based on noise maps during training, combining clear video... Degradation generates noisy video ; S3. Input the infrared denoising dataset from S1 into the model in S2 for training to obtain a trained infrared spatiotemporal noise model based on hybrid neural representations, which will then be used to refine the clear video. Degraded to a noisy video by the spatiotemporal noise generator G .

2. The infrared spatiotemporal noise generation method according to claim 1, characterized in that, The specific implementation of the spatial embedding generator SEG in step S2 to generate the spatial embedding S is as follows: the spatial embedding generator SEG consists of a 3x3 convolutional layer and three residual blocks. The spatial embedding generator SEG introduces a noise prior P with learnable parameters and modifies it with the noiseless image signal at time t. The input matrix of the SEG is concatenated and then subjected to preliminary feature extraction through a 3x3 convolutional layer. It is then mapped to a high-dimensional feature space through deep encoding of three residual blocks. Furthermore, the spatial embedding generator (SEG) also processes the noise signal at time t-1. Encoding was performed, and SEG ultimately generates a spatial embedding S containing contextual information; The specific implementation of the temporal embedding generator (TEG) to generate temporal embeddings (T) is as follows: The temporal embedding generator (TEG) consists of two parts: frequency position coding (FPE) and multilayer perceptron (MLP). First, the FPE maps the frame index to a high-dimensional space. Then, the MLP takes the encoded information as input to further mine deeper time-related information.

3. The infrared spatiotemporal noise generation method according to claim 1, characterized in that: The specific implementation method of deep fusion of the noise hybrid neural representation HNeRN in step S2 is as follows: The hybrid neural representation of noise, HNeRN, is composed of spatial embedding S, temporal embedding T, and a spatiotemporal decoder STD. STD consists of three residual blocks and one convolutional layer. A feature element-wise multiplication strategy is used to deeply fuse the spatial embedding S and temporal embedding T. Subsequently, the spatiotemporal decoder STD is used to reconstruct the fused features, generating noise corresponding to the clear image at time t. .

4. The infrared spatiotemporal noise generation method according to claim 1, characterized in that: The clear video Degradation generates noisy video The specific implementation method is as follows: A loop strategy based on the noise map is adopted to process the noise image at time t-1. After encoding, it is used as conditional information in the noise at time t. The generation process, and finally the noise With clean images The addition produces the generated noisy image. .

5. The infrared spatiotemporal noise generation method according to claim 1, characterized in that: The spatiotemporal discriminator in step S2 analyzes the noisy video. Authenticity verification is performed, specifically using a time discriminator. With the generated noisy video sequence For input, spatial discriminator With a single noisy video frame generated As input data, the temporal embeddings T of multiple video frames are concatenated together and then fused with the first convolution of the temporal discriminator.

6. The infrared spatiotemporal noise generation method according to claim 1, characterized in that: Step S2 trains the loss function. The specific construction steps are as follows: S21. First, analyze the noise image at time t. The code is cut into n 32x32 blocks. The mean and variance of each block are calculated. Then, a bidirectional consistency loss function based on noise intensity is introduced. For a time series, using The terms constrain the noise intensity of the forward and reverse loops at each time step; S22. Train the noise generator G and the spatiotemporal discriminator network using relative average adversarial loss, with the adversarial loss function... Defined as: In the above formula, Indicates spatial countermeasures loss, This indicates that time counteracts loss; S23, Stability loss during use To constrain the generated noisy image, Defined as: In the above formula, N is the number of pixels i in the mini-batch B. This represents the noise at the i-th pixel; S24. Calculate the training loss function. The calculation formula is: In the above formula, and These represent the weights of each loss.

7. The infrared spatiotemporal noise generation method according to claim 6, characterized in that: The noise intensity bidirectional consistency loss function in step S21 The calculation formula is: In the above formula, This indicates the operation of calculating the mean. This indicates the operation of calculating variance. This represents the i-th cut block in the forward loop. This represents the i-th cut block in the reverse loop.

Images