Unsupervised video denoising method and device based on continuous spatiotemporal implicit neural field

Abstract

This disclosure relates to an unsupervised video denoising method and apparatus based on a continuous spatiotemporal implicit neural field. The method includes: receiving a noisy video sequence; selecting multiple input frames centered on a target time according to a preset time window; performing forward and backward spatiotemporal feature encoding on the multiple input frames to obtain basic spatiotemporal features; mapping the basic spatiotemporal features to a continuous spatial feature representation under blind spot constraints; obtaining the temporal embedding features of the target time; determining the graph aggregation features of the target time based on graph connectivity weights and neighboring frame features; reconstructing the denoising result at the target pixel position based on the continuous spatial features, temporal embedding features, and graph aggregation features; and obtaining all denoising results corresponding to the noisy video sequence to output a denoised video. By employing the above technical solution, continuous spatiotemporal representation can be directly established and video denoising performed under unsupervised conditions, improving the efficiency and effectiveness of video denoising.

Description

Technical Field

[0001] This disclosure relates to the interdisciplinary fields of computer vision, video image processing and artificial intelligence, and in particular to an unsupervised video denoising method and apparatus based on a continuous spatiotemporal implicit neural field. Background Technology

[0002] Existing video denoising techniques, such as traditional methods including block matching, spatiotemporal filtering, nonlocal means, and transform domain sparse representation, rely on manually designed priors and have limited adaptability to complex non-Gaussian noise, fast motion, and spatiotemporal mismatch in real-world scenes. Supervised deep learning methods typically train on pairs of clean and noisy videos, modeling spatiotemporal information through convolutional neural networks, recurrent networks, optical flow registration, or Transformers (encoder-decoder neural network architectures based on self-attention mechanisms). While these methods perform well on synthetic data, they heavily rely on large-scale labeled data, and high-quality paired data is difficult to collect in real-world scenes, significantly limiting their generalization ability. Unsupervised or self-supervised video denoising methods typically construct training targets based on noise independence, blind spot constraints, or heavy contamination strategies. For example, they avoid identity mapping by masking center pixels or perform joint restoration by aligning adjacent frames through optical flow estimation, reducing reliance on real clean labels. However, most existing methods are still based on discrete pixel grids and discrete frame sequences.

[0003] Therefore, existing blind spot networks typically perform reconstruction within a fixed convolutional receptive field. After the central pixel is masked, the information available to the model is still limited to the local neighborhood, making it difficult to fully capture long-distance structural correlations and global texture priors. Existing methods rely on optical flow or explicit motion compensation to establish inter-frame correspondences, but under conditions such as high noise, fast motion, occlusion, and low illumination, optical flow itself is easily contaminated by noise, leading to misregistration, motion artifacts, and error accumulation. Actual videos are physically continuous spatiotemporal signals, and discrete frame-level modeling cannot naturally express the continuity of pixel evolution over time, easily resulting in problems such as temporal flicker, unstable details, and insufficient consistency of long sequences. Furthermore, existing self-supervised loss designs often only constrain the local reconstruction of masked pixels, lacking a unified continuous spatiotemporal representation framework, making it difficult to simultaneously consider spatial detail recovery, temporal consistency, and long-range spatiotemporal correlations under unsupervised conditions. Summary of the Invention

[0004] To address the problems of limited spatial receptive field, discretized temporal modeling, strong dependence on explicit optical flow registration, and insufficient spatiotemporal consistency of long sequences in existing unsupervised video denoising methods, this disclosure provides an unsupervised video denoising method and apparatus based on continuous spatiotemporal implicit neural fields. This method overcomes the limitations of fixed convolutional receptive fields and strong dependence on explicit optical flow, and can directly establish continuous spatiotemporal representations and perform video denoising under unsupervised conditions. To achieve the above objectives, this disclosure adopts the following technical solution: This disclosure provides an unsupervised video denoising method based on a continuous spatiotemporal implicit neural field. The method includes: receiving a noisy video sequence and selecting multiple input video segments centered on a target time from the noisy video sequence according to a preset time window; wherein any video frame in the noisy video sequence is taken as the video frame to be recovered, and the target time is the center time corresponding to the video frame to be recovered; extracting forward temporal features and reverse temporal features from the multiple input video segments respectively, and fusing the forward temporal features and the reverse temporal features to obtain basic spatiotemporal features; obtaining the target... The coordinate encoding result of the pixel position is obtained, and under blind spot constraints, continuous spatial features of the target pixel position are generated based on the coordinate encoding result and the basic spatiotemporal features; the temporal embedding feature of the target time is obtained, and the graph connection weights of the video frames at the target time and adjacent times are determined, and the graph aggregation feature of the target time is determined based on the graph connection weights and the features of neighboring frames; reconstruction is performed based on the continuous spatial features, the temporal embedding feature and the graph aggregation feature to obtain the denoising result of the target frame to be recovered, and all denoising results corresponding to the noisy video sequence are obtained to output the denoised video.

[0005] This disclosure also provides an unsupervised video denoising device based on a continuous spatiotemporal implicit neural field. The device includes: an input video construction module, used to receive a noisy video sequence and select multiple input video segments centered on a target time from the noisy video sequence according to a preset time window; wherein any video frame in the noisy video sequence is used as the video frame to be recovered, and the target time is the center time corresponding to the video frame to be recovered; a bidirectional feature extraction module, used to extract forward temporal features and reverse temporal features from the multiple input video segments respectively, and fuse the forward temporal features and the reverse temporal features to obtain basic spatiotemporal features; and a coordinate encoding module, used to obtain the coordinate encoding result of the target pixel position. The blind spot implicit neural field module is used to generate continuous spatial features of the target pixel position based on the coordinate encoding result and the basic spatiotemporal features under blind spot constraints; the implicit temporal embedding module is used to obtain the temporal embedding features of the target time; the temporal-aware spatial graph module is used to determine the graph connection weights of the video frames at the target time and adjacent time points, and to determine the graph aggregation features of the target time based on the graph connection weights and the features of neighboring frames; the denoising and reconstruction module is used to reconstruct based on the continuous spatial features, the temporal embedding features and the graph aggregation features to obtain the denoising result of the target time at the target pixel position; and the output module is used to obtain all denoising results corresponding to the noisy video sequence to output the denoised video.

[0006] This disclosure also provides an electronic device, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the unsupervised video denoising method based on a continuous spatiotemporal implicit neural field as provided in this disclosure.

[0007] This disclosure also provides a computer-readable storage medium storing a computer program for executing the unsupervised video denoising method based on a continuous spatiotemporal implicit neural field as provided in this disclosure.

[0008] This disclosure also provides a computer program product, including a computer program, wherein the computer program is executed by a processor as described in the embodiments of this application for an unsupervised video denoising method based on a continuous spatiotemporal implicit neural field.

[0009] Compared with the prior art, the technical solution provided in this disclosure has the following advantages: The unsupervised video denoising scheme based on continuous spatiotemporal implicit neural fields provided in this disclosure receives a noisy video sequence and selects multiple input video segments centered on a target time from the noisy video sequence according to a preset time window; wherein, any video frame in the noisy video sequence is taken as the video frame to be recovered, and the target time is the center time corresponding to the video frame to be recovered; forward temporal features and reverse temporal features are extracted from the multiple input video segments respectively, and the forward temporal features and reverse temporal features are fused to obtain basic spatiotemporal features; the coordinate encoding result of the target pixel position is obtained, and under blind spot constraints, continuous spatial features of the target pixel position are generated based on the coordinate encoding result and basic spatiotemporal features; the temporal embedding feature of the target time is obtained, and the graph connection weights of the video frames at the target time and adjacent times are determined, and the graph aggregation feature of the target time is determined based on the graph connection weights and the features of neighboring frames; reconstruction is performed based on continuous spatial features, temporal embedding features and graph aggregation features to obtain the denoising result of the target time at the target pixel position, and all denoising results corresponding to the noisy video sequence are obtained to output the denoised video. By adopting the above technical solution, it is possible to directly establish continuous spatiotemporal representation and perform video denoising under unsupervised conditions, thereby improving the efficiency and effectiveness of video denoising. Attached Figure Description

[0010] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0011] Figure 1 A flowchart illustrating an unsupervised video denoising method based on a continuous spatiotemporal implicit neural field, provided for embodiments of this disclosure; Figure 2 An example diagram of implicit neural field module processing provided in this disclosure embodiment; Figure 3 An example diagram of a time-aware spatial graph module processing provided in this disclosure embodiment; Figure 4 An example diagram illustrating an unsupervised video denoising method based on a continuous spatiotemporal implicit neural field, provided in an embodiment of this disclosure; Figure 5 An example diagram of an unsupervised video denoising system based on a continuous spatiotemporal implicit neural field provided for embodiments of this disclosure; Figure 6 This is a schematic diagram of the structure of an unsupervised video denoising device based on a continuous spatiotemporal implicit neural field, provided in an embodiment of this disclosure. Detailed Implementation

[0012] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0013] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0014] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0015] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0016] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0017] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0018] This disclosure provides an unsupervised video denoising method based on a continuous spatiotemporal implicit neural field. Instead of simply understanding video as a stack of a finite number of discrete frames, it represents the video to be restored as a type of coordinate-driven continuous spatiotemporal signal mapping relationship. Under blind-spot self-supervised constraints, it completes noise suppression and detail restoration of the target frame or target video sequence. Unlike traditional schemes that rely solely on convolutional receptive fields, explicit optical flow registration, or discrete temporal recursion, this disclosure achieves unsupervised video denoising through a process of "continuous representation - implicit modeling - graph structure alignment - unsupervised restoration." Specifically, Figure 1This is a flowchart illustrating an unsupervised video denoising method based on a continuous spatiotemporal implicit neural field, provided as an embodiment of this disclosure. This method can be implemented in software and / or hardware, and is generally integrated into electronic devices. Figure 1 As shown, the method includes: Step 110: Receive the noisy video sequence and select a multi-frame input video segment centered on the target time from the noisy video sequence according to a preset time window; wherein, any video frame in the noisy video sequence is taken as the video frame to be recovered, and the target time is the center time corresponding to the video frame to be recovered.

[0019] In this embodiment of the disclosure, the noisy video sequence can be a natural video, a low-light video, a microscopic video, an industrial inspection video, a vehicle monitoring video, or other video sequence recovery tasks with complex noise interference. The settings can be selected according to the actual application needs. For example, the noisy video sequence received can be an RGB (Red, Green, Blue) natural video with additive Gaussian noise. Alternatively, the noisy video sequence received can be an original video, a low-light video, or a microscopic video with noise such as Poisson noise, Gaussian noise, heteroscedastic noise, stripe noise, readout noise, or a combination thereof.

[0020] In this embodiment of the disclosure, the preset time window can be set according to the actual application needs. For example, any video frame in the noisy video sequence is taken as the video frame to be recovered, and the center time corresponding to the video frame to be recovered is obtained as the target time. The time window range centered on the target time is determined according to the preset time window radius as the preset time window.

[0021] Specifically, if the preset time window radius is K, then the time window will revolve around the target time. Select 2K+1 frames to form a multi-frame input video segment, that is ;in, This indicates that a video clip is being input. Indicates time Noisy video frames; Indicates the radius of the preset time window; This indicates the center time corresponding to the target frame to be recovered; thus, by aggregating neighboring frame information through a local time window, a basic input is provided for subsequent spatiotemporal joint modeling.

[0022] Step 120: Extract forward temporal features and reverse temporal features from multiple input video segments, and fuse the forward and reverse temporal features to obtain basic spatiotemporal features.

[0023] In this embodiment of the disclosure, multiple frames of input video clips are input into a bidirectional feature extraction network to extract forward temporal features and reverse temporal features respectively, and then fused to obtain basic spatiotemporal features. For example, ;in, Represents the basic spatiotemporal characteristics; The bidirectional feature extraction module can be implemented using a convolutional encoder, a recurrent structure, a hybrid structure of convolution and attention, or an equivalent variant of such a structure. This allows for the construction of an intermediate feature base that can be used for subsequent continuous spatial and temporal modeling, enabling the learning of continuous and unified spatiotemporal representations from noisy video sequences under unsupervised conditions.

[0024] Step 130: Obtain the coordinate encoding result of the target pixel position, and under the blind spot constraint, generate continuous spatial features of the target pixel position based on the coordinate encoding result and basic spatiotemporal features.

[0025] In this embodiment of the disclosure, the target pixel position is determined, and the coordinate encoding result of the target pixel position is obtained. Specifically, the pixel coordinates of the target pixel position are obtained, the pixel coordinates are normalized to obtain the normalized spatial coordinates of the target pixel position, and the normalized spatial coordinates of the target pixel position are encoded based on a preset coordinate encoding function to obtain the coordinate encoding result of the target pixel position.

[0026] For example, the target pixel position is represented by normalized coordinates, and a spatial coordinate description vector is generated by a coordinate encoding network; ;in, Normalized spatial coordinates representing the position of the target pixel; Represents the horizontal coordinate of a pixel; Represents the vertical coordinate of a pixel; Indicates frame width; This represents the frame height; thus, discrete pixel positions are converted into continuous coordinate inputs, providing a unified parameterized representation for constructing a continuous spatial field.

[0027] In some embodiments, the coordinate encoding module may further perform frequency-enhanced encoding on the normalized spatial coordinates to improve the high-frequency texture representation capability, i.e. ;in, This represents the coordinate encoding result; The coordinate encoding function can be represented by sine-cosine frequency mapping, periodic activation mapping, or other equivalent high-frequency position encoding methods; thereby enhancing the model's ability to express edges, textures, and fine-grained structures.

[0028] Furthermore, under the blind spot constraint, continuous spatial features of the target pixel position are generated based on the coordinate encoding results and basic spatiotemporal features. Specifically, the target pixel position is masked using one of the following methods: regular center masking, random hole masking, periodic masking, or multi-scale masking. These masking techniques are then processed based on the coordinate encoding results and basic spatiotemporal features to obtain the continuous spatial features of the target pixel position. Thus, while preserving the blind spot self-supervised constraint, the limitations of the convolutional local receptive field are overcome, achieving stronger global texture recovery capabilities.

[0029] Specifically, under blind spot constraints, the blind spot implicit neural field network generates continuous spatial features of the target pixel position based on coordinate encoding results and basic spatiotemporal features. ;in, Continuous spatial features representing the location of target pixels; The implicit neural field module represents the blind spot. The implicit mapping network of the implicit neural field can adopt SIREN network, multilayer perceptron, Fourier feature network, hash-coded implicit network or a combination thereof, which can approximate continuous video functions with arbitrary precision and provide excellent frequency modeling capabilities. This represents a blind spot constraint mask, used to suppress the direct leakage of information from the target center pixel; thus, under the premise of satisfying the self-supervised blind spot constraint, more comprehensive contextual texture information is recovered by utilizing continuous spatial mapping.

[0030] For example, such as Figure 2 As shown, the target pixel coordinates are input, undergo a normalized coordinate transformation, and the output normalized coordinates are then fed into spatial position encoding. Simultaneously, a blind spot mask constraint masks the target pixel. The spatial implicit mapping receives the coordinate encoding result and basic spatiotemporal features, and under feature modulation, completes a continuous mapping from coordinates to features. Pixel prediction generates a feature representation or pixel estimate of the target position, ultimately outputting continuous spatial features. The recovery of the target pixel position does not simply rely on a fixed local convolutional response, but rather on a coordinate-driven continuous mapping process with blind spot constraints.

[0031] This solves the problems of limited receptive field, limited local texture recovery ability, and insufficient modeling of high-frequency details in traditional blind spot networks. By constructing a blind spot implicit neural field module, the network can directly learn the mapping from spatial coordinates to pixel representation while avoiding the leakage of central pixel information, thereby enhancing global perception and fine-grained reconstruction capabilities. Among them, the blind spot implicit neural field module transforms the "recovery of the masked position" from discrete convolutional neighborhood prediction to a continuous function mapping from "coordinates to pixel representation". Compared with existing rotating blind spots, scrambled blind spots, or bidirectional blind spot convolution, the embodiments of this disclosure explicitly introduce spatial coordinates and implicit neural representations, enabling the network to complete texture inference under global feature constraints, which is especially suitable for the recovery of repetitive textures, slender structures, and weak texture regions.

[0032] Step 140: Obtain the temporal embedding features of the target time, determine the graph connection weights of the video frames at the target time and adjacent times, and determine the graph aggregation features of the target time based on the graph connection weights and the features of neighboring frames.

[0033] In this embodiment of the disclosure, the time position (such as the target time) is continuously encoded by an implicit time embedding module to establish the time representation between the target frame and neighboring frames; ;in, Indicates time Temporal embedding vector; This refers to the implicit temporal embedding module, which elevates the temporal position from a discrete frame index to a continuously interpolable functional representation, thereby improving the modeling capabilities for fast motion, non-uniform sampling, and long-term dependencies. Specifically, the implicit temporal embedding module treats time as a continuous variable rather than a simple frame number. By using a preset periodic activation function, it can obtain a more sensitive representation to high-frequency temporal changes, enabling the model to maintain more stable temporal consistency during fast motion, non-rigid body changes, and long sequence propagation. Compared with schemes that only use recurrent neural models or discrete frame stacking for modeling, it can naturally express continuous trajectories.

[0034] In some embodiments, time embedding features are obtained by calculating the target time, time mapping weight parameters, and preset time mapping bias parameters based on a preset periodic activation function. Specifically, the implicit time embedding module uses periodic activation mapping to generate time position representations. ;in, This indicates a preset periodic activation function; This represents the time-mapping weight parameters; This represents the time mapping bias parameter, thereby improving the model's ability to express periodic changes, velocity changes, and nonlinear motion trajectories. It addresses the problems of resolution dependence, severe temporal discretization, and insufficient consistency in long sequences caused by existing models that use pixel grids and fixed frame indices as basic modeling units. By using normalized spatial coordinates and normalized temporal coordinates as unified inputs, it adaptively describes the continuous changes of video content in the spatial and temporal domains.

[0035] Furthermore, to determine the graph connection weights of video frames at the target time and adjacent time points, in some embodiments, the current node features at the target time are obtained, and a query vector is generated based on the current node features. The candidate node features at adjacent time points are obtained, and a key vector is generated based on the candidate node features. The time interval between the target time and adjacent time points is obtained. The query vector, key vector, preset feature dimension, preset time interval adjustment coefficient, and time interval are calculated based on a preset connection weight formula to obtain the graph connection weights.

[0036] Specifically, through the time-aware spatial graph module, graph connection weights are established based on inter-frame feature similarity and time intervals, and cross-frame features are aggregated and enhanced. ;in, Indicates the first The node (time) for the first The graph connection weights of video frames at each node (time point); This represents the query vector generated from the features of the current node; This represents the key vector generated from the features of the candidate nodes; Indicates the feature dimension; Indicates the time interval adjustment coefficient; Represents a node With nodes The time interval between corresponding frames is considered. Thus, by simultaneously considering content similarity and temporal relative position, a more stable cross-frame association is achieved. This enables continuous temporal modeling and spatial alignment of cross-frame features without relying heavily on noise-sensitive explicit optical flow estimation, thereby improving recovery stability in fast-moving and complex dynamic scenes. The temporal-aware spatial graph module can adopt non-overlapping windows, overlapping windows, or pyramidal multi-scale window structures to construct cross-frame association graphs in local windows. Potential corresponding pixels or features in adjacent frames are used as graph nodes, and association weights are calculated through graph attention to achieve fine-grained cross-frame alignment. Compared with traditional methods that rely on pixel-level optical flow, this method is more robust to noise and can effectively reduce streaking, ghosting, and edge tearing caused by mismatches.

[0037] Furthermore, the graph aggregation features at the target time are determined based on graph connectivity weights and neighborhood frame features. Specifically, the neighborhood frame features are weighted and summed according to the graph connectivity weights to obtain the graph aggregation features. The time-aware spatial graph module outputs the enhanced alignment features, i.e. ;in, Represents a node Aggregation characteristics; Indicates the first The value vectors of each neighboring node; thus, through a learnable weight aggregation method, the effective structural information in the neighboring frames is introduced into the current target frame recovery process.

[0038] For example, such as Figure 3As shown, the current frame feature nodes and neighboring frame feature nodes are fed into query vector generation, key vector generation, and value vector generation, respectively; time interval encoding outputs time offset information; graph connection weight calculation determines the graph connection weights between nodes based on feature similarity and time interval information; cross-frame feature aggregation weights are applied to neighboring frame features based on graph connection weights, ultimately outputting aligned and enhanced features, i.e., graph aggregated features. This achieves effective cross-frame alignment without relying on explicit optical flow, utilizing graph structure weights to selectively propagate candidate features at different times.

[0039] This solves the problems of unstable time flow and significant accumulation of registration errors under strong noise and complex motion. By continuously encoding time position through the implicit temporal embedding module and establishing the correlation weight between cross-frame features through the temporal-aware spatial graph module, more stable spatiotemporal alignment and information propagation are achieved.

[0040] Step 150: Reconstruct the video based on continuous spatial features, temporal embedding features and graph aggregation features to obtain the denoising result at the target pixel position at the target time, and obtain all denoising results corresponding to all preset time windows to output the denoised video.

[0041] In this embodiment, continuous spatial features, temporal embedding features, and graph aggregation features are fed into the reconstruction module, and the denoising result at the target time is output. ;in, Indicates the target time In coordinates The denoising results at the location; This indicates the noise reduction and reconstruction module; It represents the graph aggregation features associated with the pixel location; thereby, it achieves unified fusion reconstruction of spatial continuous representation, temporal continuous representation and cross-frame associated representation.

[0042] Therefore, all denoising results corresponding to the noisy video sequence can be obtained to output the denoised video, that is, the final denoised video result of the target video frame or multiple consecutive video frames can be output, thereby simultaneously improving reconstruction quality, visual fidelity and temporal consistency in real noise, heteroscedastic noise, low-light noise and long video sequences.

[0043] In some embodiments, the prediction results at the target pixel position at each preset time are obtained. The blind spot loss value, the temporal consistency loss value, and the regularization loss value are determined based on the preset blind spot reconstruction loss function, the preset temporal consistency loss function, the preset parameter regularization loss function, and the prediction results. The total loss value is determined based on the blind spot loss value, the temporal consistency loss value, the regularization loss value, and the weight coefficients of each loss term. All network parameters are updated based on the total loss value and the preset loss threshold. Here, all network parameters refer to the specific parameters of the encoder, neural network, etc. of the aforementioned modules.

[0044] Specifically, the network parameters of each module are jointly optimized based on blind spot loss, temporal consistency loss, and regularization loss.

[0045] In this embodiment of the disclosure, clean-labeled videos are not required during the training phase. The following unsupervised objective function can be used for optimization, with the default blind spot reconstruction loss function being: ;in, This represents the blind spot reconstruction loss value; This represents the set of pixels selected by the blind spot constraint. The error metric function can be an absolute value function, Charbonnier loss function, Huber loss function, or other equivalent robust loss functions. Indicates the reconstructed value; This represents the noisy input value; therefore, pixel reconstruction relationships are learned using only undisclosed contextual information. The preset temporal consistency loss function is... ;in, This indicates a loss of time consistency. Represents the neighborhood set of the target time; This represents the alignment operator obtained based on graph alignment or implicit temporal alignment; express Norm; thus, it constrains the consistency of recovery results from adjacent time positions, suppressing flicker artifacts and temporal drift. The preset parameter regularization loss function is... ;in, This represents the parameter regularization term; Represents the set of model parameters; This represents the squared L2 norm; thus, it suppresses excessive fluctuations in model parameters, improving training stability and generalization ability. The overall optimization objective is... ;in, Indicates the total loss; , , These represent the weight coefficients of each loss term; thus, through multi-objective joint optimization, a balance is achieved between accurate single-frame pixel recovery and cross-frame spatiotemporal stability.

[0046] This solves the problem of lacking clean labeled videos. By adopting a combined training mechanism of blind spot reconstruction loss, temporal consistency constraint and parameter regularization, the model can complete parameter optimization under the condition of only inputting noisy videos and output denoised video results during the inference stage.

[0047] The method in this disclosure can be used for inference after offline training of a single model, or for test-time self-supervised optimization for a single test video.

[0048] As an example, such as Figure 4 As shown, a sliding window is input to a noisy video sequence, resulting in multiple input video frames. These frames undergo preprocessing such as alignment. Features are extracted in chronological order using a forward encoder and a backward encoder to obtain basic spatiotemporal features. Coordinate modeling and mask constraints are performed using a blind spot implicit neural field to obtain continuous spatial features. Continuous temporal encoding, i.e., temporal embedding features, is obtained through implicit temporal embedding. Finally, a temporally aware spatial module combines continuous spatial features and temporal embedding features to achieve cross-frame graph association and feature fusion through temporal and spatial alignment. The resulting graph aggregation features are then reconstructed using a reconstruction head to obtain the denoising result.

[0049] Specifically, the input noisy video sequence is sampled through a sliding time window and then subjected to bidirectional feature extraction. The output of the bidirectional feature extraction is fed into the blind spot implicit neural field and coordinate encoding to jointly construct a spatial continuous representation, and into the temporal-aware spatial map for cross-frame association enhancement. Implicit temporal embedding provides temporal representation for the target time and neighboring time. The outputs of the blind spot implicit neural field unit, implicit temporal embedding, and temporal-aware spatial map are jointly input into the denoising reconstruction to output denoised video frames. During the training phase, the overall parameters can also be updated by unsupervised loss optimization.

[0050] Specifically, during the training phase, target windows can be extracted from noisy video sequences, blind spot masks can be constructed, and input into the network; the bidirectional encoder first acquires basic spatiotemporal features; then the blind spot implicit neural field module establishes a continuous spatial representation through coordinate-driven methods, the implicit temporal embedding module performs continuous high-frequency embedding of time, and the temporal-aware spatial graph module performs fine-grained alignment and aggregation across frames; the decoder outputs predicted values ​​at the masked positions and updates parameters through unsupervised loss; during the inference phase, after removing random training perturbations, the aforementioned process is executed window by window on the target video to obtain denoising results; for long videos, an overlapping sliding window and result fusion mechanism can be adopted to balance computational overhead and temporal stability.

[0051] For example, the unsupervised video denoising method based on continuous spatiotemporal implicit neural fields in this disclosure can be applied to... Figure 5The unsupervised video denoising system based on continuous spatiotemporal implicit neural fields, as shown, acquires multiple input video segments from a noisy video sequence via a sliding window. Then, it performs bidirectional encoding and fusion using a forward encoder and a backward encoder to obtain basic spatiotemporal features. Based on the blind spot implicit neural field, the receptive field is expanded by combining the coordinate encoding results of the target pixel positions with the basic spatiotemporal features to generate coordinate features, resulting in continuous spatial features. Implicit temporal embedding establishes continuous temporal representations, yielding temporal embedding features. A temporally aware spatial graph module performs frame alignment and feature aggregation based on the continuous spatial features and temporal embedding features, obtaining graph aggregation features. Finally, a reconstruction head reconstructs the graph aggregation features, generating a denoised center frame and outputting the video, resulting in a spatiotemporally consistent denoising result. This system can be deployed on servers, workstations, vehicle terminals, microscopic imaging workstations, mobile terminals, or embedded chip platforms; it supports offline batch processing, real-time streaming processing, and adaptive optimization during testing.

[0052] Specifically, the bidirectional encoder can be replaced with a Transformer (a deep learning model architecture based on self-attention mechanism) encoder, a state-space model encoder, a convolutional, attention-based hybrid encoder, or other spatiotemporal feature extraction backbones; in addition to using two-dimensional Cartesian coordinates, additional conditions such as normalized polar coordinates, scale coordinates, camera parameter coordinates, and exposure parameter coordinates can be introduced to adapt to multi-view or multi-exposure videos; besides the SIREN network, Fourier feature mapping, learnable bandgap coding, neural frequent differential equations, piecewise spline functions, or other continuous-time parameterization methods can also be used; the window graph can be... Attention can be replaced by multi-scale graph convolution, sparse graph matching, deformable attention, or a hybrid alignment scheme combining explicit optical flow; in addition to blind spot reconstruction loss, frequency domain consistency loss, edge preservation loss, temporal smoothing loss, contrast learning loss, noise distribution constraint loss, or perceptual quality evaluation loss can also be introduced; in addition to video denoising, it can also be used for tasks such as video super-resolution, video deblurring, video de-raining and snow removal, low-light video enhancement, and spatiotemporal reconstruction of scientific imaging; in addition to outputting denoised videos, it can also simultaneously output intermediate spatiotemporal feature maps, confidence maps, noise estimation maps, temporal consistency evaluation indicators, or continuous implicit field parameters for subsequent analysis.

[0053] Therefore, in this embodiment, the input video segment is first sampled using a sliding time window, and then basic spatiotemporal features are obtained by using a bidirectional feature extraction backbone. Subsequently, a continuous spatial representation is established through a blind spot implicit neural field module, a continuous temporal representation is established through an implicit temporal embedding module, and cross-frame association enhancement is completed through a time-aware spatial graph module. Finally, the denoising result of the target frame is output by the reconstruction head, and end-to-end optimization is completed using an unsupervised loss function during the training phase. This enables the direct establishment of continuous spatiotemporal representation and video denoising under unsupervised conditions, thereby improving the efficiency and effectiveness of video denoising.

[0054] It is understood that the embodiments of this disclosure can model the video as a function mapping F(x,y,t)→c on a continuous coordinate domain, where (x,y,t) represents the spatial-temporal coordinates and c represents the predicted color or intensity value at that coordinate. For integer time points, the model outputs the denoising result of the corresponding video frame. For non-integer time points, the model can naturally support continuous time interpolation and trajectory consistency modeling.

[0055] Specifically, a noisy video sequence V={It}t=1...T of length T is obtained, where each frame It∈R^(H×W×C). A sliding time window can be used to select the video frame corresponding to the target time and its neighboring frames before and after as network input to enhance the local temporal context. Blind spot masking and bidirectional coding are used. A blind spot mask M is applied to the target pixel position so that the original pixel value of the target pixel position does not directly participate in the recovery of that position, preventing the network from degenerating into an identity mapping. The masked multi-frame input is sent to a bidirectional spatiotemporal feature encoder to obtain forward propagation features, backward propagation features, and fused features. The encoder can be composed of mask convolution, dilated convolution, linear layers, and residual structures. A blind spot implicit neural field is constructed. For any pixel position to be recovered, its spatial coordinates and corresponding local or global features are input into the implicit mapping network to directly predict the pixel representation at that coordinate. Preferably, the coordinates are first Fourier position encoded, and then concatenated with the encoder features and sent to a multilayer perceptron to break through the limitation of the fixed receptive field of convolution and realize cross-regional texture modeling.

[0056] Furthermore, an implicit temporal embedding is constructed, mapping normalized timestamps to a continuous temporal embedding space to describe the continuous evolution of pixels or features over time. Preferably, an implicit neural network constructed using a periodic activation function, such as the SIREN structure, is employed, giving the model stronger high-frequency expressive power, thus better representing rapid motion, edge changes, and temporal details. For spatiotemporal graph alignment, based on the features enhanced by the temporal embedding, a cross-frame graph structure is established within a local window. The spatial correspondence between the current frame and neighboring frames is calculated through query-key-value projection and graph attention mechanisms, completing fine-grained cross-frame feature aggregation. This module includes a temporal alignment submodule and a spatial alignment submodule. The temporal alignment submodule utilizes continuous... The system employs temporal scalar modulation to weight adjacent frames; a spatial alignment submodule performs cross-frame attention aggregation on a window-level graph structure to replace or weaken explicit optical flow; implicit decoding reconstruction fuses the obtained spatial implicit representation, temporal implicit representation, and alignment enhancement features, outputting the predicted clean pixel value at the target coordinates via the decoder, ultimately generating the denoising result of the center frame or the entire video; unsupervised training optimization, based on the blind spot self-supervision principle, calculates the reconstruction loss only at the masked pixel location; preferably, the masking mean square error is used as the basic loss value, which can be averaged over different time steps for the entire sequence, and can be jointly optimized by combining temporal consistency regularization terms, perceptual constraint terms, or frequency domain constraint terms.

[0057] Compared with existing methods, this disclosure has at least the following advantages and beneficial effects: It significantly improves continuous spatiotemporal unified modeling, elevating the video restoration problem from "local convolutional prediction on discrete frames" to "function learning driven by continuous spatial-temporal coordinates," thus reducing the limitations of fixed grids on representation capabilities at the modeling level; it is more suitable for unsupervised training conditions, avoiding direct leakage of target pixel information through blind spot constraints, enabling the model to optimize parameters using only noisy video samples without relying on clean-labeled videos, making it suitable for applications where paired data is difficult to obtain in real-world scenarios; it has stronger spatial detail restoration capabilities, as it employs coordinate-driven implicit spatial mapping and high-frequency positional encoding, providing better representation of edges, textures, microstructures, and fine-grained patterns, thus improving the problem of excessive smoothing in detail regions in traditional blind-spot convolutional models; and it exhibits better temporal continuity and long-sequence stability, achieving cross-frame association through implicit temporal embedding modules and temporally aware spatial graph modules, no longer entirely relying on noise-sensitive explicit optical flow estimation, thus improving stability in complex operations. It exhibits more stable temporal performance in dynamic, fast-moving, occluded, and long-sequence video scenarios; it is highly modular and easy to deploy, consisting of modules such as input construction, feature extraction, spatial implicit modeling, temporal embedding, graph alignment, reconstruction output, and loss optimization, with a clear structure that facilitates engineering implementation on servers, edge devices, microscopic imaging processing equipment, and dedicated inference chips; it has a wide range of applicable scenarios, not only suitable for natural video denoising but also extend to various temporal image data such as low-light video, microscopic video, medical dynamic images, industrial inspection video, security monitoring video, and vehicle-mounted perception video; it is scalable, as the bidirectional feature extraction module, coordinate encoding module, temporal embedding module, and graph aggregation module can all be implemented using equivalent replacement structures, thus possessing good technical transferability and potential for future productization; finally, its strength lies not only in its single network structure but also in the synergistic combination of multiple key technical aspects such as continuous spatiotemporal implicit representation, blind spot constraint, temporally continuous embedding, and temporally aware graph alignment, providing strong overall solution protection value.

[0058] Based on the above embodiments, the unsupervised video denoising method based on continuous spatiotemporal implicit neural fields of this disclosure receives a noisy video sequence and selects multiple input video segments centered on the target time from the noisy video sequence according to a preset time window; extracts forward and reverse temporal features from the multiple input video segments respectively, and fuses the forward and reverse temporal features to obtain basic spatiotemporal features; obtains the coordinate encoding result of the target pixel position, and generates continuous spatial features of the target pixel position based on the coordinate encoding result and the basic spatiotemporal features under blind spot constraints; obtains the temporal embedding feature of the target time, determines the graph connection weights of the video frames at the target time and adjacent times, and determines the graph aggregation feature of the target time based on the graph connection weights and the features of neighboring frames; reconstructs based on the continuous spatial features, temporal embedding features, and graph aggregation features to obtain the denoising result of the target pixel position at the target time, and obtains all denoising results corresponding to all preset time windows to output the denoised video. By adopting the above technical solution, continuous spatiotemporal representation can be directly established and video denoising can be performed under unsupervised conditions, improving the efficiency and effect of video denoising.

[0059] Figure 6 This is a schematic diagram of an unsupervised video denoising device based on a continuous spatiotemporal implicit neural field, provided as an embodiment of this disclosure. This method can be implemented by software and / or hardware, and is generally integrated into electronic devices. Figure 6 As shown, the device includes: The input video construction module 610 is used to receive a noisy video sequence and select a multi-frame input video segment centered on a target time from the noisy video sequence according to a preset time window; wherein, any video frame in the noisy video sequence is taken as the video frame to be recovered, and the target time is the center time corresponding to the video frame to be recovered; The bidirectional feature extraction module 620 is used to extract forward temporal features and reverse temporal features from the multi-frame input video segments respectively, and to fuse the forward temporal features and the reverse temporal features to obtain basic spatiotemporal features; The coordinate encoding module 630 is used to obtain the coordinate encoding result of the target pixel position; The blind spot implicit neural field module 640 is used to generate continuous spatial features of the target pixel position based on the coordinate encoding result and the basic spatiotemporal features under blind spot constraints. Implicit temporal embedding module 650 is used to obtain the temporal embedding features of the target time. The time-aware spatial graph module 660 is used to determine the graph connection weights of video frames at the target time and adjacent times, and to determine the graph aggregation features of the target time based on the graph connection weights and the features of neighboring frames. The denoising and reconstruction module 670 is used to reconstruct based on the continuous spatial features, the temporal embedding features and the graph aggregation features to obtain the denoising result of the target time at the target pixel position; The output module 680 is used to obtain all the denoising results corresponding to the noisy video sequence to output the denoised video.

[0060] Optionally, the coordinate encoding module 630 is specifically used for: obtaining the pixel coordinates of the target pixel position; performing coordinate normalization processing on the pixel coordinates to obtain the normalized spatial coordinates of the target pixel position; and encoding the normalized spatial coordinates of the target pixel position based on a preset coordinate encoding function to obtain the coordinate encoding result of the target pixel position.

[0061] Optionally, the blind spot implicit neural field module 640 is specifically used to: mask the target pixel position through one of regular center masking, random hole masking, periodic masking or multi-scale masking, and process it based on the coordinate encoding result and the basic spatiotemporal features to obtain the continuous spatial features of the target pixel position.

[0062] Optionally, the implicit temporal embedding module 650 is specifically used to: calculate the target time, time mapping weight parameters, and preset time mapping bias parameters based on a preset periodic activation function to obtain the temporal embedding features.

[0063] Optionally, the time-aware spatial graph module 660 is specifically used for: acquiring the current node features at the target time, generating a query vector based on the current node features, acquiring candidate node features at adjacent time points, and generating a key vector based on the candidate node features; acquiring the time interval between the target time point and the adjacent time points; calculating the query vector, the key vector, the preset feature dimension, the preset time interval adjustment coefficient, and the time interval based on a preset connection weight formula to obtain the graph connection weight; and performing a weighted summation of the neighboring frame features according to the graph connection weight to obtain the graph aggregation features.

[0064] Optionally, the device further includes: an unsupervised training module, used to acquire the prediction result at the target pixel position at each preset time; determine the blind spot loss value, the temporal consistency loss value, and the regularization loss value based on a preset blind spot reconstruction loss function, a preset temporal consistency loss function, a preset parameter regularization loss function, and the prediction result; determine the total loss value based on the blind spot loss value, the temporal consistency loss value, the regularization loss value, and the weight coefficient of each loss item, and update all network parameters based on the total loss value and a preset loss threshold.

[0065] The unsupervised video denoising method based on continuous spatiotemporal implicit neural fields provided in this disclosure can execute the unsupervised video denoising device based on continuous spatiotemporal implicit neural fields provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0066] This disclosure also provides a computer program product, including a computer program / instruction that, when executed by a processor, implements the unsupervised video denoising method based on a continuous spatiotemporal implicit neural field provided in any embodiment of this disclosure.

[0067] According to one or more embodiments of this disclosure, this disclosure provides an electronic device, including: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the unsupervised video denoising method based on a continuous spatiotemporal implicit neural field as provided in any of the present disclosure.

[0068] According to one or more embodiments of the present disclosure, the present disclosure provides a computer-readable storage medium storing a computer program for performing an unsupervised video denoising method based on a continuous spatiotemporal implicit neural field as described in any of the present disclosure.

[0069] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0070] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0071] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. An unsupervised video denoising method based on continuous spatio-temporal implicit neural field, characterized in that, include: Receive a noisy video sequence, and select a multi-frame input video segment centered on a target time from the noisy video sequence according to a preset time window; wherein, any video frame in the noisy video sequence is taken as the video frame to be recovered, and the target time is the center time corresponding to the video frame to be recovered; Forward temporal features and reverse temporal features are extracted from the multi-frame input video segments respectively, and the forward temporal features and the reverse temporal features are fused to obtain basic spatiotemporal features; Obtain the coordinate encoding result of the target pixel position, and under blind spot constraints, generate continuous spatial features of the target pixel position based on the coordinate encoding result and the basic spatiotemporal features; The temporal embedding features of the target time are obtained, and the graph connection weights of the video frames at the target time and adjacent times are determined. The graph aggregation features of the target time are determined based on the graph connection weights and the features of the neighboring frames. Based on the continuous spatial features, the temporal embedding features, and the graph aggregation features, the denoising result of the target frame to be recovered is obtained, and all denoising results corresponding to the noisy video sequence are obtained to output the denoised video.

2. The method according to claim 1, characterized in that, The process of obtaining the coordinate encoding result of the target pixel position includes: Obtain the pixel coordinates of the target pixel location; The pixel coordinates are normalized to obtain the normalized spatial coordinates of the target pixel position; The normalized spatial coordinates of the target pixel position are encoded based on a preset coordinate encoding function to obtain the coordinate encoding result of the target pixel position.

3. The method according to claim 1, characterized in that, The step of generating continuous spatial features of the target pixel position based on the coordinate encoding result and the basic spatiotemporal features under blind spot constraints includes: The target pixel location is masked by one of the following methods: regular center masking, random hole masking, periodic masking, or multi-scale masking. Based on the coordinate encoding result and the basic spatiotemporal features, continuous spatial features of the target pixel location are obtained.

4. The method according to claim 1, characterized in that, The acquisition of the temporal embedding features of the target time includes: The time embedding feature is obtained by calculating the target time, time mapping weight parameters, and preset time mapping bias parameters based on a preset periodic activation function.

5. The method according to claim 1, characterized in that, Determining the graph connectivity weights of the video frames at the target time and adjacent time points includes: Obtain the current node features at the target time, generate a query vector based on the current node features, obtain the candidate node features at adjacent time points, and generate a key vector based on the candidate node features; Obtain the time interval between the target time and the adjacent time; The graph connection weights are obtained by calculating the query vector, the key vector, the preset feature dimension, the preset time interval adjustment coefficient, and the time interval based on the preset connection weight formula.

6. The method according to claim 5, characterized in that, Determining the graph aggregation features at the target time based on the graph connection weights and neighborhood frame features includes: The graph aggregation features are obtained by weighting and summing the features of neighboring frames according to the graph connection weights.

7. The method according to claim 1, characterized in that, The method further includes: Obtain the prediction result at the target pixel position at each preset time; The blind spot loss value, time consistency loss value, and regularization loss value are determined based on the preset blind spot reconstruction loss function, the preset time consistency loss function, the preset parameter regularization loss function, and the prediction results. The total loss value is determined based on the blind spot loss value, the time consistency loss value, the regularization loss value, and the weight coefficients of each loss item, and all network parameters are updated based on the total loss value and a preset loss threshold.

8. An unsupervised video denoising device based on a continuous spatiotemporal implicit neural field, characterized in that, include: An input video construction module is used to receive a noisy video sequence and select a multi-frame input video segment centered on a target time from the noisy video sequence according to a preset time window; wherein, any video frame in the noisy video sequence is taken as the video frame to be recovered, and the target time is the center time corresponding to the video frame to be recovered; The bidirectional feature extraction module is used to extract forward temporal features and reverse temporal features from the multi-frame input video segments respectively, and to fuse the forward temporal features and the reverse temporal features to obtain basic spatiotemporal features; The coordinate encoding module is used to obtain the coordinate encoding result of the target pixel position; The blind spot implicit neural field module is used to generate continuous spatial features of the target pixel position based on the coordinate encoding result and the basic spatiotemporal features under blind spot constraints. An implicit temporal embedding module is used to obtain the temporal embedding features of the target time. A time-aware spatial graph module is used to determine the graph connection weights of video frames at the target time and adjacent times, and to determine the graph aggregation features of the target time based on the graph connection weights and the features of neighboring frames. The denoising and reconstruction module is used to reconstruct based on the continuous spatial features, the temporal embedding features and the graph aggregation features to obtain the denoising result of the target time at the target pixel position; The output module is used to obtain all denoising results corresponding to the noisy video sequence to output a denoised video.

9. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the unsupervised video denoising method based on continuous spatiotemporal implicit neural fields as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program for executing the unsupervised video denoising method based on a continuous spatiotemporal implicit neural field as described in any one of claims 1-7.

Images