Method for reducing motion blur of erf model by using event and frame

Abstract

The invention relates to a method for reducing motion blur of an erf model by using events and frames, and belongs to the technical field of computer vision. The method comprises the following steps: converting an event stream into triple representation; generating a bidirectional optical flow field based on the event flow, and aligning the fuzzy frame and the event flow through event-guided deformation convolution; constructing a double-branch optimization framework; the double-branch features are dynamically fused through frequency domain attention gating, and the weight of the frequency domain attention gating is generated by event frequency spectrums through MLP; generating a pseudo-true value based on an event double integral model, aligning an event spectrum and rendering a high-frequency component, and carrying out chain constraint on the continuity of a cross-frame radiation field through event brightness change; event-frame non-linear response differences are modeled by a learnable camera response function. Clear reconstruction and rendering of the moving target in a complex dynamic scene are realized.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision, and particularly relates to a method for reducing motion blur of a NeRF model by using events and frames. BACKGROUND

[0002] The traditional NeRF assumes a static scene and instantaneous exposure, and fits a static radiance field through multi-view images. However, in a motion-blurred scene, the blurred frame is the integral result of the continuous motion process, rather than a single instantaneous state, which causes the volume rendering equation of NeRF to fail (dynamic changes of the radiance field in the integral interval). High-frequency motion details are overwhelmed by the low-frequency information of the blurred frame, and NeRF cannot recover sub-pixel level motion from the blur. The pixel values of the blurred frame correspond to a non-physically consistent radiance field (the same pixel may contain a mixture of multiple motion states), which causes NeRF to fall into a geometric-appearance ambiguity during optimization.

[0003] The advantage of the event camera is that it can provide a brightness change signal with a time resolution of microseconds, which can theoretically accurately capture the edge trajectory of high-speed motion. However, there are the following utilization bottlenecks: the event camera triggers events through the logarithmic domain change of brightness, which has a non-negligible nonlinear deviation from the linear exposure of the traditional frame camera, and directly aligning the events and frames in the existing method will cause artifacts; the event stream is an asynchronous pulse sequence, and a structured representation needs to be designed to be compatible with the continuous optimization framework of NeRF; the event stream only contains the direction of brightness change, without absolute brightness values, which causes the traditional photometric loss to be unable to be directly applied.

[0004] In the existing event-frame joint deblurring method, events are only used as auxiliary input, and their high-frequency motion prior is not utilized. Events are used as an additional supervision signal, but the physical consistency between the event brightness model and the NeRF radiance field is not addressed. The event stream is a sparse spatiotemporal pulse, and the blurred frame is a dense pixel grid, which need to be jointly optimized in a unified space. The existing method lacks a differentiable cross-modal alignment mechanism. SUMMARY

[0005] To solve the above problems in the prior art, the application provides a method for reducing motion blur of a NeRF model by using events and frames, comprising: S1: converting the event stream into a triple representation: a point cloud representation, a dynamic voxel grid, and a frequency domain event graph; generating a bidirectional optical flow field based on the event stream, and aligning the blurred frame and the event stream through event-guided deformation convolution; S2: constructing a double-branch optimization framework, converting the optical flow field into a motion field in the spatial domain, using an explicit feature volume to accelerate rendering, and inputting the feature of the sampling point into an MLP; performing consistency loss on the event STFT phase and the rendering image phase in the frequency domain branch, and extracting cross-scale frequency domain features through complex convolution; S3: dynamically fusing the features of the two branches through a frequency domain attention gate, the weight of the frequency domain attention gate being generated by an MLP from the event spectrum; S4: generating pseudo-real values based on an event double integration model, aligning the event spectrum with the rendered high frequency components, and constraining the continuity of the cross-frame radiance field through the chain of event luminance changes; S5: modeling the event-frame nonlinear response difference through a learnable camera response function.

[0006] Specifically, the dynamic voxel grid adjusts the time bin size through a learnable time kernel function; the point cloud representation retains the original event spatiotemporal coordinates and is randomly sampled after normalization; the frequency domain event graph performs short-time Fourier transform on the event stream divided into windows, and extracts the phase spectrum as high frequency motion features.

[0007] Specifically, the forward optical flow of the bidirectional optical flow field is used to estimate the motion from the current frame to the next frame, and the backward optical flow is used to estimate the motion from the next frame to the current frame; based on the event gradient and luminance change constraint, the blurred frame and the event stream are aligned through event-guided deformation convolution, specifically including: S101: the event stream records the spatiotemporal coordinates of the pixel-level luminance change, the blurred frame is modeled as the integration result of the continuous latent image through the double integration model, and the event represents the change between adjacent latent images; the luminance gradient change and the time difference in the event stream are used to establish the constraint equation of the optical flow field; the optical flow field is iteratively optimized by minimizing the difference between the predicted luminance change of the event and the real change; the high spatiotemporal resolution information of the event is converted into dense optical flow; S102: based on the optical flow estimation of S101, a deformable convolution kernel is set, and the convolution sampling position is dynamically adjusted to align the blurred frame and the event stream; the optical flow field is input into a lightweight MLP to predict the sampling offset of each convolution kernel, and the deformable convolution kernel is applied to the blurred frame; S103: constructing an image pyramid from low to high resolution, refining the optical flow and offset at each level, dividing the event stream into short-time windows, and calculating the local optical flow and weighted fusion respectively; S104: integrating event-frame luminosity loss, motion consistency loss, and bidirectional cyclic consistency loss in the loss function.

[0008] Specifically, the spatial domain branch maps the optical flow field under the 2D vector field to the 3D motion field through an MLP, and the output is the motion offset of each spatial coordinate; The scene is divided into a voxel grid, each voxel stores a feature vector, and the feature of any point is queried through interpolation; spherical tracing is used to sample along the light in 3D space, and the motion field is used to adjust the sampling position; For each sampling point, the explicit feature volume is queried to obtain the feature vector; The feature vector of the sampling point is input into an MLP, and the MLP outputs the color and density of each sampling point; The input of the MLP includes the sampling point coordinates, the view direction and the adjusted motion field features; the output is used for volume rendering integration; during training, the MLP parameters are optimized through the loss function; The volume rendering integration formula is used to accumulate color and density along the light ray to generate a rendered 2D image.

[0009] Specifically, the frequency domain branch performs windowing processing on the event stream, and performs STFT in each window, and the phase spectrum is retained as a feature map; The same STFT processing is applied to the image rendered by the spatial domain branch to obtain its phase spectrum; A loss function is defined to compare the event STFT phase and the rendered image STFT phase to enforce motion consistency; A complex convolutional neural network is used to process the STFT output; the input is a complex feature map, and multi-scale features are extracted through a complex convolutional layer; the feature maps are fused through upsampling / downsampling, and the output is a multi-scale frequency domain feature vector; The extracted frequency domain feature vector is input to the frequency domain attention gate of S3, and is dynamically fused with the spatial domain branch feature.

[0010] Specifically, the process of generating pseudo-true values by the event double integration model includes: S401: Divide the event stream into event packets according to the time window, and establish the radiance field continuity equation based on the chain constraint of the luminance change; S402: Reconstruct the latent image sequence through double integration; the first-order integration integrates the event polarity to generate the luminance change, and the second-order integration accumulates along the time axis to generate the pseudo-true value corresponding image; S403: Align the pseudo-true value with the high-frequency component of the rendered image, calculate the high-frequency energy ratio in the Fourier domain, and constrain through the L1 loss.

[0011] Specifically, the learnable camera response function modeling method is: the event camera and the traditional frame camera are explicitly modeled through the frame branch CRF and the event branch CRF to model the difference in nonlinear response to luminance; the frame branch CRF is used to fit the nonlinear response of the standard camera, and the event branch CRF models the logarithmic nonlinear transformation to capture its sensitivity to luminance change; the frame branch CRF and the event branch CRF are associated through shared hidden variables.

[0012] Further, the event-frame photometric loss is used to constrain the consistency of the luminance change predicted by the event stream and the actual observed value of the blurred frame, and the loss function is designed to input the blurred frame, the latent clear image sequence generated by the double integration model from the event stream, and the offset of the optical flow aligned to the reference frame.

[0013] Further, the motion consistency loss is used to ensure that the event-generated optical flow field is consistent with the geometric motion of the NeRF rendering; the loss function is designed to input the 2D optical flow field estimated by the event stream and the motion field predicted by the spatial branch MLP after being projected to the 2D plane.

[0014] Further, the bidirectional cycle consistency loss is used to verify the cycle consistency of the event-optical flow alignment, and the input is designed as the forward optical flow and the backward optical flow; the loss function is designed based on the 2D cycle consistency and the event brightness change closure; the event brightness change closure verifies whether the brightness change is zero after the optical flow alignment.

[0015] The application effectively solves the alignment problem of the blurred frame and the event stream by performing triple representation on the event stream and generating bidirectional optical flow field, and effectively solves the alignment problem of the blurred frame and the event stream by combining the event-guided deformation convolution, which provides an accurate basis for subsequent rendering and optimization. The constructed double-branch optimization framework, the spatial branch and the frequency branch cooperate with each other, respectively process data from different angles, so that the rendered image can better restore the motion details. The introduction of the frequency domain attention gate realizes the dynamic fusion of the double-branch features, and improves the utilization efficiency of the model to the features.

[0016] The pseudo-real value generated by the event double-integral model can be effectively aligned with the high-frequency component of the rendered image, ensuring the continuity of the cross-frame radiance field. The learnable camera response function modeling method solves the nonlinear deviation problem between the event camera and the traditional frame camera, avoiding the generation of artifacts.

[0017] In terms of loss function, the comprehensive use of event-frame luminosity loss, motion consistency loss and bidirectional cycle consistency loss constrains the model from multiple dimensions, ensuring the accuracy and stability of the model. Through the optimization of these loss functions, the model can jointly optimize the event stream and the blurred frame in a unified space, effectively reducing the blur problem of the NeRF model in the motion blur scene, improving the quality of the rendered image, and making the model better adapt to dynamic scenes, providing an effective solution for the computer vision field in dealing with motion blur problems. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to facilitate the understanding of those skilled in the art, the application will be further described below in conjunction with the drawings.

[0019] Figure 1 A flowchart of a method for reducing motion blur of a NeRF model using events and frames. DETAILED DESCRIPTION

[0020] In order to further illustrate the technical means and effects adopted by the present application to achieve the predetermined object of the present application, the specific embodiments, structures, features and effects according to the present application are described in detail below in combination with the drawings and preferred embodiments.

[0021] Please refer to Figure 1 A method for reducing motion blur of a nerf model using events and frames, comprising: S1: converting an event stream into a triple representation: a point cloud representation, a dynamic voxel grid, and a frequency domain event graph; generating a bidirectional optical flow field based on the event stream, and aligning a blurred frame with the event stream through event-guided deformable convolution; S2: constructing a double-branch optimization framework, converting the optical flow field into a motion field in the spatial domain branch, accelerating rendering by using an explicit feature volume, and inputting the feature of the sampling point into an MLP; performing consistency loss on the event STFT phase and the rendered image phase in the frequency domain branch, and extracting cross-scale frequency domain features through complex convolution; S3: dynamically fusing the features of the double branches through a frequency domain attention gate, and generating a weight of the frequency domain attention gate through an MLP from an event spectrum; S4: generating a pseudo ground truth based on an event double integration model, aligning the event spectrum with the rendered high-frequency component, and constraining the cross-frame radiance field continuity through the event brightness change chain; S5: modeling the event-frame nonlinear response difference through a learnable camera response function.

[0022] Specifically, the dynamic voxel grid adjusts the time bin size through a learnable time kernel function; the point cloud representation retains the original event space-time coordinates, and is randomly sampled after normalization; and the frequency domain event graph performs short-time Fourier transform on the event stream in a window, and extracts the phase spectrum as a high-frequency motion feature.

[0023] Specifically, the forward optical flow of the bidirectional optical flow field is used to estimate the motion from the current frame to the next frame, and the backward optical flow is used to estimate the motion from the next frame to the current frame; based on the event gradient and the brightness change constraint, the blurred frame is aligned with the event stream through event-guided deformable convolution, specifically including: S101: the event stream records the space-time coordinates of the pixel-level brightness change, the blurred frame is modeled as the integration result of the continuous latent image through a double integration model, and the event represents the change amount between adjacent latent images; the brightness gradient change and the time difference in the event stream are used to establish a constraint equation of the optical flow field; the optical flow field is iteratively optimized by minimizing the difference between the predicted brightness change of the event and the actual change; and the high space-time resolution information of the event is converted into a dense optical flow; S102: based on the optical flow estimation of S101, a deformable convolution kernel is set, and the convolution sampling position is dynamically adjusted to align the blurred frame with the event stream; the optical flow field is input into a lightweight MLP to predict the sampling offset of each convolution kernel, and the deformable convolution kernel is applied to the blurred frame; S103: Constructing an image pyramid from low to high resolution, refining the optical flow and offset step by step, dividing the event stream into short time windows, calculating the local optical flow respectively and weighted fusion; S104: Integrating event-frame luminosity loss, motion consistency loss and bidirectional cycle consistency loss in the loss function.

[0024] Specifically, the spatial domain branch maps the optical flow field under the 2D vector field to the 3D motion field through the MLP, and the output is the motion offset of each spatial coordinate; Divide the scene into a voxel grid, store the feature vector in each voxel, and query the feature of any point through interpolation; use spherical tracking to sample along the light in 3D space, and the motion field is used to adjust the sampling position; For each sampling point, query the explicit feature volume to obtain the feature vector; Input the feature vector of the sampling point into an MLP, and the MLP outputs the color and density of each sampling point; The input of the MLP includes the sampling point coordinates, view direction and adjusted motion field features; the output is used for volume rendering integration; during training, the MLP parameters are optimized through the loss function; Use the volume rendering integration formula to accumulate color and density along the light to generate a rendered 2D image.

[0025] In this embodiment, the input is the bidirectional optical flow field estimated by the event stream (resolution HxWx2); map the 2D pixel coordinates (u, v) to the normalized device coordinates (NDC) space [-1, 1]; use the depth d estimated by the current NeRF (obtained by spherical tracking from coarse to fine) to back-project the 2D optical flow (Δu, Δv) to 3D motion field offset (Δx, Δy, Δz): , Where K is the camera intrinsic matrix; Input the back-projected 3D offset into a lightweight MLP (2 layers of 256-dimensional ReLU) to output the motion field of the scene space (motion offset of each 3D point); In dynamic voxel grid construction, the voxel parameters are set to cover the bounding box calculated adaptively from the event point cloud, and the voxel time bin size is dynamically adjusted through a learnable time kernel function: , Where E t is the standard deviation of the local event timestamp, w i ,b i are learnable parameters; In spherical tracking sampling guided by the motion field, for each light r(t)=o+td, the sampling position is dynamically adjusted under the action of the motion field M(x): , where t max is the maximum depth of light, ensuring that distant motion influences decay; In the explicit feature volume query, the adjusted sampling points are queried for their 8-neighborhood features, and a 32-dimensional feature vector is obtained through trilinear interpolation; the voxel features are updated in real time through sparse convolution, and only the calculation of the changed area of the motion field is triggered. The MLP structure in the embodiment is as follows: Input: sampling point coordinates (3 dimensions), view direction d (3 dimensions), motion field feature M (3 dimensions), and voxel feature f (t) (32 dimensions); (3 dimensions); Output: color c (t) and density σ (t); The implementation structure is a 3-layer 256-dimensional ReLU MLP, and the last layer outputs are activated by Softplus (density) and Sigmoid (color); The volume rendering formula is as follows: , where is the cumulative transmittance, δ i is the distance between adjacent sampling points; The total loss is the weighted sum of the photometric loss (aligned with the blurred frame), the motion consistency loss (aligned with the event optical flow), and the bidirectional cyclic consistency loss, and the regularization term is integrated, which specifically includes the motion smoothing loss to prevent the motion field from overfitting noise and the voxel sparsity loss to encourage the feature of empty voxels to be zero.

[0026] Specifically, the frequency domain branch performs windowing processing on the event stream, and performs STFT in each window, and the phase spectrum is reserved as a feature map; The same STFT processing is applied to the image rendered by the spatial domain branch to obtain its phase spectrum; The loss function is defined to compare the event STFT phase and the rendered image STFT phase, and the motion consistency is forced; The STFT output is processed using a complex convolutional neural network; the input is a complex feature map, and multi-scale features are extracted through a complex convolutional layer; the feature maps are fused through upsampling / downsampling, and the output is a multi-scale frequency domain feature vector; The extracted frequency domain feature vector is input into the frequency domain attention gate of S3, and is dynamically fused with the spatial domain branch feature.

[0027] Specifically, the process of generating pseudo-real values by the event double integration model includes: S401: divide the event stream into event packets by time window, and establish the radiation field continuity equation based on the event brightness change chain constraint; S402: reconstruct the latent image sequence by double integration; the first-order integration generates the brightness change quantity by integrating the event polarity, and the second-order integration generates the pseudo-real value corresponding image by accumulating along the time axis; S403: align the pseudo-real value with the high-frequency component of the rendered image, calculate the high-frequency energy ratio in the Fourier domain, and constrain by L1 loss.

[0028] Specifically, the learnable camera response function modeling method is: the differences in nonlinear response of brightness between event cameras and traditional frame cameras are explicitly modeled through frame branch CRF and event branch CRF; the frame branch CRF is used to fit the nonlinear response of a standard camera, and the event branch CRF models the logarithmic domain nonlinear transformation to capture its sensitivity to brightness change; the frame branch CRF and the event branch CRF are associated through shared hidden variables.

[0029] Further, the event-frame radiance loss is used to constrain the consistency of the brightness change predicted by the event stream and the actual observation value of the blurred frame, and the loss function is designed to input the blurred frame, the latent clear image sequence generated by the double integration model of the event stream, and the offset of the optical flow aligned to the reference frame.

[0030] In this embodiment, the latent clear image sequence is differentially bilinearly resampled according to the optical flow offset: , The event density is used as an adaptive weight: , , The loss after association is: ; Further, the motion consistency loss is used to ensure that the optical flow field generated by the event is consistent with the geometric motion rendered by NeRF; the loss function is designed to input the 2D optical flow field estimated by the event stream and the motion field projected to the 2D plane predicted by the spatial branch MLP.

[0031] In this embodiment, for each light ray r, the 3D sampling point x i The calculation is as follows: , where is the perspective projection, T i ,α i is determined by the volume rendering weight.

[0032] The consistency error adopts a robust Charbonnier loss: ; Further, the bidirectional cycle consistency loss is used to verify the cycle consistency of event-optical flow alignment, the input is designed as forward optical flow and backward optical flow, and the loss function is designed based on 2D cycle consistency and event brightness change closure; the event brightness change closure verifies whether the brightness change is zero after the optical flow alignment by using the event polarity.

[0033] In the embodiment, the forward optical flow is represented as , and the backward optical flow is represented as ; The calculation formula of 2D cycle consistency is as follows: , The calculation method of event brightness change closure is to verify the event polarity : , , Joint cycle loss: , The final loss is a weighted sum of three terms: , Where λ i , μ i is dynamically adjusted according to the event signal-to-noise ratio (predicted by a lightweight MLP).

[0034] The above is only the preferred embodiment of the present application, and is not intended to limit the present application in any form. Although the present application has been disclosed as above with the preferred embodiment, it is not intended to limit the present application. Any person skilled in the art can make some changes or modifications to the above disclosed technical content without departing from the scope of the technical solution of the present application, and any equivalent embodiments with equivalent changes are equivalent. Any simple modification, equivalent change and modification of the above embodiments according to the technical essence of the present application are still within the scope of the technical solution of the present application.

Claims

1. A method for reducing motion blur in Nerf models using events and frames, characterized in that, Comprise: S1: convert the event stream into a triple representation: point cloud representation, dynamic voxel grid, and frequency domain event graph; Generate a bidirectional optical flow field based on the event stream, and align the blurred frame with the event stream through event-guided deformable convolution; S2: build a double-branch optimization framework, the spatial branch converts the optical flow field into a motion field, uses an explicit feature volume to accelerate rendering, and inputs the feature of the sampling point into an MLP; The frequency domain branch imposes consistency loss on the event STFT phase and the rendered image phase, and extracts cross-scale frequency domain features through complex convolution; S3: dynamically fuse the features of the double branches through a frequency domain attention gate, and the weight of the frequency domain attention gate is generated by an MLP from the event spectrum; S4: generate pseudo ground truth based on the event double integral model, align the event spectrum with the rendered high-frequency components, and constrain the cross-frame radiance field continuity through the event brightness change chain; S5: model the event-frame nonlinear response difference through a learnable camera response function.

2. The method of claim 1, wherein, The dynamic voxel grid adjusts the time bin size through a learnable time kernel function; the point cloud representation retains the original event space-time coordinates and is randomly sampled after normalization; and the frequency domain event graph performs short-time Fourier transform on the event stream in windows to extract the phase spectrum as high-frequency motion features.

3. The method of claim 1, wherein, The forward optical flow of the bidirectional optical flow field is used to estimate the motion from the current frame to the next frame, and the backward optical flow is used to estimate the motion from the next frame to the current frame; Based on the event gradient and brightness change constraint, the blurred frame is aligned with the event stream through event-guided deformable convolution, specifically including: S101: the event stream records the space-time coordinates of the pixel-level brightness change, the blurred frame is modeled as the integral result of the continuous latent image through the double integral model, and the event represents the change between adjacent latent images; the brightness gradient change and time difference in the event stream are used to establish the constraint equation of the optical flow field; the optical flow field is iteratively optimized by minimizing the difference between the predicted brightness change of the event and the actual change; and the high space-time resolution information of the event is converted into a dense optical flow; S102: based on the optical flow estimation of S101, set a deformable convolution kernel and dynamically adjust the convolution sampling position to align the blurred frame with the event stream; input the optical flow field into a lightweight MLP to predict the sampling offset of each convolution kernel, and apply the deformable convolution kernel to the blurred frame; S103: build an image pyramid from low to high resolution, refine the optical flow and offset at each level, and divide the event stream into short-time windows to calculate the local optical flow and weighted fusion; S104: integrate the event-frame luminosity loss, motion consistency loss, and bidirectional cyclic consistency loss in the loss function.

4. The method of claim 1, wherein, The spatial branch maps the optical flow field under the 2D vector field to the 3D motion field through an MLP, and the output is the motion offset of each spatial coordinate; Divide the scene into a voxel grid, store a feature vector in each voxel, and query the feature of any point through interpolation; use spherical tracking to sample along the light in 3D space, and the motion field is used to adjust the sampling position; For each sampling point, query the explicit feature volume to obtain the feature vector; Input the feature vector of the sampling point into an MLP, and the MLP outputs the color and density of each sampling point; The input of the MLP includes sample point coordinates, view direction and adjusted motion field features; The output is used for volume rendering integral; during training, the MLP parameters are optimized by loss function; Using the volume rendering integral formula, the color and density are accumulated along the light ray to generate a rendered 2D image.

5. The method of claim 1, wherein, The frequency domain branch processes the event stream in windows, and performs STFT in each window, and the phase spectrum is reserved as a feature map; The same STFT processing is applied to the image rendered by the spatial domain branch to obtain its phase spectrum; A loss function is defined to compare the event STFT phase and the rendered image STFT phase to force motion consistency; A complex convolutional neural network is used to process the STFT output; the input is a complex feature map, and multi-scale features are extracted through a complex convolutional layer; the feature maps are fused through upsampling / downsampling, and the output is a multi-scale frequency domain feature vector; The extracted frequency domain feature vector is input to the frequency domain attention gate of S3, and is dynamically fused with the spatial domain branch features.

6. The method of claim 1, wherein, The process of generating pseudo-real values by the event double integral model includes: S401: Divide the event stream into event packets according to the time window, and establish the radiation field continuity equation based on the chain constraint of event brightness change; S402: Reconstruct the latent image sequence by double integral; the first-order integral integrates the event polarity to generate the brightness change quantity, and the second-order integral accumulates along the time axis to generate the pseudo-real value corresponding image; S403: Align the pseudo-real value with the high-frequency component of the rendered image, calculate the high-frequency energy ratio in the Fourier domain, and constrain through L1 loss.

7. The method of claim 1, wherein, The learnable camera response function modeling method is to explicitly model the difference in nonlinear response of brightness between event cameras and traditional frame cameras through frame branch CRF and event branch CRF; The frame branch CRF is used to fit the nonlinear response of the standard camera, and the event branch CRF models the logarithmic domain nonlinear transformation to capture its sensitivity to brightness change; The frame branch CRF and the event branch CRF are associated through shared hidden variables.

8. The method of claim 3, wherein, The event-frame photometric loss is used to constrain the consistency of the brightness change predicted by the event stream and the actual observation value of the blurred frame; the loss function design input is the blurred frame, the latent clear image sequence generated by the double integral model of the event stream, and the offset of the optical flow aligned to the reference frame.

9. The method of claim 3, wherein, The motion consistency loss is used to ensure that the optical flow field generated by the event is consistent with the geometric motion of the NeRF rendering; The loss function design input is the 2D optical flow field estimated by the event stream and the motion field predicted by the spatial domain branch MLP after being projected to the 2D plane.

10. The method of claim 3, wherein, The bidirectional cycle consistency loss is used to verify the cycle consistency of event-optical flow alignment; the input design is the forward optical flow and the backward optical flow, and the loss function is designed based on 2D cycle consistency and event brightness change closure; the event brightness change closure uses the event polarity to verify whether the brightness change after optical flow alignment is zero.

Images