Anti-aliasing three-dimensional gaussian reconstruction method and apparatus

Abstract

The embodiment of the specification provides an anti-fuzzy three-dimensional Gaussian reconstruction method and device, wherein the anti-fuzzy three-dimensional Gaussian reconstruction method comprises: first initialization based on a random point cloud, joint photometric blur loss, threshold-free event loss and edge-aware loss; second initialization after determining the depth information; and reconstruction based on the second initialization result. The application does not need to rely on pre-computed poses, eliminates noise and blur through threshold-free event constraints and edge-aware loss, and realizes high-fidelity three-dimensional reconstruction.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of three-dimensional reconstruction technology, and in particular to an anti-fuzzy three-dimensional Gaussian reconstruction method. Background Technology

[0002] In recent years, 3D Gaussian sputtering (3DGS) has emerged as an important paradigm for high-fidelity scene representation. Typically, these methods heavily rely on images with accurate camera poses. These poses are usually pre-estimated using Structure of Motion (SfM) methods. However, in real-world applications, motion blur remains a common bottleneck for frame-based cameras when faced with rapid self-motion. Severe motion blur often leads to suboptimal pose estimations by SfM, or even direct tracking failure. This pose inaccuracy propagates further into subsequent reconstruction stages, resulting in severe blur artifacts and geometric distortions in the 3DGS-based reconstruction results.

[0003] To address the reconstruction challenges posed by motion blur, some existing purely visual deblurring 3D reconstruction methods attempt to jointly optimize scene geometry and camera pose by integrating the physical camera's motion trajectory within the exposure time. However, in high-speed, low-frame-rate scenes, purely visual deblurring methods often struggle to resolve the inherent blurriness, highlighting the necessity of introducing high-speed sensors to provide additional motion cues.

[0004] Unlike traditional cameras, biologically inspired event cameras can asynchronously record pixel-level intensity changes with extremely low latency, down to the microsecond level. Therefore, event cameras are inherently resistant to motion blur, even under extreme self-motion. Based on this, recent research has begun exploring 3DGS frameworks with anti-blur properties, mitigating motion-induced artifacts by integrating event streams. Existing event-assisted reconstruction methods typically utilize luminance signals from the event stream to constrain image intensity changes, thereby supplementing missing scene information between consecutive frames.

[0005] Existing techniques for 3D scene reconstruction combining blurred images and event sequences suffer from three significant drawbacks: First, in terms of scene initialization and pose estimation, existing methods cannot escape their reliance on accurate prior poses. Current technologies still heavily depend on traditional SfM (such as COLMAP) to provide the initial camera pose. When faced with extreme blur caused by rapid motion, traditional SfM is prone to feature extraction failures, leading to incorrect initial poses that directly cause severe reconstruction artifacts and geometric distortions. Second, regarding noise resistance and physical modeling, existing methods cannot adapt to dynamically changing event contrast thresholds. Existing models generally assume that the contrast threshold triggered by events is a static, fixed value. However, in complex real-world environments, this physical threshold fluctuates dramatically with lighting conditions. This mismatch between the fixed preset and the actual physical threshold introduces severe background noise into the reconstruction system, fundamentally limiting rendering quality. Finally, in terms of utilizing spatiotemporal dynamic information, existing methods cannot fully extract the high-frequency continuous motion features of the event stream. Existing technologies treat events merely as simple, relatively static brightness change signals, ignoring their naturally encoded continuous spatiotemporal motion trajectories. When processing blurred images with low frame rates and large displacements, the system lacks robust dynamic kinematic constraints, making it difficult to simultaneously support high-precision scene geometry reconstruction and accurate camera trajectory recovery. Therefore, for 3D reconstruction scenes suffering from severe motion blur, a key technical challenge urgently needing to be addressed is how to effectively eliminate dynamic environmental noise by designing a threshold-free event-photometric joint supervision mechanism and spatiotemporal motion geometry constraints, while completely eliminating dependence on external SfM pre-calculated poses. This would allow for the dual goals of high-fidelity 3D scene reconstruction and accurate recovery of continuous camera motion trajectories.

[0006] Therefore, for 3D reconstruction scenes suffering from severe motion blur, how to effectively eliminate dynamic environmental noise by designing a threshold-free event-photometric joint supervision mechanism and spatiotemporal motion geometric constraints, under the premise of completely getting rid of dependence on external SfM pre-calculated pose, and thus achieve the dual goals of high-fidelity 3D scene reconstruction and accurate recovery of continuous camera motion trajectory, is a key technical problem that urgently needs to be solved in this field. Summary of the Invention

[0007] In view of this, embodiments of this specification provide an anti-fuzzy 3D Gaussian reconstruction method. One or more embodiments of this specification also relate to an anti-fuzzy 3D Gaussian reconstruction apparatus, a computing device, a computer-readable storage medium, and a computer program, to address the technical deficiencies existing in the prior art.

[0008] According to a first aspect of the embodiments of this specification, an anti-fuzzy three-dimensional Gaussian reconstruction method is provided, comprising: Random point clouds are determined based on a 3D Gaussian scene. The random point clouds are then initialized to determine the initialization result. The initialization includes photometric blurring loss, event loss without threshold constraints, and edge perception loss. Determine the depth information, and perform a second initialization based on the first initialization result and the depth information to determine the second initialization result; The reconstruction result is obtained by reconstructing based on the second initialization result.

[0009] In one possible implementation, the first initialization also includes: randomly initializing the camera pose, generating continuous camera poses using continuous spline curve interpolation, and jointly optimizing the photometric blur loss, the event loss without threshold constraints, and the edge perception loss.

[0010] In one possible implementation, the threshold-free event loss is constructed as follows: the time interval between adjacent blurred frames is uniformly divided into multiple sub-blocks and the cumulative events are obtained; for the logarithmic intensity change and cumulative events of the image rendered from the 3D Gaussian scene, the spatial mean and standard deviation are calculated in the pixel regions where events exist to normalize the local statistical distribution; the distance and structural similarity between the normalized logarithmic intensity change and the normalized cumulative events are calculated to construct the threshold-free event loss.

[0011] In one possible implementation, the edge-aware loss is constructed by aggregating the event stream over time intervals using an exponentially decaying window to generate an event boundary map; applying an edge detection filter to the image and event boundary map rendered from a 3D Gaussian scene; and constructing the edge-aware loss by calculating the cross-correlation between the filtered rendered image and the filtered event boundary map.

[0012] In one possible implementation, depth information is obtained by predicting a dense depth map of the scene using a pre-trained monocular depth network; the second initialization includes joint optimization by incorporating a contrast-maximizing variance loss, which is constructed as follows: based on the condition that optical flow remains constant in the spatiotemporal neighborhood spanned by the event, the event is projected onto a reference time using the optical flow velocity to construct a distorted event image, and the contrast-maximizing variance loss is constructed by maximizing the variance of the average distorted event image of multiple event sub-blocks.

[0013] In one possible implementation, the reconstruction result is obtained based on the second initialization result, including: alternatingly executing pose tracking and scene mapping, wherein the 3D Gaussian scene is fixed in the tracking phase to optimize the camera pose, and the camera pose is fixed in the mapping phase to update the 3D Gaussian properties; during the alternating execution, sliding window bundle adjustment is also included: within the most recent predetermined number of time blocks, the continuous camera pose and the 3D Gaussian scene parameters are simultaneously used as variable parameters for synchronous joint optimization.

[0014] One possible implementation also includes color rendering: after obtaining the reconstruction results, the camera pose and 3D Gaussian properties are frozen, and the spherical harmonic coefficients of the 3D Gaussian are optimized using color-blurred frames.

[0015] According to a second aspect of the embodiments of this specification, an anti-fuzzy three-dimensional Gaussian reconstruction apparatus is provided, comprising: The first initialization module is configured to determine a random point cloud based on a 3D Gaussian scene, perform a first initialization on the random point cloud, and determine a first initialization result; wherein, the first initialization includes photometric blurring loss, no threshold constraint event loss, and edge perception loss; The second initialization module is configured to determine depth information and perform a second initialization based on the first initialization result and the depth information to determine the second initialization result. The scene reconstruction module is configured to reconstruct the scene based on the second initialization result to obtain the reconstruction result.

[0016] According to a third aspect of the embodiments of this specification, a computing device is provided, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the above-described anti-fuzzy 3D Gaussian reconstruction method.

[0017] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the steps of the above-described anti-fuzzy 3D Gaussian reconstruction method.

[0018] According to a fifth aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the above-described anti-fuzzy three-dimensional Gaussian reconstruction method.

[0019] This specification provides an anti-blurring 3D Gaussian reconstruction method and apparatus. The anti-blurring 3D Gaussian reconstruction method includes: performing a first initialization based on random point clouds, combining photometric blurring loss, threshold-free event loss, and edge-aware loss; performing a second initialization after determining depth information; and reconstructing based on the second initialization result. This application does not rely on pre-calculated poses and eliminates noise and blur through threshold-free event constraints and edge-aware loss to achieve high-fidelity 3D reconstruction. Attached Figure Description

[0020] Figure 1 This is a flowchart of an anti-fuzzy 3D Gaussian reconstruction method provided in one embodiment of this specification; Figure 2 This is a schematic diagram illustrating the reconstruction effect of an anti-fuzzy 3D Gaussian reconstruction method provided in one embodiment of this specification; Figure 3 This is an initialization diagram of an anti-fuzzy 3D Gaussian reconstruction method provided in one embodiment of this specification; Figure 4 This is a schematic diagram of the structure of an anti-fuzzy three-dimensional Gaussian reconstruction device provided in one embodiment of this specification; Figure 5 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0021] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0022] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0023] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0024] This specification provides an anti-fuzzy 3D Gaussian reconstruction method, and also relates to an anti-fuzzy 3D Gaussian reconstruction apparatus, a computing device, and a computer-readable storage medium, which are described in detail in the following embodiments.

[0025] See Figure 1 , Figure 1 A flowchart of an anti-blurring 3D Gaussian reconstruction method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0026] Step 101: Determine random point cloud based on 3D Gaussian scene, perform first initialization on random point cloud, and determine first initialization result; wherein, first initialization includes photometric blur loss, no threshold constraint event loss and edge perception loss; Step 102: Determine the depth information, and perform a second initialization based on the first initialization result and the depth information to determine the second initialization result; Step 103: Reconstruct based on the second initialization result to obtain the reconstruction result.

[0027] In this context, a 3D Gaussian scene refers to a scene model represented using 3D Gaussian sputtering technology, which describes the scene's geometric structure and appearance attributes through a set of learnable 3D Gaussian kernels. Random point clouds refer to a set of points randomly generated within the boundary volume of 3D space; these points serve as the initial center positions of the 3D Gaussian kernels, independent of any pre-calculated external data. First initialization refers to the process of initially optimizing scene geometry and camera motion from a random state, without precise prior camera pose. Photometric blur loss is a loss function that measures the difference between the rendered blurred image and the actually captured blurred image, used to constrain the scene's appearance consistency. Threshold-free event loss is a loss function that does not rely on a preset static event trigger threshold, but rather monitors the relationship between the event stream and image brightness changes through normalized statistical distribution alignment. Edge-aware loss is a loss function that enhances structural sharpness by comparing the high-frequency edge information generated by the rendered image and the event stream. Depth information refers to the distance value corresponding to each pixel in the scene, used to provide geometric prior constraints. Second initialization refers to the process of fine-tuning by introducing depth priors and contrast maximization constraints based on the coarse geometry and motion provided by the first initialization. The reconstruction result refers to the final 3D scene model, including parameters such as the position, shape, opacity, and color of the 3D Gaussian kernel.

[0028] The present invention will be further described below through a detailed embodiment: In a specific implementation scenario, a mobile robot equipped with a traditional color camera and an event camera needs to perform 3D reconstruction of an indoor environment while rotating rapidly. Due to the extremely high camera speed, the image sequences captured by the traditional camera exhibit severe motion blur, and conventional structure-of-motion (SOR) methods cannot extract effective feature points, leading to pose estimation failure.

[0029] The aforementioned anti-blurring 3D Gaussian reconstruction method is implemented according to the following steps: First, the system randomly generates a large number of point clouds within a preset 3D boundary volume. These point clouds serve as the initial Gaussian kernel center positions for the 3D Gaussian scene. Simultaneously, the system randomly initializes the pose parameters of each camera near the identity matrix, without relying on any external calibration or pre-calculated data. Next, the system executes the first initialization process. In this process, the system interpolates the randomly initialized discrete camera poses using continuous B-spline curves to generate smooth, continuous camera motion trajectories. The system renders several potentially sharp images from the current 3D Gaussian scene, averages these potentially sharp images over the exposure time to obtain a simulated blurred image, and calculates the absolute difference between this simulated blurred image and the blurred grayscale image captured by the actual color camera to obtain the photometric blur loss. Simultaneously, the system evenly divides the time interval between two adjacent blurred images into multiple sub-blocks and acquires the cumulative event signals output by the event camera within the corresponding time window. The system renders a latent image and calculates its logarithmic brightness change. It calculates the spatial mean and standard deviation of this logarithmic brightness change only in pixel regions where events are triggered, and then normalizes it. Similarly, for cumulative events, it calculates the spatial mean and standard deviation in pixel regions where events occur and normalizes them. Then, the system calculates the Euclidean distance and structural similarity between these two normalized distributions, and sums them with weights to obtain the threshold-free event loss. Furthermore, the system aggregates event streams within time intervals using an exponentially decaying window, generates an event boundary map based on the event timestamps and polarities, and uses a Sobel filter to extract edge information from the rendered image, calculating the cross-correlation between the two as the edge-aware loss. The system then sums the three loss functions with weights and iteratively optimizes the 3D Gaussian scene parameters and continuous camera poses using gradient descent to complete the first initialization.

[0030] Then, the system determines the depth information. Specifically, the system calls a pre-trained monocular depth estimation network to predict a dense depth map for each frame of blurred image. This depth map provides a coarse distance from each pixel in the scene to the camera. Based on the 3D Gaussian scene and continuous camera poses obtained from the first initialization, the system performs a second initialization in conjunction with the aforementioned depth information. In the second initialization, the system additionally introduces a contrast maximization variance loss. This loss is calculated as follows: the system assumes that the event stream satisfies optical flow constancy in the local spatiotemporal neighborhood. Each event is projected to a common reference time based on its occurrence time using the optical flow velocity, generating a distorted event image. The system divides the event stream into multiple sub-blocks, calculates the distorted event image of each sub-block and averages it, then calculates the variance of all pixel values ​​in the average image, using the negative value of this variance as the loss. By optimizing this loss, the system further adjusts the camera pose and the position of the 3D Gaussian kernel to maximize the alignment of event projections, thereby obtaining an accurate initial geometry and motion trajectory.

[0031] Finally, the system performs reconstruction based on the second initialization result to obtain the final reconstruction result. This reconstruction process includes alternating pose tracking and scene mapping, as well as subsequent color refinement steps, ultimately outputting a 3D Gaussian scene model with clear geometric details and realistic colors.

[0032] The beneficial effects of this embodiment include: by designing first and second initialization processes that do not rely on external prior poses, reliable scene geometry and continuous camera motion trajectories can be independently recovered from scratch in extreme motion blur conditions where traditional motion reconstruction methods completely fail; by introducing threshold-free event loss and edge perception loss, noise caused by dynamic threshold mismatch of event sensors is effectively eliminated, while the clarity of geometric edges is enhanced; by adding depth prior and contrast maximization variance loss, the accuracy and robustness of initialization are further improved, providing high-quality initial conditions for subsequent reconstruction.

[0033] In one possible implementation, the first initialization also includes: randomly initializing the camera pose, generating continuous camera poses using continuous spline curve interpolation, and jointly optimizing the photometric blur loss, the event loss without threshold constraints, and the edge perception loss.

[0034] In this context, camera pose refers to the camera's position coordinates and orientation angle in 3D space, typically represented as a translation vector and a rotation matrix. Random initialization involves setting all camera pose parameters to small, random values ​​close to the identity matrix without relying on any prior information. Continuous spline interpolation uses spline functions (such as B-splines) to smoothly interpolate camera poses at discrete time points, thus obtaining continuous motion trajectories at any given moment. Joint optimization refers to the process of simultaneously optimizing both the 3D Gaussian scene parameters and the camera pose parameters, causing multiple loss functions to decrease collectively.

[0035] The present invention will be further described below through a detailed embodiment: In the specific implementation of the aforementioned first initialization step, the system performs the following operations to complete the steps of the above embodiments.

[0036] The system first randomly initializes the camera pose corresponding to each blurred image frame. Specifically, for the total input sequence of frames, the system defines a six-DOF camera pose parameter for each frame, including three-dimensional translational components and three-dimensional rotational components (represented by axis angles or quaternions). The initial values ​​of all pose parameters are subjected to small random perturbations near zero. For example, the translational component is randomly selected within a range of ±0.1 unit length, and the rotational component is randomly selected within a range of ±0.05 radians, so that the initial pose exhibits a slightly random dispersion, but does not deviate from the approximate position.

[0037] Since camera motion is continuous, the system uses cubic uniform B-spline curves to interpolate the camera poses corresponding to discrete frames, generating a continuous and smooth camera motion trajectory. Specifically, the timestamp of each frame is used as a node of the spline. The randomly initialized pose transformation matrix is ​​decomposed into translation and rotation components, and B-spline fitting is performed on the translation vector and rotation quaternion, respectively. For any given moment, the system can calculate the camera pose using spline basis functions, thus supporting integral rendering within the exposure time and precise alignment of event timestamps.

[0038] During the joint optimization process, the system simultaneously maintains the 3D Gaussian scene parameters (including the center position, covariance matrix, opacity, and spherical harmonic coefficients of each Gaussian kernel) and the continuous camera pose parameters defined by the control points of the aforementioned B-spline curves. The system iteratively updates all optimizable variables using gradient descent. In each iteration, the system first renders several potentially sharp images based on the current continuous camera pose and the 3D Gaussian scene; then, it calculates the photometric blur loss, threshold-free event loss, and edge-aware loss in the same manner as in the above embodiment; finally, it weights and sums these three losses, calculates the gradient of the loss function with respect to the 3D Gaussian parameters and camera pose parameters, and updates the parameter values. After multiple iterations, the photometric blur loss gradually decreases, indicating that the rendered simulated blurred image tends to be consistent with the actual blurred image; the threshold-free event loss decreases, indicating that the brightness changes of the rendered image are aligned with the event flow statistical distribution; and the edge-aware loss decreases, indicating that the edge structure of the rendered image matches the event boundary map. This joint optimization process enables the system to simultaneously recover reliable scene geometry and accurate continuous camera motion trajectories without external pose priors.

[0039] The beneficial effects of this embodiment include: by random initialization and continuous spline curve interpolation, the system can obtain the initial camera pose without relying on any external motion recovery structure algorithm, and can represent a smooth and continuous camera motion trajectory, providing a foundation for blur rendering and event alignment within the exposure time; by jointly optimizing the 3D Gaussian parameters and camera pose parameters, the system achieves coordinated convergence of geometry and motion, avoiding the propagation and amplification of pose error to geometric reconstruction in traditional methods, and significantly improving the reconstruction robustness under severe motion blur conditions.

[0040] In one possible implementation, the threshold-free event loss is constructed as follows: the time interval between adjacent blurred frames is uniformly divided into multiple sub-blocks and the cumulative events are obtained; for the logarithmic intensity change and cumulative events of the image rendered from the 3D Gaussian scene, the spatial mean and standard deviation are calculated in the pixel regions where events exist to normalize the local statistical distribution; the distance and structural similarity between the normalized logarithmic intensity change and the normalized cumulative events are calculated to construct the threshold-free event loss.

[0041] Among them, threshold-free event loss refers to an event-supervised loss function that does not rely on a preset fixed contrast threshold. The time interval between adjacent blurred frames refers to the time span between two consecutively captured blurred images. A sub-block refers to multiple consecutive time periods into which the time interval is evenly divided. A cumulative event refers to the set of all event signals output by the event camera within the time window of each sub-block; each event includes pixel coordinates, a timestamp, and polarity (increase or decrease in brightness). The image rendered from a 3D Gaussian scene refers to a sharp image generated by rendering using 3D Gaussian sputtering technology under the current camera pose. Logarithmic intensity change refers to the logarithmic difference in pixel brightness within the time window of the rendered image. Spatial mean and standard deviation refer to the average value and dispersion calculated at the pixel location where an event is triggered. Normalization of local statistical distribution refers to subtracting the mean from the original data and dividing by the standard deviation to transform it into a standard distribution with zero mean and unit variance. Distance usually refers to Euclidean distance, and structural similarity is an indicator that measures the structural similarity between two images.

[0042] The present invention will be further described below through a detailed embodiment: In the aforementioned first initialization step, the specific calculation process for the loss of events without threshold constraints is as follows.

[0043] Assume the system has acquired two adjacent blurred image frames, with their corresponding timestamps being the start and end times, respectively. The time interval between these two frames is evenly divided into multiple sub-blocks. For each sub-block, the system reads all events generated within that time window from the event camera. These events constitute a cumulative event set, with each event recording the pixel coordinates of its occurrence location, the time of occurrence, and its polarity.

[0044] Based on the current continuous camera pose and the 3D Gaussian scene, the system renders two clear images corresponding to the start and end times of the aforementioned time window. Then, the system calculates the brightness variation map of these two clear images in logarithmic space, that is, subtracting the logarithmic brightness of the start time image from the logarithmic brightness of the end time image to obtain the logarithmic intensity variation distribution.

[0045] Because the contrast threshold of the event camera changes dynamically in the real physical environment, directly comparing the logarithmic intensity change with the absolute values ​​of cumulative events would introduce significant noise. Therefore, the system performs the following normalization process: First, the system identifies the pixel locations of all events in the cumulative event set and constructs a mask. Then, within the pixel region defined by this mask, the spatial mean and standard deviation of the logarithmic intensity change values, as well as the spatial mean and standard deviation of the cumulative event counts (or the polarity-weighted sum of cumulative events), are calculated. Next, the system subtracts the mean within the mask region from the logarithmic intensity change map and divides it by the standard deviation to obtain the normalized logarithmic intensity change distribution; similarly, the system subtracts the mean within the mask region from the cumulative event map and divides it by the standard deviation to obtain the normalized cumulative event distribution. These two normalized distributions have eliminated the influence of unknown contrast thresholds and illumination variations, making them statistically comparable.

[0046] Finally, the system calculates the Euclidean distance and structural similarity between the two normalized distributions. The Euclidean distance measures the absolute difference in pixel values, while the structural similarity measures the similarity in local patterns, contrast, and brightness. The system multiplies the Euclidean distance by a weighting coefficient and adds the complement of the structural similarity to obtain the final threshold-free event loss. During optimization, this loss function encourages the logarithmic intensity variation distribution of the rendered image to be as consistent as possible with the distribution of the real event flow, thereby achieving effective supervision of the 3D scene and camera pose without needing to know the true contrast threshold of the event camera.

[0047] The beneficial effects of this embodiment include: by performing local normalization processing on the logarithmic intensity change and cumulative events of the rendered image, the dependence on the static contrast threshold in the traditional method is completely bypassed, and the background noise introduced by the mismatch between the preset threshold and the physical threshold is eliminated; at the same time, by combining the dual measures of Euclidean distance and structural similarity, pixel-level consistency is guaranteed and local structural information is preserved, which significantly improves the robustness of event supervision and reconstruction accuracy.

[0048] In one possible implementation, the edge-aware loss is constructed by aggregating the event stream over time intervals using an exponentially decaying window to generate an event boundary map; applying an edge detection filter to the image and event boundary map rendered from a 3D Gaussian scene; and constructing the edge-aware loss by calculating the cross-correlation between the filtered rendered image and the filtered event boundary map.

[0049] In this context, the exponential decay window refers to a weighting function that decreases exponentially over time, assigning greater weight to recently occurring events. The aggregated event stream involves summing all events within the time window according to their timestamps and exponential weights, then assigning them to their corresponding pixel locations to generate an energy map. The event boundary map is the image obtained through this aggregation, where high-value regions correspond to locations where events occur densely, typically representing edges or motion boundaries with drastic brightness changes in a scene. Edge detection filters are operators used to extract high-frequency edge information from an image, such as the Sobel operator and the Canny operator. The filtered rendered image is the edge intensity map output after applying the edge detection filter to the rendered sharp image. Cross-correlation is a similarity measure that assesses the spatial alignment between two images, typically calculated as the sum or normalized cross-correlation of pixel-wise products.

[0050] The present invention will be further described below through a detailed embodiment: In the aforementioned first initialization step, the edge-aware loss is constructed in the following way.

[0051] The system first acquires the event stream within a time window, typically corresponding to the time interval between two adjacent blurred image frames. To generate an event boundary map, the system uses an exponentially decaying window for event aggregation. Specifically, for each event within the time window, the system calculates the time difference between its timestamp and the start time of the time window, then uses the absolute value of this time difference as the exponent of a negative exponential function to obtain a decay weight. This weight is characterized by events closer to the start time of the window having a weight closer to 1, while events farther from the start time have a weight that decreases exponentially towards 0. The system multiplies the polarity of each event by its decay weight and accumulates this sum at the pixel location where the event occurred. After traversing all events, the accumulated value at each pixel is the pixel value of the event boundary map. This event boundary map reflects the boundary locations of brightness changes in the scene, and due to the use of exponential decay, it emphasizes events that are temporally close, thus exhibiting good temporal locality.

[0052] Simultaneously, the system renders a clear image corresponding to the start time of the time window from a 3D Gaussian scene. Then, the system applies a Sobel edge detection filter to the rendered image. The Sobel operator contains two convolution kernels, one horizontal and one vertical, which calculate approximate gradients in the x and y directions, respectively. The square root of the sum of the squares of the gradients in both directions yields the edge intensity map. In this edge intensity map, high-value regions correspond to edge locations in the image where grayscale changes are drastic.

[0053] Ideally, the event boundary map should be spatially aligned with the edge map of the rendered image, because event triggering essentially involves changes in pixel brightness, and these changes often occur near image edges. To measure this alignment, the system calculates the cross-correlation between the two images. Specifically, the system multiplies the event boundary map and the rendered image edge map pixel by pixel and sums the results to obtain a scalar value. The larger this value, the better the spatial overlap between the two maps, meaning a higher degree of alignment between the event boundary and the rendered edge. Since the system aims to maximize this alignment, and the loss function is typically minimized, the negative value of this cross-correlation is used as the edge-aware loss.

[0054] During the optimization process, the system minimizes this loss, forcing the 3D Gaussian scene and camera pose to update in a direction that makes the edges of the rendered image more closely match the event boundary map, thereby effectively preventing geometric degradation caused by motion blur and maintaining the clear details of the reconstructed scene.

[0055] The beneficial effects of this embodiment include: generating an event boundary map by aggregating event streams through an exponential decay window, which can effectively utilize the high-frequency spatiotemporal information of events to extract the edge structure of the scene; and achieving strong constraints on the geometry of the 3D scene by cross-correlation alignment between the edges of the rendered image and the event boundary map, which significantly improves the reconstruction clarity under motion blur conditions and avoids the edge blurring problem caused by relying solely on brightness constraints in traditional methods.

[0056] In one possible implementation, depth information is obtained by predicting a dense depth map of the scene using a pre-trained monocular depth network; the second initialization includes joint optimization by incorporating a contrast-maximizing variance loss, which is constructed as follows: based on the condition that optical flow remains constant in the spatiotemporal neighborhood spanned by the event, the event is projected onto a reference time using the optical flow velocity to construct a distorted event image, and the contrast-maximizing variance loss is constructed by maximizing the variance of the average distorted event image of multiple event sub-blocks.

[0057] In this context, a pre-trained monocular depth network refers to a deep learning model pre-trained on a large-scale dataset, capable of estimating the depth value of each pixel from a single color image. A dense depth map is an image with the same resolution as the input image, where each pixel position is assigned a depth value. Contrast-maximizing variance loss is a loss function based on event alignment quality, optimizing camera pose and scene depth by maximizing the contrast (i.e., variance) of the distorted event image. Optical flow refers to the two-dimensional vector field of pixel motion on the image plane. The spatiotemporal neighborhood spanned by an event refers to a local spatiotemporal window in which the event occurs, assuming constant optical flow within this window. Projection refers to moving an event from its occurrence position to a reference position based on the optical flow velocity. A distorted event image is an image formed by projecting all events onto the same reference time and then counting the cumulative number of events (or polarity-weighted sum) at each pixel. An average distorted event image is the result of averaging the distorted event images generated for multiple event sub-blocks. Maximizing variance means optimizing variables to maximize the difference between different pixel values ​​in the average image, corresponding to the state of best event alignment and the clearest image.

[0058] The present invention will be further described below through a detailed embodiment: In the aforementioned method, the system first determines depth information. Specifically, the system inputs each blurred image frame into a pre-trained monocular depth estimation network. This network employs an encoder-decoder structure and is pre-trained on a depth dataset of tens of millions of pixels. The network outputs a dense depth map with the same resolution as the input image, where the depth value of each pixel represents the distance from the corresponding spatial point to the camera. Although monocular depth estimation suffers from scale blur and local errors, the relative depth relationships it provides are sufficient as geometric constraints.

[0059] In the second initialization process, in addition to using the photometric blurring loss, threshold-free event loss, and edge perception loss from the first initialization, the system also adds a contrast-maximizing variance loss for joint optimization. The specific construction method of this loss is as follows.

[0060] The system is based on the assumption that optical flow remains constant within the local spatiotemporal neighborhood of an event, projecting the event onto a reference time using optical flow velocity. Specifically, for the event stream between every two adjacent blurred frames, the system defines a reference time. For each event in the event stream, the time difference between its occurrence time and the reference time is multiplied by the optical flow velocity at the event's location to obtain a displacement vector; this displacement vector is then subtracted (or added) from the event's original pixel coordinates to obtain the event's projected coordinates at the reference time. The optical flow velocity itself is a variable to be optimized, related to the camera pose and the 3D Gaussian position.

[0061] After projecting all events in the manner described above, the system counts the cumulative number of projected events (or polarity-weighted sum) on each pixel in the image plane, generating a distorted event image. In this image, if the event projections are well aligned, a large number of events will accumulate at the scene edges, forming clear bright stripes, while the background area will be relatively uniform, resulting in a high-contrast overall image.

[0062] To enhance the robustness of the optimization, the system divides the entire event stream into multiple sub-blocks, each of which independently generates distorted event images in the manner described above. Then, it calculates the average image of these distorted event images. Finally, the system calculates the variance of this average image, i.e., the variance of all pixel values, and uses the negative value of this variance as the contrast-maximizing variance loss.

[0063] During optimization, the system simultaneously adjusts the camera pose, 3D Gaussian position, and optical flow velocity to minimize the loss (equivalent to maximizing the variance). When the event projection alignment is optimal, the distorted event image exhibits sharp peaks and troughs, reaching its maximum variance and minimum loss. This loss function provides the system with strong spatiotemporal geometric constraints, effectively utilizing the continuously encoded motion information in the event stream, resulting in more accurate second initialization results.

[0064] The beneficial effects of this embodiment include: by introducing dense depth priors provided by a pre-trained monocular depth network, a reliable geometric reference is provided for the second initialization, alleviating the depth blur problem caused by motion blur; by maximizing the variance loss through contrast, the spatiotemporal continuity and motion information in the event stream are explicitly utilized, and the event alignment quality is used as the optimization objective, which effectively improves the estimation accuracy of camera pose and 3D geometry, especially showing significant advantages in extreme motion scenarios with low frame rates and large displacements.

[0065] In one possible implementation, the reconstruction result is obtained based on the second initialization result, including: alternatingly executing pose tracking and scene mapping, wherein the 3D Gaussian scene is fixed in the tracking phase to optimize the camera pose, and the camera pose is fixed in the mapping phase to update the 3D Gaussian properties; during the alternating execution, sliding window bundle adjustment is also included: within the most recent predetermined number of time blocks, the continuous camera pose and 3D Gaussian scene parameters are simultaneously used as variable parameters for synchronous joint optimization.

[0066] Pose tracking refers to the process of fitting current observation data by optimizing camera pose parameters using a pre-built 3D Gaussian scene. Scene mapping refers to the process of optimizing 3D Gaussian scene parameters to better match all observation data given the known camera pose. Alternating execution means that these two processes are iterated cyclically, optimizing only one type of parameter while keeping the other type fixed each time. Sliding window bundle adjustment is a global optimization technique that simultaneously optimizes camera pose and scene geometry within a local time window to eliminate accumulated errors. A predetermined number of time blocks refers to a pre-set window size, such as the most recent consecutive time segments. Continuous camera poses, as described in the previous embodiments, are generated by continuous spline curve interpolation. 3D Gaussian scene parameters include the center position of the Gaussian kernel, covariance matrix, opacity, etc.

[0067] The present invention will be further described below through a detailed embodiment.

[0068] After completing the second initialization, the system obtained relatively reliable initial parameters for the 3D Gaussian scene and continuous camera pose trajectories defined by B-spline curves. To further improve reconstruction quality and adapt to long input sequences, the system entered a phase of alternating pose tracking and scene mapping.

[0069] During the tracking phase, the system fixes all current 3D Gaussian properties (i.e., the center, shape, opacity, and spherical harmonics of the Gaussian kernel remain unchanged), optimizing only the parameters of the continuous camera pose trajectory. Specifically, for each newly arrived blurred image and the event stream within the corresponding time window, the system calculates photometric blur loss, unconstrained event loss, and edge-aware loss; the weighted sum of these losses serves as the loss function for the tracking phase. The system adjusts the control points of the B-spline curve using gradient descent, enabling the continuous camera pose trajectory to better fit the current observation. Since the 3D Gaussian scene is fixed, the computational load of the tracking process is relatively small, allowing for real-time response to camera motion.

[0070] During the mapping phase, the system retains the camera pose trajectory obtained from the previous optimization step and instead optimizes all parameters of the 3D Gaussian scene. The system utilizes all previously processed image and event data to recalculate the joint loss (which also includes photometric blur loss, unconstrained event loss, and edge-aware loss, and optionally incorporates depth loss and contrast-maximizing variance loss), and then updates the position, covariance, opacity, and other attributes of the 3D Gaussian kernel. The mapping phase typically requires more iterations to fully integrate multi-view information.

[0071] The above tracking and mapping phases are executed alternately: after each tracking optimization, a mapping optimization is performed, and then the next tracking is performed, and so on.

[0072] During alternating execution, incremental processing inevitably introduces error accumulation, leading to drift in long-sequence reconstruction. To address this, the system introduces a sliding window bundle adjustment mechanism. Specifically, the system maintains a fixed-size sliding window covering the most recent time blocks. Each time block corresponds to the interval between a pair of adjacent blurred frames. After each tracking and mapping alternation, the system triggers sliding window bundle adjustment. During this adjustment process, the system treats all continuous camera pose parameters (control points defined by B-spline curves) and all 3D Gaussian scene parameters within the window as simultaneously optimizable variables, no longer fixing any single category. The system uses a comprehensive loss function (containing all the aforementioned loss terms) for joint optimization. After optimization, earlier parameters outside the window are frozen, and the window slides forward to include new time blocks.

[0073] The sliding window bundle adjustment mechanism effectively suppresses drift, ensures global geometric consistency, and makes the reconstruction results coherent throughout the long sequence.

[0074] The beneficial effects of this embodiment include: by alternately executing pose tracking and scene mapping, an incremental processing flow of motion estimation and reconstruction is realized, which takes into account both real-time performance and accuracy; by introducing sliding window bundle adjustment, the camera trajectory and scene geometry are jointly optimized within a local window, which effectively eliminates the cumulative drift in long sequence reconstruction and ensures global consistency; this mechanism is naturally compatible with continuous camera pose representation, forming a complete incremental reconstruction system.

[0075] One possible implementation also includes color rendering: after obtaining the reconstruction results, the camera pose and 3D Gaussian properties are frozen, and the spherical harmonic coefficients of the 3D Gaussian are optimized using color-blurred frames.

[0076] Color rendering refers to the process of adding realistic color information to the reconstructed 3D Gaussian scene. Freezing refers to setting the camera pose and 3D Gaussian geometric properties (including position, shape, and opacity) to be non-optimizable and kept fixed. A color-blurred frame refers to a color image with motion blur captured by a traditional color camera. Spherical harmonics are a parameterization method used in 3D Gaussian sputtering to represent color variations with viewing angle, fitting view-dependent colors using the coefficients of a set of basis functions. Optimizing spherical harmonics means updating only color-related parameters without changing the geometry.

[0077] The present invention will be further described below through a detailed embodiment, which refers to the steps of the foregoing embodiments.

[0078] After completing alternating tracking and mapping, as well as sliding window bundle adjustment, the system has obtained a 3D Gaussian scene with accurate geometry, including the spatial location, shape (determined by the covariance matrix), and opacity of each Gaussian kernel. Simultaneously, the system has also obtained the accurate camera pose trajectories for all frames. However, because the system primarily relies on brightness variation signals provided by the event stream and threshold-free event loss during the tracking and mapping phases, and the event camera itself does not output color information, the spherical harmonic coefficients of the current 3D Gaussian may be inaccurate or missing, resulting in a rendered scene lacking realistic colors.

[0079] To address this issue, the system performs a color rendering phase. During this phase, all camera pose parameters are frozen, meaning adjustment of the B-spline control points is no longer permitted. Simultaneously, all geometric properties of the 3D Gaussian kernel are frozen, including center position, covariance matrix, and opacity; these parameters are fixed to the optimal values ​​obtained during the tracking and mapping phases. The system only sets the spherical harmonic coefficients of the 3D Gaussian kernel as optimizable variables.

[0080] The system then uses the original color-blurred frames as supervision signals. Specifically, for each color-blurred image frame, the system renders a simulated blurred color image based on the frozen camera pose and 3D Gaussian geometric properties. Since the spherical harmonic coefficients are the only variable, the rendered image color changes with adjustments to the spherical harmonic coefficients. The system calculates the absolute difference between the simulated blurred color image and the actually captured color-blurred image as the color loss function. The system optimizes the spherical harmonic coefficients using gradient descent to minimize the color loss.

[0081] Since the geometry and camera pose are precisely fixed, the color optimization process does not destroy existing sharp geometric details. The luminance priors provided by the event stream in the pre-processing stage (i.e., the luminance or grayscale values ​​of each Gaussian kernel) are also preserved, and color optimization only adds color information based on these. After multiple iterations, the spherical harmonic coefficients converge, enabling the system to render a colored 3D scene with both clear geometry and high-fidelity colors. This color rendering process can be performed on the entire sequence or on selected keyframe subsets to improve efficiency.

[0082] The beneficial effects of this embodiment include: by explicitly decoupling geometry optimization from color optimization, freezing geometry and pose parameters during the color rendering stage, and optimizing only the spherical harmonic coefficients, the sharp geometric structure and brightness priors provided by the event flow are perfectly preserved, avoiding the destruction of high-precision geometry during the color restoration process; and by using the original color blurred frame as a supervision signal, multi-view data can be restored without the need for additional clear images. Figure 1 The realistic texture effectively reduces color degradation caused by motion blur, ultimately outputting a high-fidelity, anti-blur 3D reconstruction result.

[0083] In one overall embodiment, the technical solution of this application is specifically described as follows: 3D Gaussian sputtering technology represents a scene using a set of learnable 3D Gaussian kernels and, ideally, optimizes the network by minimizing the photometric error between the rendered image and the sharp captured image. However, in high-speed motion scenes, the image captured by a traditional camera is actually the physical time integral of the scene's radiance over the entire exposure time. This results in the rendered sharp image failing to match the actual observation that produces motion blur, rendering conventional pure photometric supervision methods ineffective. Therefore, to overcome the limitations of traditional scene reconstruction methods based on 3D Gaussian sputtering, this invention employs an asynchronous event camera. The basic physical principle of an event camera is that when the logarithmic change in brightness at a pixel exceeds its underlying physical contrast threshold, the sensor triggers and outputs a polarized event signal. Existing event-assisted 3DGS reconstruction methods typically attempt to supervise the network by directly calculating the absolute difference between the "brightness change in the rendered image" and the "cumulative event amount." However, this traditional physical model has a fatal flaw: it assumes that the contrast threshold is a static, known constant; but in real physical environments, this threshold fluctuates dynamically and drastically with ambient lighting and other conditions. The mismatch between the preset fixed threshold and the actual physical threshold will introduce severe background noise into the reconstruction system, directly leading to rendering quality degradation. The specific implementation is as follows: A coarse-to-fine initialization process is implemented to establish an initial 3D Gaussian scene and continuous camera trajectories. To address the issue that severe motion blur prevents traditional SfM methods from providing accurate initial poses, an initial 3DGS scene is constructed from scratch. In the coarse stage, the 3DGS scene is initialized as a random point cloud within the boundary volume, and the initialized camera pose is randomly initialized near the identity matrix. During optimization, continuous camera poses are obtained using B-spline interpolation. Three loss functions are used in this stage: as well as Among them, photometric fuzziness constraint Averaging the N potential images to create a blurred frame and compared with the grayscale image of the actual blurred image. :

[0084] Without threshold event constraints In the middle, adjacent blurred frames and The time interval between events is evenly divided into M sub-blocks, and the cumulative events are acquired through an event camera. The logarithmic intensity change ΔL_(i,j→k) and cumulative events E_(i,j→k) of the latent image are normalized using the following formulas:

[0085]

[0086] in, and Let represent the spatial mean and standard deviation, calculated only for pixels where an event exists. The no-threshold event loss is defined as the distance and structural similarity (SSIM) between normalized distributions, where λ is the weighting coefficient.

[0087] Edge sensing constraints In the middle, the point is used to decay exponentially. In time interval Internal aggregation of event flows generates an event boundary graph. :

[0088] Sobel filter For potential images Event Boundary Diagram And by calculating pixels Cross-correlation is used to assess structural clarity and alignment:

[0089] In summary, the loss function in the coarse initialization phase is defined as:

[0090] in and The weights are used to introduce depth priors for scene prediction from a pre-trained monocular depth network during the fine initialization phase. Contrast-maximizing variance loss is then introduced at this stage. Assuming the optical flow remains constant within the spatiotemporal neighborhood spanned by the event, the optical flow velocity... Current time The location of The incident Projection (Warp) to reference time , For projection function, Indicates the projection point:

[0091] Construct the average projected image IWE as , Indicates the polarity of the event. Indicates the location of the event after compensation. Represents the original coordinates of the event. Indicates the number of events:

[0092] By maximizing Average IWE of each event sub-block The variance is used to optimize camera pose and Gaussian position:

[0093] In summary, the loss function during the fine initialization phase is:

[0094] in , All are weighted coefficients, and pose tracking and scene mapping are performed alternately based on the geometric depth information provided by the coarse-to-fine initialization. In the tracking phase, the 3DGS scene is fixed to optimize the camera pose; in the mapping phase, the trajectory is fixed to update the 3D Gaussian properties.

[0095] A sliding window bundle adjustment method is introduced to achieve global geometric consistency optimization. In the incremental processing of long sequences (alternating tracking and mapping), to eliminate the error accumulation caused by local pose estimation and ensure the global geometric consistency of the scene, this invention introduces a local sliding window optimization mechanism. Specifically: In recent... Within a time block, the system removes the alternating freeze restriction on camera trajectory and 3D Gaussian properties, and simultaneously uses continuous camera trajectory and 3D Gaussian scene parameters as variable parameters for synchronous joint optimization (i.e., bundle adjustment).

[0096] A two-stage color refinement process is performed to restore realistic scene colors. Since the tracking and mapping stages primarily rely on intensity changes in the event stream, the resulting reconstruction lacks accurate color information. To restore high-fidelity textures, this method utilizes color-blurred frames for two-stage color refinement. In this stage, all camera poses and 3DGS geometric properties are completely frozen, and only the spherical harmonic (SH) coefficients of the 3D Gaussian are optimized. The color loss function is defined as:

[0097] in This means averaging N potential images to synthesize the i-th blurred frame. This represents the i-th blurred frame with color. This strategy allows the model to focus on recovering accurate scene colors while perfectly preserving the sharp structure and brightness priors provided by the event flow during the geometry optimization phase, effectively mitigating color degradation caused by motion blur.

[0098] For scene reconstruction with motion blur, see [link to relevant documentation]. Figure 2 and Figure 3 , Figure 2 The quality of image reconstruction under blurred conditions is demonstrated, where GT stands for Ground Truth, which is the real image corresponding to the reconstructed image. The proposed solution can still obtain clear reconstruction results even when the RGB image is blurred, and the reconstruction results are close to the GT real image. Figure 3 This demonstrates the effectiveness of the two-stage initialization method. Through coarse initialization and fine initialization stages—namely, the first initialization (coarse initialization) and the second initialization (fine initialization) described in the above embodiment—the method accurately restores the scene's precise geometry (accurate image and depth). After coarse initialization, the approximate depth and shape of the scene are obtained. After fine initialization, the image becomes even clearer.

[0099] This application addresses the weakness of existing 3D Gaussian reconstruction and neural radiation field methods, which heavily rely on external SfM methods such as COLMAP for accurate initial pose. To address this, this application proposes a bootstrap initialization technique that starts entirely from scratch. This technique initializes the scene as a random point cloud and the pose as an identity matrix. In the coarse stage, it jointly optimizes the photometric and event losses and uses B-spline interpolation to ensure continuous trajectory. Subsequently, in the fine stage, it introduces monocular depth priors and contrast-maximizing variance loss. This approach enables the system to independently and robustly recover reliable initial 3D geometry and continuous camera motion trajectories even under extreme motion blur and when traditional SfM methods completely fail.

[0100] Existing event-assisted reconstruction methods generally employ predefined static contrast thresholds, which are severely inconsistent with the dynamically changing contrast thresholds in real physical environments, thus introducing a large amount of inherent noise into the system. This application abandons the traditional approach of directly calculating the absolute difference between the logarithmic brightness change and the cumulative event flow of the rendered image, and innovatively proposes a threshold-free constraint technique. This method normalizes the logarithmic intensity change and the cumulative event flow of the rendered image by extracting the spatial mean and standard deviation of pixel regions containing only events. By aligning these two locally normalized statistical distributions in the loss function, this method bypasses the unknown dynamic contrast threshold parameter, fundamentally eliminating noise caused by threshold mismatch and achieving highly robust event supervision.

[0101] To address the issue of large displacements in discrete low-frame-rate blurred images, where brightness constraints alone cannot guarantee spatial geometric accuracy, this application proposes an explicit extraction of spatiotemporal information from the event stream. This technique incorporates two strong constraints: first, a contrast maximization constraint, which uses optical flow to project distorted events onto a reference frame to generate distorted event images and maximizes their variance, thereby jointly optimizing camera pose and 3D Gaussian position; second, an edge-aware constraint, which utilizes an exponentially decaying window to generate an event boundary map and aligns it with the Sobel high-frequency edge map of the rendered image through cross-correlation. This approach effectively prevents geometric degradation under motion blur and achieves accurate restoration of sharp spatial details.

[0102] Since event cameras are essentially sensors sensitive to changes in brightness and do not contain color information, 3DGS results reconstructed based on event streams often lack realistic color information. This application proposes an innovative strategy that explicitly decouples geometry optimization from appearance optimization. During the color refinement stage, all camera poses and 3DGS geometric attributes are completely frozen, and only the spherical harmonic coefficients of the 3D Gaussian are supervised and optimized using blurred color frames. This method effectively avoids the destruction of existing high-precision geometric structures during the color restoration process, perfectly preserving the sharp structure and brightness priors provided by the event stream while achieving consistency in color across multiple views and high-fidelity texture restoration.

[0103] Corresponding to the above method embodiments, this specification also provides embodiments of an anti-fuzzy 3D Gaussian reconstruction device. Figure 4 A schematic diagram of an anti-blurring three-dimensional Gaussian reconstruction device according to one embodiment of this specification is shown. Figure 4 As shown, the device includes: The first initialization module 401 is configured to determine a random point cloud based on a 3D Gaussian scene, perform a first initialization on the random point cloud, and determine a first initialization result; wherein, the first initialization includes photometric blur loss, threshold-free event loss, and edge perception loss; The second initialization module 402 is configured to determine depth information and perform a second initialization based on the first initialization result and the depth information to determine the second initialization result. The scene reconstruction module 403 is configured to perform reconstruction based on the second initialization result to obtain the reconstruction result.

[0104] In one possible implementation, the first initialization also includes: randomly initializing the camera pose, generating continuous camera poses using continuous spline curve interpolation, and jointly optimizing the photometric blur loss, the event loss without threshold constraints, and the edge perception loss.

[0105] In one possible implementation, the threshold-free event loss is constructed as follows: the time interval between adjacent blurred frames is uniformly divided into multiple sub-blocks and the cumulative events are obtained; for the logarithmic intensity change and cumulative events of the image rendered from the 3D Gaussian scene, the spatial mean and standard deviation are calculated in the pixel regions where events exist to normalize the local statistical distribution; the distance and structural similarity between the normalized logarithmic intensity change and the normalized cumulative events are calculated to construct the threshold-free event loss.

[0106] In one possible implementation, the edge-aware loss is constructed by aggregating the event stream over time intervals using an exponentially decaying window to generate an event boundary map; applying an edge detection filter to the image and event boundary map rendered from a 3D Gaussian scene; and constructing the edge-aware loss by calculating the cross-correlation between the filtered rendered image and the filtered event boundary map.

[0107] In one possible implementation, depth information is obtained by predicting a dense depth map of the scene using a pre-trained monocular depth network; the second initialization includes joint optimization by incorporating a contrast-maximizing variance loss, which is constructed as follows: based on the condition that optical flow remains constant in the spatiotemporal neighborhood spanned by the event, the event is projected onto a reference time using the optical flow velocity to construct a distorted event image, and the contrast-maximizing variance loss is constructed by maximizing the variance of the average distorted event image of multiple event sub-blocks.

[0108] In one possible implementation, the reconstruction result is obtained based on the second initialization result, including: alternatingly executing pose tracking and scene mapping, wherein the 3D Gaussian scene is fixed in the tracking phase to optimize the camera pose, and the camera pose is fixed in the mapping phase to update the 3D Gaussian properties; during the alternating execution, sliding window bundle adjustment is also included: within the most recent predetermined number of time blocks, the continuous camera pose and the 3D Gaussian scene parameters are simultaneously used as variable parameters for synchronous joint optimization.

[0109] One possible implementation also includes color rendering: after obtaining the reconstruction results, the camera pose and 3D Gaussian properties are frozen, and the spherical harmonic coefficients of the 3D Gaussian are optimized using color-blurred frames.

[0110] The above is a schematic scheme of an anti-fuzzy 3D Gaussian reconstruction device according to this embodiment. It should be noted that the technical solution of this anti-fuzzy 3D Gaussian reconstruction device and the technical solution of the above-described anti-fuzzy 3D Gaussian reconstruction method belong to the same concept. For details not described in detail in the technical solution of the anti-fuzzy 3D Gaussian reconstruction device, please refer to the description of the technical solution of the above-described anti-fuzzy 3D Gaussian reconstruction method.

[0111] Figure 5A structural block diagram of a computing device 500 according to one embodiment of this specification is shown. The components of the computing device 500 include, but are not limited to, a memory 510 and a processor 520. The processor 520 is connected to the memory 510 via a bus 530, and a database 550 is used to store data.

[0112] The computing device 500 also includes an access device 540, which enables the computing device 500 to communicate via one or more networks 560. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 540 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0113] In one embodiment of this specification, the above-described components of the computing device 500 and Figure 5 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 5 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0114] The computing device 500 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 500 can also be a mobile or stationary server.

[0115] The processor 520 executes the following computer-executable instructions, which, when executed by the processor, implement the steps of the aforementioned anti-fuzzy 3D Gaussian reconstruction method. The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the aforementioned anti-fuzzy 3D Gaussian reconstruction method belong to the same concept. Details not described in detail in the technical solution of the computing device can be found in the description of the technical solution of the aforementioned anti-fuzzy 3D Gaussian reconstruction method.

[0116] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the above-described anti-fuzzy 3D Gaussian reconstruction method.

[0117] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the technical solution of the above-described anti-fuzzy three-dimensional Gaussian reconstruction method. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the above-described anti-fuzzy three-dimensional Gaussian reconstruction method.

[0118] An embodiment of this specification also provides a computer program, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the above-described anti-fuzzy 3D Gaussian reconstruction method.

[0119] The above is an illustrative example of a computer program according to this embodiment. It should be noted that the technical solution of this computer program belongs to the same concept as the aforementioned anti-fuzzy 3D Gaussian reconstruction method. Details not described in detail in the computer program's technical solution can be found in the description of the aforementioned anti-fuzzy 3D Gaussian reconstruction method.

[0120] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0121] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0122] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0123] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0124] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A method for anti-fuzzy 3D Gaussian reconstruction, characterized in that, include: Random point clouds are determined based on a 3D Gaussian scene, and the random point clouds are first initialized to determine the first initialization result; wherein, the first initialization includes photometric blurring loss, threshold-free event loss and edge perception loss; Determine the depth information, and perform a second initialization based on the first initialization result and the depth information to determine the second initialization result; The reconstruction result is obtained by reconstructing based on the second initialization result.

2. The method according to claim 1, characterized in that, The first initialization further includes: randomly initializing the camera pose, generating continuous camera poses using continuous spline curve interpolation, and jointly optimizing the photometric blur loss, the threshold-free event loss, and the edge perception loss.

3. The method according to claim 1, characterized in that, The threshold-free event loss is constructed as follows: the time interval between adjacent blurred frames is uniformly divided into multiple sub-blocks and cumulative events are obtained; for the logarithmic intensity change of the image rendered from the 3D Gaussian scene and the cumulative events, the spatial mean and standard deviation are calculated in the pixel regions where events exist to normalize the local statistical distribution; the distance and structural similarity between the normalized logarithmic intensity change and the normalized cumulative events are calculated to construct the threshold-free event loss.

4. The method according to claim 1, characterized in that, The edge-aware loss is constructed by aggregating event streams within a time interval using an exponentially decaying window to generate an event boundary map; applying an edge detection filter to the image rendered from the 3D Gaussian scene and the event boundary map; and constructing the edge-aware loss by calculating the cross-correlation between the filtered rendered image and the filtered event boundary map.

5. The method according to claim 1, characterized in that, The depth information is obtained by predicting the dense depth map of the scene through a pre-trained monocular depth network; the second initialization includes joint optimization by adding a contrast-maximizing variance loss, which is constructed in the following way: based on the condition that the optical flow remains constant in the spatiotemporal neighborhood spanned by the event, the event is projected to a reference time through the optical flow velocity to construct a distorted event image, and the contrast-maximizing variance loss is constructed by maximizing the variance of the average distorted event image of multiple event sub-blocks.

6. The method according to claim 2, characterized in that, The reconstruction result is obtained based on the second initialization result, including: alternatingly executing pose tracking and scene mapping, wherein, in the tracking phase, the 3D Gaussian scene is fixed to optimize the camera pose, and in the mapping phase, the camera pose is fixed to update the 3D Gaussian properties; during the alternating execution, it also includes sliding window bundle adjustment: within the most recent predetermined number of time blocks, the continuous camera pose and the 3D Gaussian scene parameters are simultaneously used as variable parameters for synchronous joint optimization.

7. The method according to claim 6, characterized in that, It also includes color rendering: after obtaining the reconstruction results, the camera pose and 3D Gaussian properties are frozen, and the spherical harmonic coefficients of the 3D Gaussian are optimized using color blurred frames.

8. An anti-fuzzy three-dimensional Gaussian reconstruction device, characterized in that, include: The first initialization module is configured to determine a random point cloud based on a 3D Gaussian scene, perform a first initialization on the random point cloud, and determine a first initialization result; wherein, the first initialization includes photometric blurring loss, no threshold constraint event loss, and edge perception loss; The second initialization module is configured to determine depth information and perform a second initialization based on the first initialization result and the depth information to determine the second initialization result. The scene reconstruction module is configured to reconstruct the scene based on the second initialization result to obtain the reconstruction result.

9. A computing device, characterized in that, include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the anti-fuzzy three-dimensional Gaussian reconstruction method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the anti-fuzzy three-dimensional Gaussian reconstruction method according to any one of claims 1 to 7.

Images