Crowdsourced street view video 3d reconstruction enhancement method based on semantic evolution loop

Abstract

The application relates to a crowd-sourced street view video three-dimensional reconstruction enhancement method based on a semantic evolution loop. The method comprises the following steps: obtaining an initial three-dimensional reconstruction result, identifying a rendering artifact area in the initial three-dimensional reconstruction result and generating an artifact mask; performing extension processing on the artifact mask to obtain a repair mask; combining the repair mask and a semantic repair instruction to perform semantic repair on the artifact area, thereby obtaining a three-dimensional reconstruction intermediate result; constructing a tubular constraint space based on an original collection track, generating a virtual camera track covering an unobserved area, performing perspective expansion on the intermediate result, and obtaining a reconstruction view under an expanded perspective; calculating a geometric compatibility index of the reconstruction view and a corresponding re-projection rendering view, and if the consistency requirement is met, the reconstruction view is used as supplementary training data to feed back optimization, the three-dimensional reconstruction result is iteratively updated, and the method is stopped until the stop condition is met. The method can improve the fidelity and geometric consistency of the three-dimensional reconstruction result of fragmented crowd-sourced data.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method for enhancing 3D reconstruction of crowdsourced street view videos based on semantic evolutionary loops. Background Technology

[0002] With the rapid development of urban digitalization and computer vision technologies, neural rendering technologies such as Neural Radiation Field (NeRF) and 3D Gaussian Splashing (3DGS) have emerged. These technologies, with their precise scene modeling capabilities, have enabled the transformation from street view images to immersive 3D representations, becoming core technologies for building urban digital twins and supporting 3D Geographic Information System (GIS) applications. This has also made urban 3D reconstruction based on street view data a research and application hotspot. Against this backdrop, crowdsourced street view videos, due to their low acquisition cost, high update frequency, and wide coverage, have become an important supplement to professional surveying and mapping data, providing a high-timeliness, massive data source for urban remote sensing and 3D reconstruction, and gradually replacing some traditional professional surveying and mapping methods in relevant scenarios.

[0003] However, current methods for urban 3D reconstruction using crowdsourced street view videos combined with neural rendering technology still suffer from several technical shortcomings that urgently need to be addressed. Crowdsourced street view videos are limited by acquisition equipment, environment, and trajectory, generally exhibiting sparse trajectories and strong unidirectionality. This results in insufficient angular coverage of the urban scene and a lack of multi-view constraints, leading to severe rendering artifacts in the neural rendering model during reconstruction, such as geometric floating objects, blurred building facade textures, and structural distortions. Furthermore, existing neural reconstruction methods often employ static data augmentation strategies, failing to actively explore unobserved areas to compensate for the sparsity of crowdsourced data. Some reconstruction methods combining generative models also lack effective geometric constraints and feedback mechanisms. This approach can easily introduce non-physical generation illusions, causing 3D manifold collapse and making it difficult to guarantee the geometric consistency and semantic correctness of the reconstruction results. In addition, existing technologies lack precise localization and targeted repair methods for artifacts generated during the reconstruction process, often resorting to global re-rendering or simple image spatial filtering. This not only destroys the effective geospatial details in the original crowdsourced data but also fails to fundamentally solve the fidelity problem of 3D reconstruction. Ultimately, this results in a huge fidelity-consistency gap between 3D reconstruction results based on crowdsourced street view videos and professional-grade surveying and mapping data, making it difficult to use as a reliable geospatial asset in high-fidelity GIS-related scenarios. This severely limits the large-scale application of crowdsourced street view videos in urban digital construction. Summary of the Invention

[0004] Therefore, it is necessary to provide a crowdsourced street view video 3D reconstruction enhancement method based on semantic evolution loop to address the above-mentioned technical problems.

[0005] A method for 3D reconstruction and enhancement of crowdsourced street view videos based on semantic evolutionary loops, the method comprising: Obtain the initial 3D reconstruction result corresponding to the crowdsourced street view video, identify the rendering artifact regions in the initial 3D reconstruction result and generate an artifact mask; The artifact mask is expanded to obtain a repair mask containing the artifact region and surrounding context information; Based on the repair mask and the pre-set semantic repair instructions, semantic repair is performed on the artifact regions in the initial 3D reconstruction result to obtain intermediate 3D reconstruction results. Based on the original acquisition trajectory of the crowdsourced street view video, a local orthogonal basis is constructed at each original camera pose. A tubular constraint space is constructed to enclose the original acquisition trajectory by dynamically adjusting the curriculum evolution factor. A virtual camera trajectory covering the unobserved area is generated in the tubular constraint space. The virtual camera trajectory is used to expand the perspective of the intermediate results of the 3D reconstruction to obtain the reconstructed view under the expanded perspective. Calculate the geometric compatibility index between the reconstructed view under the extended view and the corresponding reprojected rendered view, and determine whether the reconstructed view under the extended view meets the three-dimensional manifold consistency requirements based on the geometric compatibility index; If the three-dimensional manifold consistency requirement is met, the reconstructed view under the extended perspective is fed back as supplementary training data to the three-dimensional reconstruction optimization process, and the three-dimensional reconstruction results are iteratively updated until the preset iteration stop condition is met, at which point the iteration stops and the current three-dimensional reconstruction result is output.

[0006] The aforementioned 3D reconstruction enhancement method based on semantic evolutionary loops for crowdsourced street view videos acquires the initial 3D reconstruction results from the crowdsourced street view videos and identifies rendering artifact regions to generate artifact masks. This allows for precise localization of defects such as geometric floating objects and texture distortion, providing clear targets for targeted repair. The artifact masks are then expanded to obtain repair masks, incorporating surrounding contextual information to ensure semantic coherence between the repaired area and the environment. Semantic repair based on the repair masks and semantic repair instructions eliminates artifacts in the initial reconstruction, improving the fidelity of intermediate 3D reconstruction results. A local orthogonal basis and tubular constraint space are constructed based on the original acquisition trajectory, and a virtual camera trajectory is generated by dynamically adjusting the curriculum evolutionary factor. This fills in observation blind spots and enhances scene angle coverage. Calculating geometric compatibility indicators and selecting extended views that conform to 3D manifold consistency eliminates non-physical illusions, ensuring the geometric integrity of the reconstruction. Iterative updates using effective extended views as supplementary training data continuously optimize reconstruction quality until the stopping condition is met. The embodiments of the present invention can effectively make up for the limitations of sparse trajectory and single perspective in crowdsourced street view videos, significantly reduce the proportion of rendering artifacts, improve the fidelity and geometric consistency of 3D reconstruction results, and narrow the gap between fragmented crowdsourced data and professional-grade 3D GIS data. Attached Figure Description

[0007] Figure 1 This is a flowchart illustrating a crowdsourced street view video 3D reconstruction enhancement method based on semantic evolutionary loops in one embodiment. Figure 2 This is a schematic diagram of the SAGE-Loop framework based on semantic evolutionary loops and the evolution of virtual camera trajectory in one embodiment, wherein, Figure 2 (a) is the flowchart of the SAGE-Loop framework. Figure 2 (b) is a schematic diagram of the virtual camera's spiral trajectory and offset range; Figure 3 This is a schematic diagram of a Progressive Virtual Trajectory Evolution (PVTE) mechanism in one embodiment, wherein, Figure 3 (a) is a schematic diagram of the construction of a local orthonormal basis. Figure 3 (b) is a schematic diagram of the virtual camera displacement. Figure 3 (c) is a schematic diagram of orientation realignment based on gaze constraints. Figure 3 (d) is a schematic diagram of the curriculum evolution from near-field fine-tuning to omnidirectional exploration; Figure 4 This is a schematic diagram illustrating an example of R-artifact annotation samples for the CS3D-Continuous dataset in one embodiment; Figure 5 This is a schematic diagram illustrating the trend of artifact pixel rate (APR) during training in one embodiment; Figure 6 This is a qualitative comparison diagram of 3D reconstruction results on the CS3D-Continuous dataset in one embodiment, wherein, Figure 6 (a) is a schematic diagram comparing the texture of building facades and the cleanliness of roads using different methods from a conventional perspective. Figure 6 (b) is a schematic diagram comparing the new perspective synthesis effect of the method of the present invention in extreme off-axis camera offset scenarios. Detailed Implementation

[0008] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0009] In one embodiment, such as Figure 1 As shown, a method for 3D reconstruction enhancement of crowdsourced street view videos based on semantic evolutionary loops is provided, including the following steps: Step 102: Obtain the initial 3D reconstruction result corresponding to the crowdsourced street view video, identify the rendering artifact regions in the initial 3D reconstruction result and generate artifact masks.

[0010] Crowdsourced street view video refers to street view video data captured and uploaded by a large number of non-professional collectors using various devices (such as dashcams and mobile phones). The initial 3D reconstruction result refers to the 3D scene representation obtained after preliminary processing of the crowdsourced street view video using neural rendering techniques such as Neural Radiation Field (NeRF) and 3D Gaussian Splashing (3DGS). An artifact mask is a binary image used to accurately mark the location of rendering artifact regions, where a pixel value of 1 indicates the presence of artifacts, and a pixel value of 0 indicates the absence of artifacts.

[0011] Step 104: Expand the artifact mask to obtain a repair mask containing the artifact region and surrounding context information.

[0012] Artifact mask expansion refers to the process of performing morphological operations (such as dilation) on the initially generated artifact mask. Surrounding context information refers to the effective scene content surrounding the artifact region, including adjacent building textures, road structures, environmental elements, etc. This information is crucial for maintaining the semantic coherence of the restored scene. The restored mask is the mask obtained after expansion processing, and its coverage includes the original artifact region and a certain range of surrounding context.

[0013] It is understandable that by extending the artifact mask to include surrounding contextual information, sufficient scene background reference can be provided for subsequent semantic repair, ensuring that the repaired area is naturally connected with the surrounding environment, avoiding problems such as abrupt repair edges and semantic contradictions, and helping to improve the semantic consistency and visual realism of the repair results.

[0014] Step 106: Based on the repair mask and pre-set semantic repair instructions, perform semantic repair on the artifact regions in the initial 3D reconstruction result to obtain the intermediate 3D reconstruction result.

[0015] Semantic inpainting instructions are pre-defined text commands designed to guide the inpainting model in understanding the inpainting target. They aim to make the model clearly understand that artifacts need to be removed and background content consistent with the logic of the urban scene needs to be restored. Semantic inpainting refers to inpainting operations that incorporate scene semantic information, unlike simple pixel filling.

[0016] It is understandable that targeted semantic repair based on repair masks and semantic repair instructions can leverage the semantic reasoning capabilities of multimodal models to generate repair content that conforms to the structural logic of urban scenes, effectively eliminate artifacts in the initial reconstruction results, improve the fidelity of intermediate 3D reconstruction results, and provide high-quality basic data for subsequent perspective expansion.

[0017] Step 108: Based on the original acquisition trajectory of the crowdsourced street view video, a local orthogonal basis is constructed at each original camera pose. A tubular constraint space is constructed to enclose the original acquisition trajectory by dynamically adjusting the curriculum evolution factor. A virtual camera trajectory covering the unobserved area is generated in the tubular constraint space. The virtual camera trajectory is used to expand the viewpoint of the intermediate 3D reconstruction results to obtain the reconstructed view under the expanded viewpoint.

[0018] The original acquisition trajectory refers to the actual movement path of the camera during crowdsourced street view video capture, consisting of a series of continuous original camera poses. The original camera pose is the camera state at a single time point on the original acquisition trajectory, including camera position and orientation information. The local orthogonal basis is a Frenet-Serret-like coordinate system constructed at each original camera pose, composed of tangent vectors, normal vectors, and binormal vectors, used to constrain the motion space of the virtual camera. The curriculum evolution factor is a parameter that monotonically increases with training iterations, used to control the degree of offset of the virtual camera trajectory relative to the original acquisition trajectory. The tubular constraint space is the three-dimensional spatial range enclosing the original acquisition trajectory, used to limit the motion boundaries of the virtual camera and prevent the virtual trajectory from deviating from the real scene. The virtual camera trajectory is the camera motion path generated within the tubular constraint space to supplement the blind spots of the original viewpoint. The reconstructed view under the extended viewpoint is an image of the viewpoint not covered by the original acquisition trajectory, obtained by rendering the intermediate results of the 3D reconstruction using the virtual camera trajectory.

[0019] It is understandable that by constructing a local orthogonal basis and a tubular constraint space, and combining it with a curriculum-based evolutionary factor to generate a virtual camera trajectory, it is possible to actively explore the blind spots of observation not covered by the original acquisition trajectory, significantly improve the angular coverage of the scene, provide more vantage point constraints for radiation field optimization, and enhance the geometric consistency of the 3D reconstruction results in the unobserved area.

[0020] Step 110: Calculate the geometric compatibility index between the reconstructed view under the extended view and the corresponding reprojected rendered view, and determine whether the reconstructed view under the extended view meets the 3D manifold consistency requirements based on the geometric compatibility index.

[0021] Geometric compatibility metrics are quantitative indicators used to measure the geometric consistency between the reconstructed view under the extended view and the existing 3D manifold. In this invention, it specifically refers to the Residual Convergence Score (RCS). The reprojected rendering view is the rendered image obtained by reprojecting the temporarily optimized 3D model, after the extended view reconstruction, onto the corresponding extended view. The 3D manifold consistency requirement means that the geometric information contained in the reconstructed view under the extended view must be compatible with the geometric structure of the existing 3D reconstruction results, without non-physical contradictions such as building position offsets or road structure distortions.

[0022] It is understandable that by calculating geometric compatibility indices and determining whether they meet the requirements of 3D manifold consistency, we can effectively screen out extended view reconstructions that conform to physical laws, eliminate possible non-physical generation illusions, provide reliable data support for subsequent iterative updates, and help maintain the geometric integrity and rationality of 3D reconstruction results.

[0023] Step 112: If the three-dimensional manifold consistency requirement is met, the reconstructed view under the extended view is fed back as supplementary training data to the three-dimensional reconstruction optimization process. The three-dimensional reconstruction results are iteratively updated until the preset iteration stop condition is met, at which point the iteration stops and the current three-dimensional reconstruction result is output.

[0024] Supplementary training data refers to reconstructed views from expanded perspectives that have undergone consistency verification. This type of data can be used as pseudo-labels to optimize the 3D reconstruction model. By feeding qualified expanded perspective reconstructed views into the optimization process as supplementary training data, the quality of 3D reconstruction is continuously improved through iterative updates until the preset conditions are met, at which point the output result is released. This approach fully utilizes effective supplementary data to gradually optimize the model, further enhancing the fidelity and geometric consistency of the reconstruction results, and effectively narrowing the gap between fragmented crowdsourced data and professional-grade 3D reconstruction results.

[0025] The method of this invention employs the SAGE-Loop framework to implement a self-supervised optimization pipeline, aiming to iteratively improve the reconstruction fidelity of the neural radiation field. A schematic diagram of the SAGE-Loop framework based on semantic evolutionary loops and the evolution of virtual camera trajectories is shown below. Figure 2 As shown, where, Figure 2 (a) is the flowchart of the SAGE-Loop framework. Figure 2 (b) is a schematic diagram of the virtual camera's spiral trajectory and offset range. This pipeline contains three iterative modules: (1) artifact detection and semantic repair; (2) progressive virtual trajectory evolution (PVTE); and (3) geometric consistency verification based on residual convergence score (RCS). The system constructs a closed-loop mechanism that integrates two-dimensional generation priors with three-dimensional structural constraints to solve common defects in crowdsourced data reconstruction.

[0026] SAGE-Loop consists of three interconnected modules: 1. Detection and Repair Module: Identifies rendering artifacts in the training and synthetic views. The Artifact Segmentation Network (AS-Net) generates domain masks for "cloud-like" floating objects and texture distortions, and expands the masks using a GIS-based spatial buffering strategy to incorporate surrounding contextual information, enabling the Multimodal Model (LMM) to achieve context-aware semantic repair. 2. Active Exploration Module: The Progressive Virtual Trajectory Evolution (PVTE) method is adopted to actively explore unknown or sparse viewpoints along the virtual camera path, thereby improving the angular coverage of the scene and constraining the optimization process of the radiation field in the original sparse observation area. 3. Verification and Feedback Mechanism: Ensuring the Reliability of Synthesized Content. To avoid introducing non-physical generation illusions, each image restored using the multimodal model is evaluated using the Residual Convergence Score (RCS) as a pseudo-label. The Residual Convergence Score measures the geometric alignment between the 2D generated prior and the existing 3D manifold. If an update exceeds a preset inconsistency threshold, the system will perform a rollback operation to maintain the structural integrity of the model.

[0027] As a plug-and-play framework, SAGE-Loop acts as a self-supervised data augmentation engine, actively synthesizing "missing" observation data to compensate for the inherent sparsity of crowdsourced street view videos, effectively bridging the gap between fragmented crowdsourced data and professional-grade 3D reconstruction results.

[0028] In the aforementioned 3D reconstruction enhancement method based on semantic evolutionary loops for crowdsourced street view videos, the initial 3D reconstruction results of the crowdsourced street view videos are obtained, and artifact masks are generated by identifying rendering artifact regions. This allows for precise localization of defects such as geometric floating objects and texture distortion, providing clear targets for targeted repair. The artifact mask is expanded to obtain a repair mask, which incorporates surrounding contextual information, ensuring the semantic coherence between the repaired area and the environment. Semantic repair is performed based on the repair mask and semantic repair instructions, which can eliminate artifacts in the initial reconstruction and improve the fidelity of intermediate 3D reconstruction results. A local orthogonal basis and tubular constraint space are constructed based on the original acquisition trajectory, and the virtual camera trajectory is generated by dynamically adjusting the curriculum evolutionary factor, which can fill the observation blind spots and enhance scene angle coverage. The geometric compatibility index is calculated and extended views that conform to the consistency of the 3D manifold are selected, which can eliminate non-physical illusions and ensure the geometric integrity of the reconstruction. The effective extended views are used as supplementary training data for iterative updates, which can continuously optimize the reconstruction quality until the stopping condition is met. The embodiments of the present invention can effectively make up for the limitations of sparse trajectory and single perspective in crowdsourced street view videos, significantly reduce the proportion of rendering artifacts, improve the fidelity and geometric consistency of 3D reconstruction results, and bridge the gap between fragmented crowdsourced data and professional-grade 3D GIS data.

[0029] In one embodiment, the method further includes: if the three-dimensional manifold consistency requirement is not met, then cancel the parameter update corresponding to the current viewpoint expansion and roll back to the three-dimensional reconstruction state before this optimization.

[0030] In one embodiment, the step of identifying rendering artifact regions in the initial 3D reconstruction result and generating an artifact mask includes: fine-tuning an image segmentation network to obtain an artifact recognition model based on a preset urban street scene artifact dataset; inputting the rendered view of the initial 3D reconstruction result into the artifact recognition model; and outputting an original artifact mask; the original artifact mask is used to label the pixel location of the artifact.

[0031] In this embodiment, accurate localization of rendering defects is a prerequisite for effective repair. Although the subsequent repair process does not require pixel-level perfect segmentation, the detector must robustly identify artifact regions with high recall to avoid omissions. To this end, this invention employs Mask2Former—a general image segmentation architecture that demonstrates state-of-the-art performance in various segmentation tasks.

[0032] This invention fine-tunes the Mask2Former model on a self-built R-Artifact dataset. This dataset contains common artifacts specific to neural rendering, such as "cloud-like" floating objects and high-frequency noise. Formally, given a rendered image... The network will predict a binary probability map. By setting a confidence threshold, the original artifact mask is obtained. The value of 1 indicates the presence of an artifact at that location. It is worth noting that this invention prioritizes detecting the presence of artifacts rather than precisely defining their boundaries, because subsequent buffering steps can compensate for potential boundary errors.

[0033] After artifact localization is complete, the core challenge lies in removing the artifacts while synthesizing reasonable background content that conforms to the scene's semantics. This process involves two key steps: spatial buffer-based mask refinement and instruction-guided semantic editing.

[0034] In one embodiment, expanding the artifact mask to obtain a repair mask containing the artifact region and surrounding context information includes: morphologically dilating the original artifact mask using a circular structuring element of a preset radius to obtain a repair mask covering the artifact region and surrounding semantic context.

[0035] In this embodiment, a naive approach to defining the repair region is to use the minimum bounding circle or bounding rectangle of the detected artifacts. However, rendering artifacts in street scene scenes (especially those generated by 3DGS) often appear as long strips or irregular fragments that traverse the image. Using convex bounding shapes for these artifacts will inevitably occlude a large amount of effective background area, forcing the generation model to overgenerate content, which may compromise 3D consistency.

[0036] Inspired by the buffer operation in Geographic Information Systems (GIS)—which creates a buffer at a specified distance around geometric features—this invention proposes a morphological mask dilation strategy. This strategy ensures that the repair mask fits the irregular shape of artifacts while providing sufficient contextual information for the generative model. Mathematically, the repair mask refined by morphological dilation... By modifying the original mask Morphological dilation using structural elements yields: ; in Represents the morphological dilation operator. For radius The circular structuring element. This operation effectively creates a uniform buffer around the detected artifacts, ensuring that transition boundaries are incorporated into the regeneration process without over-masking the valid scene structure.

[0037] In one embodiment, semantic repair of artifact regions in the initial 3D reconstruction result based on the repair mask and pre-set semantic repair instructions includes: inputting the rendered view of the initial 3D reconstruction result, the repair mask, and the semantic repair instructions into a multimodal large language model, and outputting the repaired intermediate 3D reconstruction result.

[0038] In this embodiment, for the restoration task, the present invention employs an image editing model based on multimodal semantics, specifically Qwen-Image-Edit, which can perform precise image editing based on text commands. Unlike traditional diffusion restoration models that rely on random noise filling or simple category labels, this type of model can understand complex semantic contexts, and therefore performs better in street scene scenarios where structural logic is crucial, such as building facades and road continuity.

[0039] The restoration process is modeled as an instruction-following-generation task, with the model rendering the image. Repair mask refined by morphological dilation and text prompts For input. Text prompt. Designed to "remove artifacts and restore a clean street scene background," the model outputs the repaired image. : ; in This represents the forward propagation process of the editing model. Leveraging the semantic reasoning capabilities of the multimodal semantic image editing model, the synthesized content maintains a high degree of consistency with the surrounding environment, effectively eliminating artifacts while preserving the photorealistic quality required for subsequent 3D optimization.

[0040] To address the inherent geometric ill-posedness of crowdsourced street view videos—where camera acquisition trajectories are typically non-circular, fragmented, or misaligned with the main geometric structure of the scene—this invention proposes a Progressive Virtual Trajectory Evolution (PVTE) strategy, such as... Figure 3 The diagram shown illustrates the Progressive Virtual Trajectory Evolution (PVTE) mechanism, in which... Figure 3 (a) shows a schematic diagram of the construction of a local orthonormal basis; Figure 3 (b) Schematic diagram of virtual camera displacement; Figure 3 (c) Schematic diagram of orientation realignment based on gaze constraints; Figure 3 (d) Schematic diagram of the curriculum-based evolution from near-field fine-tuning to omnidirectional exploration. Unlike traditional data augmentation methods, PVTE actively synthesizes a series of "complementary" virtual camera paths to explore the observation blind spots in the original sparse trajectory, thereby promoting the detection and suppression of rendering artifacts in unobserved areas.

[0041] In one embodiment, constructing a local orthogonal basis at each original camera pose includes: at each original camera pose point on the original acquisition trajectory, taking the instantaneous forward direction of the camera as the tangent vector, and taking two orthogonal vectors perpendicular to the forward direction as the normal vector and the binormal vector, and spanning the local normal plane by the normal vector and the binormal vector.

[0042] In one embodiment, constructing a tubular constraint space encompassing the original acquisition trajectory by dynamically adjusting the curriculum evolution factor includes: setting a maximum offset radius within the local normal plane, centered on the original camera pose. The range for generating the virtual camera position is limited to a circle centered on the original camera pose. Within a circular region with radius [radius], a tubular constraint space is formed that encloses the original acquisition trajectory. To vary with the number of training iterations A monotonically increasing curriculum evolution factor is used to control the degree of deviation of the virtual position relative to the original pose.

[0043] In this embodiment, the core challenge of crowdsourced street view video lies in its prevalent non-professional acquisition characteristics: the raw acquisition trajectory The lack of complete concentric rings around the scene results in anisotropic reconstruction quality. For the first i The original camera's full 6-DOF pose. For the first i The rotation matrix of the original camera in the frame. For the first i The translation vector of the original camera in the frame. To systematically expand the observable manifold, this invention defines tubular constraints. This constraint allows for controlled geometric offsets while encompassing the original acquisition path.

[0044] like Figure 3 As shown in (a), for each original camera pose point, a local class Frenet-Serret orthonormal basis is constructed. ,in Let be the tangent vector representing the instantaneous heading of the camera. It is a local normal unit vector. This is the local binormal unit vector. For example... Figure 3 As shown in (b), to synthesize a new perspective that can "fill" the blind spot of the original trajectory angle, in the local normal plane Inside, the virtual camera center is adjusted by radial displacement. Perform parameterization: ; in The maximum offset radius, As a factor in curriculum evolution, The envelope function is a squared sine wave, used to guarantee the position of the endpoints of the trajectory. Continuity and seamless convergence, For the first The azimuth angle of the virtual camera corresponding to the original camera in the frame within the local normal plane. These are the trajectory parameters.

[0045] In one embodiment, generating a virtual camera trajectory covering an unobserved area within a tubular constrained space includes: realigning the virtual camera's orientation based on Lie group theory, calculating a rotation matrix that makes the virtual camera's principal axis point to a preset target area, adjusting the virtual camera's orientation to ensure that the expanded viewpoint is focused on the unobserved area; and using a curriculum-based evolution strategy to gradually expand and generate a spiral virtual camera trajectory through radial displacement in the local normal plane to achieve full coverage of the unobserved area.

[0046] In this embodiment, since the acquisition trajectory of crowdsourced street view video may not always be oriented towards the scene of interest (e.g., a dashcam looking forward while the target building is located to the side), this invention introduces Lie theory's "gaze" constraint to realign the virtual sensor, such as... Figure 3 As shown in (c). For the virtual camera center Deducing the finely adjusted camera orientation This allows the camera to focus on the target area on the reference manifold. : ; in for The corresponding antisymmetric matrix, For Lie group exponent mapping. Rotation axis and rotation angle The computational objective is to minimize the difference between the camera principal axes and the vector. The angular distance between them. This mechanism ensures that the evolved virtual view can effectively point to the geometric area that needs fine-tuning, regardless of the orientation of the original camera.

[0047] To ensure the synthesized virtual trajectory has physical plausibility and is free from random jitter (jatter would introduce temporal inconsistencies during the repair process), this invention utilizes Lie algebras. Using cubic B-splines to represent discrete camera pose sequences Reparameterize. Continuous virtual trajectory. Defined as: ; in As control points, It is the control point number. The total number of control points, whose sampling variance is determined by the curriculum evolution factor. Scaling For the cubic B-spline, the th basis functions These are spline parameters. This form ensures a smooth transition in the motion of the virtual camera to simulate the physical camera, providing a stable sequence of views for the diffusion repair model.

[0048] like Figure 3 As shown in (d), the transition from local manifold fine-tuning to global exploration is driven by the curriculum evolution factor. The control factor is implemented as a monotonic sigmoid mapping function to ensure that the strength of data augmentation is increased in a controllable manner.

[0049] In the initial optimization phase Constrained to a sub-decimeter scale to achieve near-field manifold fine-tuning: The system generates fine-grained viewpoint perturbations near the baseline of the original acquisition trajectory to solve the high-frequency depth blur problem and suppress "floating object" artifacts near the main view frustum; As the radiation field gradually converges and stabilizes Gradually increase the size, and switch the strategy to the manifold expansion stage: introduce a virtual observation view with a wide baseline, force multi-view geometric consistency from previously unobserved angles, and thus regularize the 3D geometry to address the inherent sparsity of crowdsourced street view videos.

[0050] To compensate for the lack of vertical or circumferential coverage in crowdsourced data collection, this strategy employs monotonic azimuth drift. Introducing an omnidirectional exploration mechanism, in which, The initial azimuth angle, The angular frequency is used. This form transforms the tubular manifold into a series of helical trajectories, systematically exposing the scene's geometry from different heights and off-axis angles. By dynamically expanding the sampling space, the PVTE strategy enables a generative feedback loop, consisting of AS-Net and a multimodal model-based insulator, to proactively correct inconsistencies in the 3D structure. Finally, a high-fidelity synthesized view is obtained. Provides robust supervision signals to anchor 3D representations to a globally consistent manifold—even if the source input is only a unidirectional acquisition trajectory.

[0051] This curriculum-driven approach effectively bridges the gap between two-dimensional generative priors and the integrity of three-dimensional structures, ensuring that the enhanced radiation field can maintain geometric anchoring even if the original crowdsourced collection trajectory has fragmented characteristics.

[0052] In one embodiment, the step of calculating the geometric compatibility index between the reconstructed view under the extended view and the corresponding reprojected rendered view includes: using the reconstructed view under the extended view as a pseudo-label, performing a preset number of temporary optimizations on the intermediate results of the 3D reconstruction, and re-rendering the corresponding view based on the optimized 3D reconstruction results; calculating the structural similarity and pixel difference between the re-rendered view and the reconstructed view under the extended view, and fusing them to obtain a residual convergence score as the geometric compatibility index. The step of determining whether the reconstructed view under the extended view meets the 3D manifold consistency requirements based on the geometric compatibility index includes: comparing the current residual convergence score with the moving average of the residual convergence scores from the previous K valid updates; if the current score is greater than the product of a preset attenuation coefficient and the moving average, it is determined that it meets the 3D manifold consistency requirements; otherwise, it is determined that it does not meet the requirements.

[0053] In this embodiment, to ensure the scientific rigor of the view repaired by the multimodal model, this invention proposes a closed-loop verification mechanism to evaluate the dynamic convergence of the 3D geometry. This invention does not use the output of the multimodal model. Instead of treating it as an absolute truth value, it is used as a candidate for a pseudo-label, and its validity is cross-checked by the intrinsic consistency of the three-dimensional radiation field.

[0054] A New Perspective Generate repaired image Then, the system through Step-by-step optimization, changing scene parameters from Temporary update to : ; in Indicates perspective The rendering function at that location. This process allows the radiation field to attempt a geometric mapping of the newly injected generated content: if If the underlying 3D manifold is consistent, the model should be able to converge to a stable and uniform solution; otherwise, geometric inconsistencies will manifest as high residuals.

[0055] The reliability of the composite view is quantitatively evaluated using the residual convergence score (RCS). After temporary optimization, the scene is re-rendered to obtain... The residual convergence fraction is defined as: ; in To balance the weights. High The scores indicate that the 2D generated content of most modal models has been successfully "absorbed" by the 3D representation without violating spatial constraints; conversely, low scores indicate that the output of most modal models contradicts the established sparse view geometry, resulting in the inability to synthesize a consistent 3D structure from the radiation field. This mechanism effectively uses the 3DGS / NeRF framework as a "geometric filter" to eliminate physically unreasonable multimodal model generation results.

[0056] To maintain the stability of the reconstruction process, this invention implements an adaptive rollback strategy based on feedback of the residual convergence score: the system retains parameter updates only when the following criteria are met: ; in The attenuation coefficient is... For the number of iterations, To add a chapter to history and mark a successful chapter. The moving average window size represents the number of successful historical injections included in the calculation. The terms on the right are the previous... The moving average of the residual convergence scores of each successful injection. If this criterion is not met, the system will trigger a state rollback, changing the parameters from... Restore to And discard the current virtual perspective.

[0057] This "optimize first, verify later" cycle ensures that only those multimodal model enhancement views that can improve the globally consistent 3D structure are included in the optimization process, preventing the accumulation of geometric artifacts and gradually enhancing the reconstruction results under sparse perspectives, rather than being destroyed by unconstrained 2D generation priors.

[0058] To rigorously evaluate the effectiveness of the proposed progressive enhancement framework, extensive experiments were conducted on the CS3D-Continuous dataset. This dataset is a carefully selected subset of the CS3D100 dataset, specifically choosing samples with high trajectory continuity, representing the most challenging and common scenes in crowdsourced street view videos. Unlike standard object-centric datasets, the CS3D-Continuous dataset exhibits significant scale variations and sparse viewpoint coverage, making it an ideal benchmark for artifact suppression tasks.

[0059] To train the artifact segmentation network (AS-Net), this invention uses the R-Artifact dataset. This dedicated dataset is constructed by systematically filtering the "perceptual artifact localization" dataset: only urban street scene categories (such as building structures, road surfaces, and vegetation) are retained, while irrelevant subjects such as human figures and animals are removed to ensure domain alignment of the dataset.

[0060] This invention framework is implemented within the NeRFstudio ecosystem. To demonstrate its versatility, it is evaluated on two representative radiation field representation models: Nerfacto (a NeRF-based model) and 3D Gaussian splash (3DGS). The artifact segmentation module (AS-Net) employs the Mask2Former architecture and is trained directly on the R-Artifact dataset to identify "cloud-like" floating objects, distorted textures, and geometric blur.

[0061] The repair task uses Qwen-Image-Edit (an instruction-guided multimodal model) and configures the morphological buffer radius. Pixels ensure that the repair process captures sufficient local context to achieve semantic repair. In terms of optimized scheduling, Progressive Virtual Trajectory Evolution (PVTE) and the subsequent "detect-repair-feedback" loop are triggered every 3000 iterations; in each enhancement cycle, a virtual view is synthesized along the evolutionary manifold defined by Lie theory constraints.

[0062] This invention compares the proposed method with several representative baseline models: 1. Basic reconstruction models (Nerfacto and 3DGS); 2. An SDS-based optimized variant inspired by DiffDreamer; 3. Post-repair pipeline (similar to Instruct-NeRF2NeRF): The generated model is only used as the final image spatial filter and does not participate in the 3D optimization loop.

[0063] like Figure 4 The image shows an example of R-artifact annotation samples from the CS3D-Continuous dataset, including geometric floating objects and semantic distortions.

[0064] This invention quantitatively evaluates the framework's performance from two core dimensions: global reconstruction fidelity and local artifact suppression. As shown in Table 1, the best results are highlighted in bold, and the second-best results are underlined. APR (Augmentation Pixel Rate) is used to quantify the geometric cleanliness of the new perspective rendering results. For both NeRF and 3DGS, the method of this invention demonstrates significant advantages on all standard evaluation metrics (PSNR, SSIM, and perceptual alignment LPIPS). Notably, SAGE-Loop achieves a peak signal-to-noise ratio of 22.84 dB, a 2.69 dB improvement over the basic 3DGS model; the LPIPS metric decreases from 0.382 to 0.168, a more significant improvement than PSNR. This indicates that although pixel-level brightness values ​​are relatively stable, the visual plausibility and high-frequency structural integrity of the reconstruction results are significantly enhanced.

[0065] Table 1. Quantitative comparison of different methods on the CS3D-Continuous dataset.

[0066] In addition to traditional evaluation metrics, this invention introduces Artifact Pixel Rate (APR) to quantify rendering defects at a more granular level. APR represents the proportion of pixels identified as artifacts by AS-Net in the new perspective rendering result, and is a direct measure of the "geometric cleanliness" of the reconstructed scene. As shown in Table 1, the framework of this invention achieves a minimum APR of 1.32%, which is significantly better than the SDS-based optimization method (5.88%).

[0067] This invention observes a key phenomenon: although SDS-based methods attempt to regularize the scene using global priors, they often introduce overly smooth, "waxy" textures, failing to capture the fine details of the urban environment. Therefore, even with a certain reduction in artifacts, the LPIPS index remains high. In contrast, the "detect-repair-feedback" mechanism of this invention ensures that the generated priors are accurately applied (only to problem areas), effectively suppressing rendering distortion while preserving the realistic details of the street scene.

[0068] The iterative collaboration between the two-dimensional generation prior and the three-dimensional representation further confirms the effectiveness of the method in this invention. To further investigate this dynamic process, this invention... Figure 5 The variation trend of APR during training is visualized. Unlike post-restoration baselines that only mask defects in the final output (manifested as a sharp drop in APR in the later stages of training), this invention's framework uses the restored image as a "pseudo-ground value" to supervise the underlying 3D manifold throughout the entire training process. Figure 5The diagram showing the trend of artifact pixel rate (APR) during the training process demonstrates that SAGE-Loop exhibits a clear "step-like" convergence pattern, which confirms that the closed-loop mechanism of this invention fundamentally optimizes the integrity of the three-dimensional structure, rather than just performing spatial filtering on the image.

[0069] Compared to basic 3DGS and post-repair methods, SAGE-Loop exhibits a typical “step-like” descent characteristic. Each vertical descent (marked by a red dashed line) represents an enhancement cycle triggered by PVTE, indicating that the radiation field gradually internalizes semantic priors to cull geometric floaters.

[0070] To further demonstrate the practical effectiveness of the framework, this invention conducts a comprehensive qualitative comparison between the proposed method and baseline methods in various challenging scenarios on the CS3D-Continuous dataset. For example... Figure 6 The diagram shows a qualitative comparison of 3D reconstruction results on the CS3D-Continuous dataset. Figure 6 (a) is a schematic diagram comparing the texture of building facades and the cleanliness of roads using different methods from a conventional perspective. Figure 6 (b) is a schematic diagram comparing the new perspective synthesis effect of the method of the present invention in extreme off-axis camera offset scenarios. The visual results show that the method of the present invention has achieved significant improvements in both local detail repair and global structural coherence.

[0071] For 3DGS-based reconstruction results, the base model frequently generates "cloud-like" floating objects and semi-transparent artifacts in areas with sparse viewpoint coverage, severely reducing perceptual realism. In contrast, the framework of this invention, guided by AS-Net and a semantically aware inpainting loop, can effectively identify these transient floating objects and replace them with stable, high-fidelity textures. Figure 6 As shown in the magnified area of ​​(a), the distorted textures on the building facade and the “fog” artifacts near the road boundary were successfully corrected, and the reconstructed surface became clear and clean.

[0072] For NeRF-based representation models (Nerfacto), traditional SDS-based optimization methods (such as DiffDreamer) often introduce "waxy" or oversaturated textures. The method of this invention achieves a better balance between sharpness and realism: with the deep semantic understanding capabilities of Qwen-Image-Edit, the synthesized content (such as the structural pattern of a window) not only highly conforms to semantic logic but also remains consistent with the surrounding environment.

[0073] Furthermore, the effect of Progressive Virtual Trajectory Evolution (PVTE) is visually demonstrated in the new perspective synthesis results: when the camera deviates from the original acquisition trajectory (such as lateral shift), the reconstruction performance of the baseline method degrades significantly, while the framework of this invention provides robust supervision through the synthesized "pseudo-truth" view. Figure 6 The side-by-side comparison in (b) clearly demonstrates that the method of the present invention effectively suppresses the "ghosting" effect and geometric distortion, and can generate a more immersive and visually consistent 3D experience.

[0074] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated in this invention, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Furthermore, Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0075] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0076] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for 3D reconstruction and enhancement of crowdsourced street view videos based on semantic evolutionary loops, characterized in that, The method includes: Obtain the initial 3D reconstruction result corresponding to the crowdsourced street view video, identify the rendering artifact regions in the initial 3D reconstruction result and generate an artifact mask; The artifact mask is expanded to obtain a repair mask containing the artifact region and surrounding context information; Based on the repair mask and the pre-set semantic repair instructions, semantic repair is performed on the artifact regions in the initial 3D reconstruction result to obtain intermediate 3D reconstruction results. Based on the original acquisition trajectory of the crowdsourced street view video, a local orthogonal basis is constructed at each original camera pose. A tubular constraint space is constructed to enclose the original acquisition trajectory by dynamically adjusting the curriculum evolution factor. A virtual camera trajectory covering the unobserved area is generated in the tubular constraint space. The virtual camera trajectory is used to expand the perspective of the intermediate results of the 3D reconstruction to obtain the reconstructed view under the expanded perspective. Calculate the geometric compatibility index between the reconstructed view under the extended view and the corresponding reprojected rendered view, and determine whether the reconstructed view under the extended view meets the three-dimensional manifold consistency requirements based on the geometric compatibility index; If the three-dimensional manifold consistency requirement is met, the reconstructed view under the extended perspective is fed back as supplementary training data to the three-dimensional reconstruction optimization process, and the three-dimensional reconstruction results are iteratively updated until the preset iteration stop condition is met, at which point the iteration stops and the current three-dimensional reconstruction result is output.

2. The method according to claim 1, characterized in that, The method further includes: If the three-dimensional manifold consistency requirement is not met, the parameter update corresponding to the current viewpoint expansion will be canceled, and the system will roll back to the three-dimensional reconstruction state before this optimization.

3. The method according to claim 1, characterized in that, The step of identifying rendering artifact regions in the initial 3D reconstruction result and generating an artifact mask includes: Based on a preset urban street scene artifact dataset, the image segmentation network is fine-tuned to obtain an artifact recognition model. The rendered view of the initial 3D reconstruction result is input into the artifact recognition model, and the original artifact mask is output. The original artifact mask is used to mark the pixel position of the artifact.

4. The method according to claim 1, characterized in that, The process of expanding the artifact mask to obtain a repair mask containing artifact regions and surrounding context information includes: Morphological dilation of the original artifact mask is performed using a circular structuring element with a preset radius to obtain a repair mask that covers the artifact region and the surrounding semantic context.

5. The method according to claim 1, characterized in that, The semantic repair of artifact regions in the initial 3D reconstruction result based on the repair mask and pre-set semantic repair instructions includes: The rendered view of the initial 3D reconstruction result, the repair mask, and the semantic repair instructions are input into the multimodal large language model, and the repaired intermediate 3D reconstruction result is output.

6. The method according to claim 1, characterized in that, The construction of a local orthogonal basis at each original camera pose includes: At each original camera pose point on the original acquisition trajectory, the instantaneous forward direction of the camera is taken as the tangent vector, and two orthogonal vectors perpendicular to the forward direction are taken as the normal vector and the binormal vector. The normal vector and the binormal vector span a local normal plane.

7. The method according to claim 6, characterized in that, The process of constructing a tubular constraint space encompassing the original acquisition trajectory by dynamically adjusting the curriculum evolution factors includes: With the original camera pose as the center, set the maximum offset radius in the local normal plane. The range for generating the virtual camera position is limited to a circle centered on the original camera pose. Within a circular region with radius [radius], a tubular constraint space is formed that encloses the original acquisition trajectory. To vary with the number of training iterations A monotonically increasing curriculum evolution factor is used to control the degree of deviation of the virtual position relative to the original pose.

8. The method according to claim 6, characterized in that, The process of generating a virtual camera trajectory covering the unobserved area within the tubular constrained space includes: Based on Lie group theory, the virtual camera is realigned by calculating the rotation matrix that makes the virtual camera's main axis point to the preset target area, and adjusting the virtual camera's orientation to ensure that the extended view is focused on the unobserved area. By adopting a curriculum-based evolution strategy, a spiral virtual camera trajectory is generated by gradually expanding radial displacement within the local normal plane, thereby achieving full coverage of unobserved areas.

9. The method according to claim 1, characterized in that, The step of calculating the geometric compatibility index between the reconstructed view and the corresponding reprojected rendered view under the extended view includes: Using the reconstructed view under the extended perspective as a pseudo-label, the intermediate results of the 3D reconstruction are temporarily optimized by a preset number of steps, and the re-rendered view of the corresponding perspective is obtained by re-rendering based on the optimized 3D reconstruction results. Calculate the structural similarity and pixel difference between the re-rendered view and the reconstructed view under the extended view, and fuse them to obtain the residual convergence score as a geometric compatibility index.

10. The method according to claim 1, characterized in that, The step of determining whether the reconstructed view under the extended perspective conforms to the three-dimensional manifold consistency requirement based on the geometric compatibility index includes: The current residual convergence score is compared with the moving average of the residual convergence scores of the previous K valid updates. If the current score is greater than the product of the preset decay coefficient and the moving average, it is determined that the three-dimensional manifold consistency requirement is met; otherwise, it is determined that the requirement is not met.

Images