Three-dimensional scene reconstruction method

Abstract

The application discloses a three-dimensional scene reconstruction method, and relates to the field of data processing, and comprises the following steps: acquiring multi-view images of a target scene; initializing a plurality of three-dimensional rendering primitives based on the multi-view images; judging and suppressing the splitting operation of part of the three-dimensional rendering primitives according to the projection area of each three-dimensional rendering primitive under a rendering perspective, thereby significantly reducing invalid calculation amount and memory occupation; normally splitting and cloning the three-dimensional rendering primitives which are not suppressed; and screening the initialized three-dimensional rendering primitives, the split or cloned three-dimensional rendering primitives based on the projection area, thereby obtaining target three-dimensional rendering primitives. The application effectively retains the geometric detail features of complex concave-convex topography; the three-dimensional scene model obtained based on the target three-dimensional rendering primitives has higher precision and better rendering effect, and the application effectively balances the overall three-dimensional rendering primitive distribution and reduces the calculation and memory overhead.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method for three-dimensional scene reconstruction. Background Technology

[0002] Currently, in the field of 3D scene reconstruction, 3D Gaussian Splatting (3DGS) technology is widely used due to its efficient real-time rendering capabilities. This technology uses explicit anisotropic 3D rendering primitives to represent the geometric and textural information of the scene.

[0003] When this technology is applied to scenes where large areas of flat terrain coexist with localized complex terrain such as mountains, deserts, and snowfields, existing 3D scene modeling methods generally employ a uniform, undifferentiated primitive densification and pruning strategy, which has significant drawbacks. For example, for flat areas with simple geometric features and gradual changes, the undifferentiated densification mechanism can lead to excessive proliferation of 3D rendering primitives. The existence of a large number of redundant 3D rendering primitives will significantly increase the computational overhead during the training and rendering process of 3D scene models, resulting in a large overall computational load, long training time, and excessive consumption of video memory resources, as well as serious waste of hardware resources.

[0004] Therefore, there is an urgent need for a 3D scene reconstruction method that can reduce the overall computational load and save computational resources. Summary of the Invention

[0005] To address the problems in existing technologies where the imbalance in the distribution of 3D rendering primitives in 3D scene reconstruction leads to redundant accumulation in flat areas and underfitting of details in complex areas, this application proposes a 3D scene reconstruction method that achieves adaptive suppression of 3D rendering primitive growth, screening based on projected area, and directional completion of geometrically missing areas, thereby improving the modeling accuracy of 3D scene models.

[0006] The first aspect of this application provides a method for reconstructing a three-dimensional scene, including:

[0007] Acquire multi-view images of a target scene, the target scene including flat terrain and locally uneven terrain;

[0008] Multiple 3D rendering primitives are obtained based on the multi-view image initialization;

[0009] For each of the three-dimensional rendering primitives, based on the projected area of ​​the three-dimensional rendering primitive in the rendering view, it is determined whether to perform the operation of suppressing the splitting of the three-dimensional rendering primitive.

[0010] Perform a splitting or cloning operation based on the unsuppressed state of the 3D rendering primitive to obtain multiple split or cloned 3D rendering primitives.

[0011] Based on the projected area of ​​each of the three-dimensional rendering primitives in the set of three-dimensional rendering primitives under the rendering view, a target three-dimensional rendering primitive is obtained by filtering from the set of three-dimensional rendering primitives. The set of three-dimensional rendering primitives includes multiple three-dimensional rendering primitives obtained by initialization and multiple three-dimensional rendering primitives that are split or cloned.

[0012] Based on the target 3D rendering primitives, a 3D scene model of the target scene is obtained.

[0013] In one possible implementation, before obtaining the 3D scene model of the target scene based on the target 3D rendering primitives, the method further includes:

[0014] A rendered image is obtained by rendering all the three-dimensional rendering primitives in the set of three-dimensional rendering primitives along the rendering viewpoint in the forward direction.

[0015] Based on the rendered image, identify areas lacking detail, and generate new 3D rendering primitives in those areas.

[0016] In one possible implementation, obtaining the 3D scene model of the target scene based on the target 3D rendering primitive includes:

[0017] Based on the new 3D rendering primitive and the target 3D rendering primitive, a 3D scene model of the target scene is obtained.

[0018] In one possible implementation, after generating new 3D rendering primitives in the region of missing details, and before obtaining the 3D scene model of the target scene based on the new 3D rendering primitives and the target 3D rendering primitives, the following steps are also included:

[0019] Set the color attribute of the new 3D rendering primitive to the color information derived from the low-order spherical harmonic function;

[0020] Set the opacity parameter of the new 3D rendering primitive to the initial opacity value.

[0021] In one possible implementation, the step of identifying regions of missing detail based on the rendered image and generating new 3D rendering primitives in the direction of these regions includes:

[0022] Based on the cumulative opacity or depth difference corresponding to the rendered image, target pixels with geometric defects are identified. The cumulative opacity refers to the sum of the opacities of multiple three-dimensional rendering primitives under the rendering viewpoint. The depth difference refers to the difference between the rendering depth value corresponding to the pixel in the rendered image and the actual depth value corresponding to the pixel in the actual scene depth map.

[0023] The region of missing detail is defined based on the target pixel;

[0024] Extract the three-dimensional spatial coordinates and real texture color corresponding to the target pixel;

[0025] Based on the three-dimensional spatial coordinates and the real texture colors, the new three-dimensional rendering primitives are generated in the areas where details are missing.

[0026] In one possible implementation, the target 3D rendering primitive is obtained by filtering from the 3D rendering primitive set based on the projected area of ​​each of the 3D rendering primitives in the rendering viewpoint. The 3D rendering primitive set includes multiple 3D rendering primitives initialized and multiple 3D rendering primitives split or cloned, including:

[0027] For each of the three-dimensional rendering primitives in the set of three-dimensional rendering primitives, the rendering contribution of the three-dimensional rendering primitive to the rendering effect of the target scene is determined based on the projected area of ​​the three-dimensional rendering primitive under the rendering view.

[0028] Retain the target 3D rendering primitives in the set of 3D rendering primitives whose rendering contribution meets the preset conditions.

[0029] In one possible implementation, determining the rendering contribution of the 3D rendering primitive to the rendering effect of the target scene based on the projected area of ​​the 3D rendering primitive under the rendering viewpoint includes:

[0030] Obtain the opacity and view contribution of the 3D rendering primitive under the rendering viewpoint;

[0031] The rendering contribution of the 3D rendering primitive is calculated based on the projected area, the opacity, and the viewpoint contribution.

[0032] In one possible implementation, the target 3D rendering primitives in the set of 3D rendering primitives whose rendering contribution meets a preset condition include at least one of the following:

[0033] Normalize the rendering contribution of each 3D rendering primitive in the set of 3D rendering primitives to obtain the retention probability of each 3D rendering primitive; retain the target 3D rendering primitives whose retention probability is greater than or equal to a preset retention threshold among the multiple 3D rendering primitives.

[0034] Based on the first number of 3D rendering primitives with non-zero rendering contribution in the 3D rendering primitive set, the preset sampling factor, and the total number of 3D rendering primitives in the 3D rendering primitive set, a second number is obtained; the second number of target 3D rendering primitives are retained by sampling without replacement.

[0035] All the three-dimensional rendering primitives in the set of three-dimensional rendering primitives are sorted in descending order of their rendering contribution to obtain a sorting result; a cumulative distribution function is calculated based on the sorting result; and the minimum number of target three-dimensional rendering primitives that make the cumulative distribution function reach a preset cumulative threshold are retained.

[0036] In one possible implementation, obtaining the 3D scene model of the target scene based on the target 3D rendering primitive includes:

[0037] Based on the projected area of ​​each 3D rendering primitive in the primitive set under the rendering viewpoint, the rendering contribution of each 3D rendering primitive to the rendering effect of the target scene is determined; the primitive set includes at least the target 3D rendering primitive.

[0038] Retain the 3D rendering primitives in the primitive set whose rendering contribution meets the preset conditions;

[0039] Detect whether the candidate scene model composed of the retained 3D rendering primitives satisfies the preset convergence condition;

[0040] If the preset convergence condition is met, the candidate scene model is determined to be the 3D scene model;

[0041] If the preset convergence condition is not met, return to the execution step and determine whether to perform the operation to suppress the splitting of the 3D rendering primitive based on the projected area of ​​the 3D rendering primitive in the rendering view.

[0042] In one possible implementation, determining whether to perform a splitting operation to suppress the 3D rendering primitive based on the projected area of ​​the 3D rendering primitive in the rendering viewpoint includes:

[0043] Determine the ratio between the projected area of ​​the 3D rendering primitive under the rendering viewpoint and the total area of ​​the rendered image;

[0044] If the ratio is greater than or equal to a preset ratio threshold, the 3D rendering primitive is prohibited from performing a splitting operation.

[0045] A second aspect of this application provides a three-dimensional scene reconstruction apparatus, comprising:

[0046] The image acquisition module is used to acquire multi-view images of a target scene, which includes flat terrain and locally uneven terrain;

[0047] A primitive generation module is used to initialize multiple 3D rendering primitives based on the multi-view image;

[0048] The split suppression module is used to determine, for each of the three-dimensional rendering primitives, whether to perform a split suppression operation on the three-dimensional rendering primitive based on the projected area of ​​the three-dimensional rendering primitive in the rendering view.

[0049] The split-clone execution module is used to perform split or clone operations based on the unsuppressed state of the three-dimensional rendering primitives to obtain multiple split or cloned three-dimensional rendering primitives.

[0050] The primitive filtering module is used to filter out target three-dimensional rendering primitives from the three-dimensional rendering primitive set based on the projected area of ​​each three-dimensional rendering primitive in the rendering view. The three-dimensional rendering primitive set includes multiple three-dimensional rendering primitives initialized and multiple three-dimensional rendering primitives split or cloned.

[0051] The model generation module is used to obtain a 3D scene model of the target scene based on the target 3D rendering primitives.

[0052] A third aspect of this application provides a computer program product including computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the three-dimensional scene reconstruction method of the first aspect or any implementation thereof.

[0053] A fourth aspect of this application provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0054] The memory is used to store computer programs;

[0055] The processor is used to execute the computer program so that the electronic device can implement the three-dimensional scene reconstruction method of the first aspect or any implementation thereof.

[0056] The fifth aspect of this application provides a computer storage medium carrying one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the three-dimensional scene reconstruction method of the first aspect or any implementation thereof.

[0057] The sixth aspect of this application provides a vehicle that includes a three-dimensional scene reconstruction apparatus as described in the second aspect above, or an electronic device as described in the fourth aspect above.

[0058] Using the above technical solution, this application provides a 3D scene reconstruction method to acquire multi-view images of the target scene; multiple 3D rendering primitives are initialized based on the multi-view images; the initialized 3D rendering primitives have two major defects: on the one hand, they contain invalid impurity primitives, and on the other hand, they lack the ability to restore the details of the scene's uneven terrain. To supplement scene details, it is necessary to perform a splitting operation on the 3D rendering primitives to refine the geometric texture, but not all primitives are suitable for splitting: if 3D rendering primitives with large projected areas are continuously split, a large amount of redundancy will be generated. Therefore, it is necessary to first determine whether to suppress the splitting operation based on the primitive's projected area. For 3D rendering primitives that are not suppressed from splitting, a splitting or cloning operation is performed according to their own state to generate a batch of refined 3D rendering primitives, effectively supplementing the detailed information of local uneven terrain. However, in the 3D rendering primitives after splitting, cloning and expansion, in addition to the effective 3D rendering primitives that carry scene details, there are also a large number of low-contribution noise and redundant 3D rendering primitives mixed in. Therefore, the rendering contribution of the initialized 3D rendering primitives and the classified or cloned 3D rendering primitives under the rendering view is quantified, and the target 3D rendering primitives are selected. The selected target 3D rendering primitives retain scene details while eliminating noise redundancy. The 3D scene models built based on these target 3D rendering primitives have higher accuracy and better rendering effects. This effectively balances the overall distribution of 3D rendering primitives and reduces computational and memory overhead. Attached Figure Description

[0059] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements.

[0060] Figure 1 A schematic diagram of a system architecture is provided for this application;

[0061] Figure 2 A flowchart illustrating a three-dimensional scene reconstruction method provided in an embodiment of this application;

[0062] Figure 3 This is a schematic diagram of the structure of a three-dimensional scene reconstruction device provided in an embodiment of this application;

[0063] Figure 4 This is a schematic diagram of the structure of an electronic device provided in this application. Detailed Implementation

[0064] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0065] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0066] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0068] This application can be applied to the field of data processing. The following section will introduce several application scenarios that have been implemented in products, taking the reconstruction of 3D scenes for off-road environments as an example.

[0069] In practical applications such as autonomous driving off-road simulation, unmanned vehicle field navigation, and military exercises, target scenes often exhibit significant terrain duality, simultaneously containing large areas of flat terrain (such as deserts, snow-covered plains, and paved roads) and localized uneven terrain (such as gravel gullies, steep slopes, and muddy ruts). This mixed terrain feature places extremely high demands on 3D reconstruction algorithms: they must avoid wasting computational resources in simple, flat areas while ensuring sufficient geometric accuracy in complex, detailed areas. This application presents an adaptive modeling scheme designed specifically for such scenarios.

[0070] See Figure 1 , Figure 1 A schematic diagram of a system architecture is shown. The system may include an in-vehicle terminal 100 and a server 200. The server 200 can provide the methods provided in the embodiments of this application to one or more in-vehicle terminals.

[0071] The vehicle terminal 100 may be equipped with an application for 3D scene modeling. The application can provide an interface. The vehicle terminal 100 can receive relevant parameters input by the user on the interactive interface of the vehicle terminal 100, such as parameters required for scene modeling and multi-view images, and send the parameters to the server 200. The server 200 can obtain the processing result based on the received parameters and return the processing result to the vehicle terminal 100.

[0072] It should be understood that in some optional implementations, the vehicle terminal 100 can also complete the action of obtaining the processing result based on the received parameters on its own, without the need for the server to cooperate. This application embodiment is not limited to this.

[0073] The vehicle-mounted terminal 100 may include a radio frequency unit, a memory, an input unit, a display unit, a camera (optional), an audio circuit (optional), a speaker (optional), a microphone (optional), a headphone jack (optional), a processor, an external interface, a power supply, and other components. Those skilled in the art will understand that the above-mentioned components are merely examples and do not constitute a limitation on the vehicle-mounted terminal or multi-functional device; it may include more or fewer components, or combine certain components, or use different components.

[0074] The input unit can be used to receive input numeric or character information, and to generate key signal inputs related to user settings and function control of the portable multi-functional device. Specifically, the input unit may include a touchscreen (optional) and / or other input devices. Other input devices may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0075] Among them, the input device can receive input data, etc.

[0076] The display unit can be used to display information input by the user or information provided to the user, various menus of the vehicle terminal, interactive interfaces, file display, and / or playback of any multimedia file. In the embodiments of this application, the display unit can be used to display the interactive operation interface for 3D scene modeling, the modeling parameter configuration interface, the interface for the running status of the modeling task, processing results, etc.

[0077] The memory can be used to store software code related to the 3D scene reconstruction method, the processor can execute the steps of the 3D scene reconstruction method, and can also schedule other units (such as the above-mentioned input unit and display unit) to achieve the corresponding functions.

[0078] This radio frequency unit (optional) can be used to receive and send signals during information transmission or calls.

[0079] In this embodiment of the application, the radio frequency unit can send data to the server 200 and receive the processing results sent by the server 200.

[0080] It should be understood that this radio frequency unit is optional and can be replaced with other communication interfaces, such as a network port.

[0081] The vehicle terminal 100 also includes a power source (such as a battery) that supplies power to the various components.

[0082] The vehicle terminal 100 also includes an external interface, which can be a standard Micro USB interface or a multi-pin connector. It can be used to connect the vehicle terminal 100 to other devices for communication, or to connect a charger to charge the vehicle terminal 100.

[0083] Server 200 includes a bus, a processor, a communication interface, and memory. The processor, memory, and communication interface communicate with each other via the bus.

[0084] The memory can be used to store software code related to the 3D scene reconstruction method, the processor can execute the steps of the 3D scene reconstruction method, and can also schedule other units to achieve corresponding functions.

[0085] Reference Figure 2 , Figure 2 This is a flowchart illustrating a three-dimensional scene reconstruction method provided in an embodiment of this application, as shown below. Figure 2 As shown in the figure, a three-dimensional scene reconstruction method provided in this application embodiment may include steps S201 to S206, which are described in detail below.

[0086] Step S201: Obtain multi-view images of the target scene, which includes flat terrain and locally uneven terrain.

[0087] Multi-view images refer to sequences of images of a target scene captured from different shooting positions and angles using a camera array or a mobile acquisition platform. The image sequences of the target scene have sufficient overlap to support 3D geometric reconstruction.

[0088] The target scene is a real outdoor 3D scene to be modeled in this application. It is a mixed terrain scene consisting of flat terrain areas and local uneven terrain areas, such as an off-road scene.

[0089] Flat terrain refers to areas with little variation in surface curvature and relatively simple and continuous texture, such as open sandy land or compacted dirt roads; while locally uneven terrain refers to areas with drastic changes in surface normal vectors and high-frequency geometric details, such as scattered piles of stones or eroded gullies.

[0090] It should be understood that this embodiment uses an off-road environment as an example for illustration. This method can also be applied to various mixed scenarios that combine flat and complex areas, such as urban roads and building facades, indoor floors and furniture.

[0091] Step S202: Obtain multiple 3D rendering primitives based on the multi-view image initialization.

[0092] The 3D rendering primitive is the smallest explicit representation unit of a 3D scene model, such as an anisotropic 3D Gaussian ellipsoid.

[0093] Each 3D rendering primitive not only has position coordinates in 3D space, but also carries a covariance matrix (representing shape, size, and rotation), opacity, and color attributes (such as spherical harmonic coefficients).

[0094] For example, sparse point clouds can be parsed from multi-view images using the Structure from Motion (SfM) algorithm, and each sparse point can be initialized as a 3D rendering primitive; or initial 3D rendering primitives can be distributed in the target scene through random sampling.

[0095] As an explicit representation of the target scene, 3D rendering primitives can fit various geometries, from flat ground to steep rocks, by adjusting their own parameters. For example, a flat sandy area may only require a small number of large, flat 3D rendering primitives to cover it; while a pile of gravel requires a large number of small, differently oriented 3D rendering primitives to approximate its complex surface contours.

[0096] Step S203: For each 3D rendering primitive, based on the projected area of ​​the 3D rendering primitive in the rendering view, determine whether to perform the operation to suppress the splitting of the 3D rendering primitive.

[0097] The split suppression logic in this step remains effective throughout the entire model training process, covering the pre-training and densification stage, the pruning stage, the resampling stage, and all subsequent iterations and optimization stages.

[0098] The rendering perspective refers to the observation perspective of the virtual camera when the 3D scene model is rendered in the forward direction. It is the viewpoint for rendering the target scene, whether fixed or traversed.

[0099] The projected area refers to the number of pixels covering the elliptical region projected onto the two-dimensional image plane by the 3D rendering primitive from the rendering perspective. It intuitively reflects the potential influence range of the 3D rendering primitive on the rendering result from the rendering perspective.

[0100] Splitting operations refer to the process of replacing a parent 3D rendering primitive with two or more child 3D rendering primitives. They are typically used to increase the primitive density in a local area to improve fitting accuracy.

[0101] In scenes containing large areas of flat terrain, indiscriminately allowing all 3D rendering primitives to split would result in a massive number of redundant 3D rendering primitives being generated in the flat areas. Therefore, this step establishes a suppression mechanism based on projected area: when the projected area of ​​a 3D rendering primitive is too large, it indicates that it is fitting a large-scale flat surface. In this case, the 3D rendering primitive is sufficient to express the current geometric features and does not need further refinement, so its splitting should be suppressed. Conversely, if the projected area is moderate or small, it is allowed to participate in subsequent splitting operations. This mechanism blocks the explosive proliferation of invalid primitives at the source of growth, avoiding the waste of computational resources.

[0102] Step S204: Perform a splitting or cloning operation based on the state of the unsuppressed 3D rendering primitive to obtain multiple split or cloned 3D rendering primitives.

[0103] Based on the state of the unsuppressed 3D rendering primitives, a splitting or cloning operation is performed, with one of the two operations chosen. The specific judgment rules and execution logic are as follows: First, the gradient magnitude and spatial fitting state of the 3D rendering primitives are combined for classification and screening. For 3D rendering primitives with large gradient magnitudes, a splitting operation is performed. If the geometric shape and texture features of the region where the 3D rendering primitive is located change drastically, a single 3D rendering primitive cannot achieve fine fitting. In this case, the splitting operation can decompose a single parent 3D rendering primitive into multiple smaller child 3D rendering primitives, effectively improving the detail expression capability of local areas. For 3D rendering primitives with inaccurate spatial position fitting and large positional deviations, a cloning operation is performed. This 3D rendering primitive cannot accurately fit the real surface of the target scene. The cloning operation will use this 3D rendering primitive as a template to copy and generate a new 3D rendering primitive. The new 3D rendering primitive inherits the core attributes of the original 3D rendering primitive, such as shape, orientation, color, and opacity, and makes minor adjustments to the spatial position, supplementing the primitive density in the local area and correcting the position fitting error.

[0104] By selectively performing splitting and cloning operations as described above, it is ensured that the 3D rendering primitives are mainly concentrated in complex terrain areas where accuracy truly needs to be improved, rather than being uniformly spread across the entire target scene.

[0105] Step S205: Based on the projected area of ​​each of the three-dimensional rendering primitives in the set of three-dimensional rendering primitives under the rendering viewpoint, a target three-dimensional rendering primitive is obtained by filtering from the set of three-dimensional rendering primitives. The set of three-dimensional rendering primitives includes multiple three-dimensional rendering primitives obtained by initialization and multiple three-dimensional rendering primitives that are split or cloned.

[0106] Although step S203 suppresses the splitting operations of certain 3D rendering primitives, some 3D rendering primitives may still become redundant due to historical reasons or optimization oscillations during long-term iterations. Step S205 introduces projected area as a core evaluation factor to simplify all 3D rendering primitives.

[0107] For example, the projected area can be used as a normalization factor or a penalty term.

[0108] Understandably, large-sized flat 3D rendering primitives that cover a large number of pixels but contribute little to detail have their relative importance reduced by dividing them by their projected area to prevent them from gaining an advantage in the selection process; while small-sized detailed 3D rendering primitives that cover few pixels but are crucial to key geometric structures have their relative importance increased.

[0109] The target 3D rendering primitives that are retained after screening constitute the essential subset of the target scene expression, which maintains the basic coverage of flat areas while fully preserving the geometric details of complex areas.

[0110] Step S206: Based on the target 3D rendering primitives, obtain the 3D scene model of the target scene.

[0111] In one optional implementation, steps S203 to S206 are iteratively performed until the 3D scene model meets preset convergence conditions, such as the loss function no longer decreasing or reaching the maximum number of iterations. Through this multi-stage iterative optimization, the 3D scene model can continuously approach the optimal representation of the real target scene through dynamic adjustments. The final 3D scene model perfectly reproduces the full-scale geometric features of the target scene, from vast flatlands to fine gravel, while maintaining extremely low memory usage and high rendering frame rate.

[0112] This application provides a 3D scene reconstruction method, which acquires multi-view images of the target scene and initializes multiple 3D rendering primitives based on these images. The initialized 3D rendering primitives suffer from two major drawbacks: they contain invalid impurity primitives and lack sufficient ability to reproduce details of the scene's uneven terrain. To supplement scene details, a splitting operation needs to be performed on the 3D rendering primitives to refine the geometric texture. However, not all primitives are suitable for splitting: continuous splitting of 3D rendering primitives with large projected areas generates a large amount of redundancy. Therefore, it is necessary to first determine whether to suppress the splitting operation based on the primitive's projected area. For 3D rendering primitives that are not suppressed from splitting, a splitting or cloning operation is performed based on their own state to generate a batch of refined 3D rendering primitives, effectively supplementing the detailed information of local uneven terrain. However, in the 3D rendering primitives after splitting and cloning, in addition to the effective 3D rendering primitives carrying scene details, a large number of low-contribution noise and redundant 3D rendering primitives are mixed in. Therefore, the rendering contribution of the initialized 3D rendering primitives and the splitting or cloning 3D rendering primitives under the rendering viewpoint is quantified, and the target 3D rendering primitives are selected. The selected target 3D rendering primitives retain scene details while eliminating noise redundancy. The 3D scene models built based on these target 3D rendering primitives have higher accuracy and better rendering effects. This effectively balances the overall distribution of 3D rendering primitives and reduces computational and memory overhead.

[0113] It is understandable that complex detail areas in the target scene, such as uneven terrain, typically have small-sized and densely distributed 3D rendering primitives. During the conventional 3D rendering primitive selection process, the importance of these small-sized, detailed 3D rendering primitives is easily diluted by large-area, large-sized 3D rendering primitives, making them prone to being mistakenly removed. The loss of these detailed 3D rendering primitives directly leads to incomplete local geometry and missing texture details, easily causing local underfitting of the 3D scene model, ultimately resulting in low overall 3D scene modeling accuracy. Therefore, after step S204 and before step S205, step S2051 is included: rendering a rendered image based on all 3D rendering primitives in the set along the rendering viewpoint. After step S2051 and before step S206, step S2052 is included: identifying areas with missing details based on the rendered image and generating new 3D rendering primitives in these areas.

[0114] For example, the 3D rendering primitive in step S2051 may include the 3D rendering primitive in step S202, the split 3D rendering primitive, and the cloned 3D rendering primitive.

[0115] For example, a block-based rasterizer can be used to project all the 3D rendering primitives in the 3D rendering primitive set onto the image plane after sorting them by depth, and accumulate color and transparency through the alpha blending formula to finally generate the rendered image from the rendering perspective.

[0116] The set of 3D rendering primitives contains all the 3D rendering primitives.

[0117] For example, by comparing the rendered image with real-world multi-view images, the photometric loss and geometric loss can be calculated, thereby guiding the parameter updates of the 3D rendering primitives.

[0118] In this context, "detail-deficient regions" refer to areas in a rendered image that exhibit geometric incompleteness, blurriness, depth discontinuity, and structural underfitting. Essentially, these are localized areas that significantly differ from realistic multi-view images.

[0119] For example, in the shaded areas of steep mountain slopes or in the ruts of muddy roads, if areas lacking detail, such as rendering holes, are detected, 3D rendering primitives are added to fill the geometric gaps. This targeted generation strategy avoids the noise interference caused by blindly adding 3D rendering primitives, ensuring the efficiency and accuracy of the completion operation.

[0120] "Directed generation" differs from existing random initialization or gradient-driven adaptive densification; its technical essence lies in a geometric compensation strategy based on visual error feedback. This application analyzes the explicit differences between the current rendering result and the real scene, precisely placing the newly generated 3D rendering primitives at visually perceived missing 3D spatial locations. This mechanism ensures that each newly added 3D rendering primitive has a clear geometric repair purpose, avoiding the generation of invalid 3D rendering primitives.

[0121] Understandably, the newly generated 3D rendering primitives in step S2052 are combined with the target 3D rendering primitives selected in step S205 to form an updated primitive set. This primitive set, as the output of the current iteration, can be used directly as the final 3D scene model or as the input basis for the next round of optimization.

[0122] Accordingly, step S206 includes: obtaining a 3D scene model of the target scene based on the new 3D rendering primitive and the target 3D rendering primitive.

[0123] This application provides a 3D scene reconstruction method, which acquires multi-view images of the target scene; initializes multiple 3D rendering primitives based on the multi-view images; judges and suppresses the splitting operation of some 3D rendering primitives based on the projected area of ​​each 3D rendering primitive under the rendering view, significantly reducing invalid computation and memory usage, solving the problem of resource waste in traditional uniform density strategies; then, the unsuppressed 3D rendering primitives are normally split to ensure the basic scene fitting ability; and then the initialized 3D rendering primitives, split or cloned 3D rendering primitives are used for forward rendering to obtain the rendered image; based on the projected area, the initialized 3D rendering primitives, split or cloned 3D rendering primitives are... The method uses a projection area-based evaluation approach to correct the shortcomings of traditional screening mechanisms, which often result in low weighting and accidental removal of small-sized detail 3D rendering primitives. This effectively preserves the geometric details of complex terrain. Simultaneously, it identifies areas with missing details by rendering images and generates new 3D rendering primitives accordingly. It also performs targeted detail completion for underfitting areas of the model. Finally, it merges the selected target 3D rendering primitives with the newly added 3D rendering primitives to complete scene modeling. By effectively balancing the overall distribution of 3D rendering primitives and reducing computational and memory overhead, it significantly improves the geometric fidelity of mixed terrain scenes and substantially enhances the overall modeling accuracy of 3D scene models.

[0124] It is understandable that although the above embodiments achieve precise spatial positioning of the new 3D rendering primitive through depth guidance, improper setting of the attribute parameters of the new 3D rendering primitive can still seriously affect the subsequent optimization effect. Specifically, if the new 3D rendering primitive directly inherits the parameters of the discarded 3D rendering primitive or neighboring 3D rendering primitives, it may bring the prior information that originally caused the fitting error (such as incorrect high-frequency color components or excessive opacity) into the new round of iterations, causing the optimization process to get stuck in local minima or produce oscillations. Conversely, if a completely random initialization method is used, the new 3D rendering primitive lacks effective optical prior guidance and requires a large number of iterations to converge from the disordered state to a reasonable value. This not only slows down the overall modeling speed, but may also destroy the rendering quality of the converged area in the early stage of training due to the appearance of random bright or highly occluded 3D rendering primitives. To solve this problem, after step S2052 and before step S206, this application also provides the following steps A1 to A2.

[0125] Step A1: Set the color attribute of the new 3D rendering primitive to the color information derived from the low-order spherical harmonic function.

[0126] For example, the low-order spherical harmonic function can be a 0th-order spherical harmonic function; the color information can be RGB values.

[0127] The RGB (Red, Green, Blue) values ​​derived from the 0th-order spherical harmonic function refer to the color values ​​that retain only the 0th-order coefficients (i.e., DC components) in the expansion of the spherical harmonic function. These color values ​​characterize the average basic radiance of the object's surface in all observation directions and do not contain any high-frequency color information that varies with the viewing angle.

[0128] The core principle behind choosing 0th-order rather than higher-order spherical harmonic coefficients is that while the newly generated 3D rendering primitives fill in the missing regions geometrically, their covariance matrices (shape and orientation) have not yet been fully optimized, remaining in an isotropic or approximately spherical initial state. If higher-order spherical harmonic coefficients are assigned at this point, the viewpoint-dependent colors expressed by the higher-order terms will produce severe artificial artifacts or high-frequency noise during rendering, interfering with the gradient calculation of the loss function, because the shape of the 3D rendering primitive has not yet matched the normal distribution of the real surface. By forcibly resetting the color to 0th-order RGB values, it is equivalent to giving the new 3D rendering primitive a pure, non-directional base color, allowing it to exist only as a low-frequency color filling the 3D rendering primitive in the early stages of training. As the shape gradually converges in subsequent iterations, the gradient naturally drives the growth of higher-order coefficients. This implicit constraint effectively avoids the introduction of visual mutations by the new 3D rendering primitives.

[0129] Step A2: Set the opacity parameter of the new 3D rendering primitive to the initial opacity value.

[0130] The initial opacity value is a preset small positive value, for example, it can be set to any value between 0.01 and 0.1.

[0131] Understandably, the opacity cannot be set to 0; otherwise, the 3D rendering primitive will not participate in rendering at all during alpha blending, resulting in a zero backpropagation gradient and rendering the primitive unupdateable. The opacity should also not be set to 1 or a large value, because the position of the new 3D rendering primitive is estimated based on the actual scene depth map, inevitably containing slight deviations. If it is completely opaque from the beginning, it is very easy to form abrupt color spots in the rendered image or obscure correctly fitted details, causing a surge in local errors. Using a small initial opacity value makes the new 3D rendering primitives semi-transparent in the early stages of training, and their contribution to the rendering results is weak and gradual. As optimization progresses, only those new 3D rendering primitives that are truly located on the correct geometric surface will gradually increase their opacity to a normal level due to gradient accumulation, while erroneous 3D rendering primitives with large positional deviations will remain low in opacity due to a lack of positive gradient feedback for a long time or even be removed by subsequent pruning mechanisms. This mechanism gives the 3D scene model an inherent self-correcting ability.

[0132] For example, when completing the missing details in muddy ruts in an off-road scene, 50 new 3D rendering primitives were generated in that area. The colors of these new primitives were uniformly reset to the average mud color (0th-order SH RGB value) obtained from multi-view images, and their opacity was initialized to 0.05. In the first few hundred iterations, these primitives appeared as a light mist covering the rut surface, without altering the original clear structure. However, as training continued, the 20 primitives at the bottom of the ruts quickly became opaque and richly colored due to their precise alignment with real geometry, while the 30 falsely generated primitives floating in the air remained semi-transparent and were eventually removed by the CDF pruning strategy. This process was smooth and fluid, without causing any flickering in the rendered image or abnormal jumps in the loss function.

[0133] It should be understood that the aforementioned 0th-order spherical harmonic function and specific range of opacity are only optional implementation methods. In other embodiments, a low-order (e.g., 1st-order) spherical harmonic function can also be used as the initial color value to retain a small amount of directional information, or the initial value can be dynamically set according to the local average opacity of the region where the new 3D rendering primitive is located.

[0134] On the one hand, by stripping high-level color information and setting low initial opacity, this application effectively prevents optimization oscillations and visual artifacts caused by the inheritance of incorrect priors or random initialization of new 3D rendering primitives, thus ensuring the numerical stability of the training process. On the other hand, this gentle initialization strategy provides a reasonable starting point for gradient ascent for new 3D rendering primitives, enabling them to quickly integrate into the model and accurately fit local details, significantly shortening the convergence time required for detail repair and improving the overall efficiency and robustness of 3D reconstruction for target scenes.

[0135] It is understandable that the aforementioned embodiments effectively solve the problem of redundant 3D rendering primitives in flat areas by suppressing the splitting operation and filtering the rendering contribution. However, during the simplification process, due to the discreteness of the evaluation index or the oscillation of the optimization process, some key 3D rendering primitives located at the terrain edge or microstructure may be mistakenly removed, resulting in local geometric collapse, texture blurring, or holes in the rendered image. If traditional global random point scattering or uniform resampling strategies are used to remedy this, not only is the computational efficiency low, but unnecessary noise interference is also easily introduced into the converged flat area. To address this problem, this application performs a targeted completion operation in step S2052, specifically including the following steps B1 to B4.

[0136] Step B1: Identify target pixels with geometric defects based on the cumulative opacity or depth difference corresponding to the rendered image.

[0137] Cumulative opacity refers to the sum of the opacities of multiple 3D rendering primitives from the rendering perspective. From the perspective of physical optics, when the line of sight passes through the target scene, if the cumulative opacity of all 3D rendering primitives along the line of sight is close to 1, it indicates that the line of sight is completely occluded by the scene surface, and the corresponding position has a complete geometric representation. Conversely, if the cumulative opacity is significantly lower than 1, it means that the line of sight has penetrated the target scene, and there is a geometric gap or poor surface fitting at the spatial position corresponding to that pixel.

[0138] Depth difference refers to the difference between the rendered depth value of a pixel in the rendered image and the actual depth value of a pixel in the actual scene depth map.

[0139] The actual depth values ​​in a real-world scene depth map are typically derived from ground-value depth estimates from multi-view images or point cloud depth data collected by LiDAR. When the deviation between the rendered depth value and the actual depth value of a pixel exceeds a preset depth tolerance, it indicates that the 3D rendering primitive at that location has failed to accurately fit the real surface, resulting in geometric misalignment or missing data.

[0140] For example, step B1 includes steps B11 to B12.

[0141] Step B11: For each pixel in the rendered image, calculate the pixel replacement probability using the missing transparency value as the sampling weight.

[0142] Wherein, the pixel's missing transparency value = 1 - the pixel's cumulative opacity; the pixel replenishment probability = the pixel's missing transparency value / the sum of the missing transparency values ​​of all pixels in the rendered image.

[0143] The formula for calculating the pixel supplementation probability of a pixel is as follows: ,in, The pixel supplement probability for pixel (x,y); The cumulative opacity of pixel (x,y); the image resolution of the rendered image is w1×w2.

[0144] Step B12: Randomly sample according to the pixel supplementation probability of each pixel, and select the obtained pixel as the target pixel with geometric missing.

[0145] Weighted random sampling is performed based on the pixel supplementation probability corresponding to each pixel: the higher the pixel supplementation probability, the greater the probability that the pixel will be selected for sampling. This method prioritizes pixels with missing geometry and texture, and determines the sampled pixels as target pixels, which serve as the basis for the position of new 3D rendering primitives in subsequent generation.

[0146] Step B2: Delineate the areas lacking detail based on the target pixels.

[0147] For example, a cumulative opacity threshold (e.g., 0.9) or a preset depth tolerance (e.g., 0.05 meters) can be set to mark pixels below the cumulative opacity threshold or above the preset depth tolerance as target pixels.

[0148] Understandably, operations are typically not performed on isolated pixels alone. Instead, connected component analysis or morphological dilation is applied to the target pixel, clustering adjacent pixels into continuous regions lacking detail. This is done to avoid false triggers caused by single-pixel noise, while ensuring the spatial continuity of the supplemented 3D rendering primitives, which is more conducive to fitting sheet-like geometric details.

[0149] Step B3: Extract the three-dimensional spatial coordinates and real texture color corresponding to the target pixel.

[0150] The method of obtaining three-dimensional spatial coordinates depends on the recognition metrics used: if the target pixel is identified based on cumulative opacity, the expected position of the pixel in the three-dimensional world can be calculated by back-projecting the camera parameters of the rendering viewpoint with the actual scene depth map; if the target pixel is obtained based on depth difference, the three-dimensional point corresponding to the actual depth value can be directly taken as the target coordinate.

[0151] The true texture color is the true RGB value of that location, sampled from multi-view images based on the camera projection relationship.

[0152] This step ensures that the newly generated 3D rendering primitives are not only precisely aligned with the missing areas in terms of geometry, but also inherit the characteristics of the real target scene in terms of appearance attributes.

[0153] Step B4: Based on 3D spatial coordinates and real texture colors, generate new 3D rendering primitives in areas where details are missing.

[0154] The new 3D rendering primitive is centered on the extracted 3D spatial coordinates, uses real texture color as the initial color attribute, and can be set to a small initial size and opacity to avoid causing abrupt visual interference to the surrounding converged 3D rendering primitives.

[0155] It should be understood that although this embodiment uses cumulative opacity and depth difference as the main identification criteria, in other embodiments, other rendering feedback indicators such as color residual and normal vector consistency can also be combined to comprehensively determine the areas of missing details.

[0156] For example, when reconstructing a target scene containing a gravel slope and muddy ruts, after the initial pruning optimization, the rendered image showed that the cumulative opacity deep in the rut depressions was only 0.6, far below the cumulative opacity threshold of 0.9, indicating a severe lack of geometric representation in this area. This region was automatically identified as a missing detail area, and dozens of 3D spatial coordinate points were extracted based on the actual scene depth map at the bottom of the ruts. Simultaneously, the corresponding soil texture color was sampled from multi-view images. Subsequently, new 3D rendering primitives were generated at these coordinate points. After hundreds of iterations of fine-tuning, these new 3D rendering primitives were quickly integrated into the 3D scene model, perfectly filling the geometric gaps in the ruts and restoring the rut details in the rendered image to clarity and sharpness.

[0157] This embodiment achieves precise repair of geometric defects in 3D scene models by establishing a detail loss identification and targeted generation mechanism based on rendering differences. On one hand, this embodiment utilizes cumulative opacity and depth difference as intuitive geometric integrity measures to accurately locate areas where details are lost due to insufficient pruning or optimization, avoiding computational waste and noise introduction caused by blindly patching. On the other hand, by extracting 3D spatial coordinates and real texture colors for targeted initialization, the new 3D rendering primitives possess extremely high prior quality, significantly shortening the convergence time of local details. This ensures that the final output 3D scene model maintains extreme lightweight design while still faithfully reproducing various complex micro-geometric features of the target scene.

[0158] Understandably, in mixed scenes containing both flat and locally complex undulating terrain, traditional filtering strategies often rely solely on gradient magnitude or opacity as evaluation criteria. This single-dimensional evaluation method has inherent flaws: large-size 3D rendering primitives covering flat areas such as deserts and snowfields, due to their massive number of pixels projected onto the image plane, are prone to gaining inflated weights in gradient accumulation or error statistics, even if their efficiency in representing geometric details is extremely low, and are thus largely retained during the filtering process; conversely, small-size 3D rendering primitives used to fit fine structures such as gravel and gullies, while crucial for restoring the realism of the target scene, are often overlooked due to their small pixel coverage and are easily mistakenly removed during the simplification stage. To improve this imbalance, this embodiment introduces rendering contribution as an evaluation criterion, and the corresponding step S205 specifically includes steps C1 to C2.

[0159] Step C1: For each 3D rendering primitive in the set of 3D rendering primitives, determine the rendering contribution of the 3D rendering primitive to the rendering effect of the target scene based on the projected area of ​​the 3D rendering primitive in the rendering view.

[0160] The rendering contribution of a 3D rendering primitive is a comprehensive metric that represents the amount of effective information contained in the overall rendering quality of the target scene from the rendering perspective.

[0161] For example, the projected area is a "normalization factor" or "inverse penalty term," meaning that when calculating rendering contribution, the projected area is used as a denominator or a negatively correlated parameter in the calculation. That is, the larger the projected area of ​​a 3D rendering primitive, the more its positive gain in rendering contribution is diluted or canceled out. Through this inverse adjustment, the score of 3D rendering primitives that cover large flat areas but lack substantial geometric information can be effectively suppressed, while the relative importance of small 3D rendering primitives that precisely fit the edges of complex terrain can be significantly increased. For instance, in an off-road scene, a 3D rendering primitive located in the center of open sand may have a large projected area, but its contribution to texture changes is negligible. After area normalization, its rendering contribution will be significantly reduced. Conversely, a 3D rendering primitive located in a rock crevice, although with a small projected area, carries crucial depth abrupt change information; after area normalization, its rendering contribution is actually higher than the former.

[0162] The above mechanism ensures that the screening process can identify the 3D rendering primitives that are truly valuable for reconstructing the simulated scene, beyond the apparent pixel coverage.

[0163] Step C2: Retain the target 3D rendering primitives in the 3D rendering primitive set whose rendering contribution meets the preset conditions.

[0164] The preset conditions can be a fixed contribution threshold, a dynamically adjusted retention ratio, or a truncation strategy based on ranking. Regardless of the specific form used, the core logic is to prioritize retaining 3D rendering primitives with high rendering contribution and eliminate or suppress 3D rendering primitives with low rendering contribution.

[0165] This embodiment fundamentally corrects the bias of inflated weights for large-size 3D rendering primitives in traditional methods by integrating the projection area depth into the calculation logic of rendering contribution. On the one hand, this embodiment significantly improves the retention rate of 3D rendering primitives in complex terrains, avoiding geometric damage and blurring caused by erroneous culling; on the other hand, it effectively suppresses the overfitting tendency of 3D scene models to flat areas, allowing limited computing resources and memory space to be adaptively allocated to scene parts requiring higher precision representation. Thus, while ensuring overall modeling accuracy, it achieves a balance between rendering efficiency and storage overhead.

[0166] Understandably, some 3D rendering primitives, despite having large projected areas, have extremely low opacity and hardly participate in color mixing, thus being ineffective 3D rendering primitives. Conversely, some 3D rendering primitives, while of moderate size, contribute negligibly to the rendering perspective due to being positioned at a grazing angle or being severely occluded by other objects. Evaluating based solely on projected area could easily lead to the incorrect retention of these inefficient 3D rendering primitives, while truly high-quality primitives are eliminated due to low overall scores. To avoid the shortcomings of evaluation based on a single metric, step C1 of this application is refined into steps C11 to C12.

[0167] Step C11: Obtain the opacity and view contribution of the 3D rendering primitive in the rendering view.

[0168] Opacity refers to the quantified value of a 3D rendering primitive's ability to block or transmit light during alpha blending.

[0169] For example, opacity typically ranges from 0 to 1, reflecting the presence and color intensity of the 3D rendering primitive as a geometric entity in the target scene. 3D rendering primitives with excessively low opacity often indicate that they have not yet converged to an effective surface fit, or that they are transitional artifacts created to fill gaps.

[0170] Viewpoint contribution refers to the degree to which a 3D rendering primitive is effectively observed from the rendering viewpoint. This parameter comprehensively considers the angle between the normal vector of the 3D rendering primitive and the viewing direction, the distance attenuation factor from the center of the 3D rendering primitive to the camera, and potential visibility weights. For example, when the normal of a flat 3D rendering primitive is nearly parallel to the viewing direction, although its projected volume remains unchanged, it has visually degenerated into a line or even become invisible. In this case, its viewpoint contribution should be significantly reduced.

[0171] Step C12: Calculate the rendering contribution of the 3D rendering primitives based on the projected area, opacity, and viewpoint contribution.

[0172] In this embodiment, the rendering contribution is calculated using the following logical combination: the product of opacity and viewpoint contribution is used as the numerator, and the projected area is used as the denominator, thus obtaining the effective visual information density per unit projected area. This can be expressed as:

[0173] , among which, I i Let α represent the rendering contribution of the i-th 3D rendering primitive, V be the training view set containing multiple rendering views, and α be the rendering contribution of the i-th 3D rendering primitive. i (v) represents the opacity of the i-th 3D rendering primitive at the rendering viewpoint v, where T i (v) represents the viewpoint contribution of the i-th 3D rendering primitive. Let be the projected area of ​​the i-th 3D rendering primitive.

[0174] Among them, the molecular part α i (v)×T i (v) Only 3D rendering primitives that possess both sufficient realism and clear visibility from the rendering perspective can achieve high score gains; denominator part This continues the projection area penalty mechanism from the above embodiments, preventing large-size 3D rendering primitives from monopolizing importance rankings simply by covering more pixels.

[0175] For example, suppose there are two 3D rendering primitives: 3D rendering primitive X is located in the center of a flat sandy area, with a projected area of ​​2000 pixels, but its opacity is only 0.1, and its viewpoint contribution is 0.5 due to the tilted viewpoint; 3D rendering primitive Y is located deep in a rock crevice, with a projected area of ​​only 50 pixels, but its opacity is 0.9, and its viewpoint contribution is 0.8. According to the above formula, the rendering contribution of 3D rendering primitive X is approximately (0.1 × 0.5) / 2000 = 0.000025, while the rendering contribution of 3D rendering primitive Y is approximately (0.9 × 0.8) / 50 = 0.0144. Obviously, although the projected area of ​​3D rendering primitive X is 40 times that of 3D rendering primitive Y, its rendering contribution is hundreds of times lower.

[0176] It should be understood that the above formula is only one possible implementation. In other implementations, weighted summation, exponential decay, or other nonlinear combination functions can also be used to fuse these three parameters, as long as the core logic maintains the constraint relationship between "negative correlation of projected area" and "positive correlation of opacity and viewpoint contribution". For example, for some scenarios that are not sensitive to changes in transparency, opacity can be used as a threshold filtering condition instead of a continuous product factor; or the viewpoint contribution can be replaced with a visibility mask based on a depth buffer.

[0177] This embodiment significantly improves the accuracy and robustness of rendering contribution by constructing a multi-dimensional rendering contribution calculation model that integrates projected area, opacity, and viewpoint contribution. On one hand, by introducing optical and observational state parameters, this embodiment effectively eliminates 3D rendering primitives that occupy screen space but contribute nothing to imaging, avoiding unnecessary consumption of computational resources. On the other hand, the mutual verification of multi-dimensional indicators ensures that rendering contribution is no longer misled by a single geometric feature, guaranteeing that in complex off-road scenes, whether it's a highly reflective snow surface or a low-texture muddy area, a reasonable importance ranking can be obtained based on its actual rendering performance.

[0178] To further enhance the adaptability of the 3D rendering primitive selection mechanism to scenes of varying complexity, this embodiment provides three parallel 3D rendering primitive retention strategies. It is understood that existing fixed-number pruning methods cannot adapt to changes in scene geometric complexity. For example, in 3D reconstruction for off-road environments, flat areas such as deserts and snowfields often coexist with complex areas such as mountains and ravines in the same scene, and the distribution density of effective information in the 3D scene model dynamically evolves as training iterations progress. Using a fixed retention number or a single threshold can easily lead to excessive retention of redundant 3D rendering primitives in simple areas, wasting computational power, or excessive pruning in complex areas, resulting in the loss of critical details. Therefore, further limitations are made to step C2 as follows. There are multiple ways to implement step C2; this embodiment provides, but is not limited to, the following three.

[0179] The first method of implementing step C2 includes steps D11 to D12.

[0180] Step D11: Normalize the rendering contribution of each 3D rendering primitive in the 3D rendering primitive set to obtain the retention probability of each 3D rendering primitive.

[0181] Step D12: Retain the target 3D rendering primitive with a retention probability greater than or equal to the preset retention threshold among multiple 3D rendering primitives.

[0182] Normalization refers to the process of mapping the rendering contribution of all 3D rendering primitives to the range [0,1], for example, through the Softmax function or Min-Max linear scaling.

[0183] The normalized value is the retention probability, which represents the relative importance level of the 3D rendering primitive among all 3D rendering primitives.

[0184] For example, the formula for calculating the retention probability is: Where N is the total number of all 3D rendering primitives. P i Let be the retention probability of the i-th 3D rendering primitive.

[0185] The preset retention threshold is a screening threshold, which can be set to, for example, 0.5 or other empirical values. When the retention probability of a certain 3D rendering primitive exceeds the preset retention threshold, it is determined to be the target 3D rendering primitive and retained. The retention or abandonment of each 3D rendering primitive depends only on whether its own quality meets the standard, and is not affected by fluctuations in the number of other 3D rendering primitives.

[0186] The second method of implementing step C2 includes steps D21 to D22.

[0187] Step D21: Based on the first number of 3D rendering primitives with non-zero rendering contribution in the 3D rendering primitive set, the preset sampling factor, and the total number of multiple 3D rendering primitives, obtain the second number.

[0188] Step D22: Use sampling without replacement to retain the second number of target 3D rendering primitives.

[0189] Among them, the first number of 3D rendering primitives with non-zero rendering contribution reflects the effective unit scale that actually participates in the expression of the target scene in the 3D scene model; the total number of all 3D rendering primitives includes a large number of redundant 3D rendering primitives that may have degenerated into transparent or invalid ones.

[0190] The calculation logic of the second number reflects the adaptive perception of the effective information density of the target scene, and its calculation formula can be expressed as: N'=N×f×(n non-zero / N), where N' is the second number, N is the total number, and n non-zero is the first number, and f is the preset sampling factor.

[0191] For example, the preset sampling factor can range from 0.5 to 0.7, which is used to control the overall pruning aggressiveness.

[0192] It is understandable that if there are many invalid 3D rendering primitives in the target scene (i.e., n) non-zero When the / N ratio is low, the number of elements to be retained will be reduced significantly to speed up the cleanup of redundancy; conversely, when most 3D rendering primitives are effective, the retention limit will be appropriately relaxed to protect details.

[0193] Sampling without replacement ensures that each selected 3D rendering primitive is unique, avoiding redundant computation. This strategy is particularly suitable for the mid-stage of training, enabling the model size to be dynamically adjusted based on the actual convergence state of the target scene while ensuring that the core geometry is not lost, thus achieving a rapid decrease in computational load.

[0194] The third method of implementing step C2 includes steps D31 to D33.

[0195] Step D31: Sort all 3D rendering primitives in the 3D rendering primitive set in descending order of rendering contribution to obtain the sorting result.

[0196] Step D32: Calculate the cumulative distribution function based on the sorting results.

[0197] The Cumulative Distribution Function (CDF) is used here to quantify the cumulative coverage of the overall rendering effect by the top K 3D rendering primitives with the highest rendering contribution.

[0198] After sorting all 3D rendering primitives from highest to lowest rendering contribution, the rendering contributions of each 3D rendering primitive are summed sequentially and divided by the sum of the total contributions to obtain a normalized cumulative curve.

[0199] For example, the formula for the cumulative distribution function is as follows: , where CDF(k) is the cumulative distribution function calculated based on the first k 3D rendering primitives in the sorting results.

[0200] Step D33: Retain the minimum number of target 3D rendering primitives that make the cumulative distribution function reach the preset cumulative threshold.

[0201] The preset cumulative threshold represents the proportion of the total scene information that is expected to be retained, and can be set to 0.80 to 0.85. This means that the core 3D rendering primitives that contribute 80% to 85% of the visual information will be automatically extracted, while the remaining long tail portion that contributes only a small amount will be discarded.

[0202] The CDF-based strategy does not concern itself with the absolute score or number of 3D rendering primitives, but only with whether the retained 3D rendering primitives are sufficient to support the main visual representation of the target scene. This strategy is particularly suitable for the later fine-tuning stage of training, which can strictly lock the lower limit of modeling accuracy while maximally removing redundant 3D rendering primitives that are not completely ineffective but have extremely low marginal benefits for improving overall image quality, thus achieving the optimal balance between accuracy and efficiency.

[0203] Assume the total number of 3D rendering primitives is N=10; sort all 3D rendering primitives from highest to lowest according to their rendering contribution, and the sorting results are as follows: I1=25, I2=20, I3=15, I4=12, I5=8, I6=7, I7=5, I8=4, I9=3, I10=1. The preset cumulative threshold is 0.80; calculate the cumulative distribution function CDF(k). When k=4, CDF(4)=0.72<0.80, which is not satisfied; when k=5, CDF(5)=0.80=0.80, which meets the requirements. Then, retain the minimum number of 3D rendering primitives that make the cumulative distribution function reach the preset cumulative threshold as the top 5 3D rendering primitives in the sorting result.

[0204] It should be understood that the three implementation methods mentioned above are not mutually exclusive and can be flexibly combined or switched according to the training stage or scene type in practical engineering applications. For example, the first implementation method can be used to quickly remove obvious noise in the early stage of training, the second implementation method can be switched to adaptively compress the model size in the middle stage, and the third implementation method can be used for final accuracy alignment in the later stage. This multi-dimensional preservation strategy system enables this application to cope with various target scenes from open deserts to broken mountains, ensuring high-fidelity restoration of 3D scene models while reducing memory usage and inference latency.

[0205] It is understandable that while the splitting operation suppression, rendering contribution screening, and detail completion operations in the aforementioned embodiments can optimize the distribution of 3D rendering primitives within a single training phase, a single optimization often fails to reach the global optimum when facing target scenes containing large areas of flat terrain and locally complex uneven terrain, such as off-road environments. For example, the first round of pruning may inadvertently damage some minor geometric features while removing redundant 3D rendering primitives in flat areas, and the subsequent directional completion, although correcting the missing features, may also introduce new 3D rendering primitives with small positional deviations or inaccurate attributes. Without multi-stage iterative feedback and dynamic adjustment, these local errors will gradually accumulate, causing the 3D scene model to eventually fall into the predicament of overfitting or underfitting. To solve this problem, this application completes multi-stage iterative optimization through step S206, which specifically includes steps E1 to E6. Specifically, the process of creating the 3D scene model of the target scene is not completed in one go, but is achieved through iterative loops.

[0206] Step E1: Based on the projected area of ​​each 3D rendering primitive in the primitive set under the rendering viewpoint, determine the rendering contribution of each 3D rendering primitive to the rendering effect of the target scene; the primitive set includes at least the target 3D rendering primitive.

[0207] The primitive set includes new 3D rendering primitives and target 3D rendering primitives. Specifically, the primitive set is a container for all valid 3D rendering primitives participating in the optimization training within the current iteration cycle. It carries the results of the previous optimization round and the supplementary information added in this round, serving as a link between the previous and subsequent training phases.

[0208] The target 3D rendering primitives are those with high rendering contribution retained after being filtered through the aforementioned embodiments, representing the main structure and core details of the target scene; the new 3D rendering primitives are primitives generated specifically for areas lacking detail, used to fill geometric gaps. Merging the two into a unified primitive set allows subsequent optimizations to be performed on a complete data basis, avoiding inconsistencies in optimization caused by data fragmentation.

[0209] Step E2: Retain the 3D rendering primitives in the primitive set whose rendering contribution meets the preset conditions.

[0210] The preset conditions can be consistent with the threshold, sampling factor, or cumulative distribution function strategy described in the previous embodiments, or they can be adaptively adjusted according to the characteristics of the current iteration stage, thereby achieving continuous refinement of the primitive set.

[0211] Step E3: Detect whether the candidate scene model composed of the retained 3D rendering primitives meets the preset convergence conditions.

[0212] The preset convergence condition refers to the termination criterion used to determine whether the training of the candidate scene model has reached a stable state or the expected quality. In this embodiment, the preset convergence condition can take various forms. For example, it can monitor the rate of change of the loss function (such as the weighted sum of photometric loss and structural similarity loss) with the number of iterations. When the rate of change is lower than a preset threshold (such as lim10) for several consecutive iterations, the condition is considered closed. -5 When the candidate scene model has converged, it can be determined that the model has converged; or, a maximum iteration limit can be set (e.g., 30,000 times), and training can be forcibly stopped when the iteration limit is reached to prevent overfitting; or, the change in the number of 3D rendering primitives can be monitored, and convergence can be considered when the total number of 3D rendering primitives tends to stabilize and the rendering quality no longer improves significantly.

[0213] It should be understood that the above convergence criteria are only examples, and in practical applications, they can be flexibly combined and used according to the specific complexity of the off-road scenario and computing resources.

[0214] After completing one round of primitive updates and filtering, it is necessary to determine whether the candidate scene model meets the preset convergence conditions. If the candidate scene model meets the preset convergence conditions, then the candidate scene model is a 3D scene model. At this time, the output 3D scene model is the final result after multiple rounds of closed-loop optimization, and its 3D rendering primitive distribution has achieved a balance between lightweighting in flat areas and high precision in complex areas.

[0215] Step E4: If the preset convergence condition is met, the candidate scene model is determined to be the 3D scene model.

[0216] Step E5: If the preset convergence condition is not met, return to step S203.

[0217] Starting with step S203, which suppresses the splitting operation, the process proceeds through steps S204 (splitting and cloning), S2051 (forward rendering), S205 (primitive selection), S2052 (detail completion), and E1 (contribution re-evaluation). Finally, the convergence status determines whether to initiate a new iteration. In each new iteration, as the candidate scene model increasingly better fits the geometric features of the target scene, the criteria for suppressing the splitting operation in step S203, the rigor of selection in step S2051, and the targeting accuracy of completion in step S2052 all dynamically evolve, thereby driving the 3D scene model to continuously approximate the realistic representation of the target scene.

[0218] For example, when reconstructing an off-road course containing scree slopes and flat sand, the first iteration may mainly cover the basic terrain and remove large areas of redundant primitives; the second iteration fills in the missing 3D rendering primitives in the scree gaps through depth guidance, but at the same time finds that some of the newly added 3D rendering primitives overlap with the surrounding original primitives; the third iteration removes these overlapping 3D rendering primitives during the re-evaluation process and further fine-tunes the shape and color parameters of the remaining 3D rendering primitives; until the Gth iteration, when the loss function curve tends to flatten and the difference between the rendered image and the real photo is reduced to an acceptable range, the system determines that it has converged and outputs the final 3D rendering model.

[0219] On the one hand, the embodiments of this application effectively eliminate the local errors and noise interference that may be introduced by a single operation through multiple rounds of iteration, ensuring the stability and geometric accuracy of the final output of the 3D scene model; on the other hand, the clear convergence criteria avoid the waste of computing power caused by invalid iterations, enabling the candidate scene model to stop training in a timely manner while meeting the quality requirements, which significantly improves the engineering efficiency and practicality of 3D reconstruction for target scenes.

[0220] It is understandable that if only the projected area is used as the criterion for suppressing splitting (for example, stipulating that splitting is prohibited if the projected area exceeds 500 pixels), then this threshold will be strongly coupled with the image resolution of the acquisition device. When the application scenario switches from high-definition monitoring to low-resolution embedded devices, or from close-range fine scanning to long-range macroscopic observation, the fixed threshold will completely fail, resulting in either the inability to effectively suppress the growth of large 3D rendering primitives, or the accidental damage to detailed 3D rendering primitives that should split. To address the above problems, this application implements adaptive determination through step S203, specifically including the following steps F1 to F2.

[0221] Step F1: Determine the ratio of the projected area of ​​the 3D rendering primitive to the total area of ​​the rendered image from the rendering perspective.

[0222] The total area of ​​the rendered image refers to the total number of effective pixels on the imaging plane from the rendering viewpoint. The total area of ​​the rendered image depends on the camera's field of view and output resolution settings.

[0223] The proportional relationship represents the relative spatial share of a single 3D rendering primitive in the total area.

[0224] Step F2: If the scale is greater than or equal to the preset scale threshold, the 3D rendering primitive is prohibited from performing splitting operations.

[0225] The preset ratio threshold is the dividing line between "overgrowth in flat areas" and "normal geometric fit".

[0226] When the projected area of ​​a certain 3D rendering primitive reaches or exceeds the threshold, it is determined that the 3D rendering primitive is covering a flat surface that is large enough relative to the current field of view, such as a distant desert horizon or a flat road surface nearby. Its existing geometric expression accuracy is sufficient to meet the visual requirements. Continuing to split will only produce a large number of redundant 3D rendering primitives that overlap and do not substantially improve the image quality. Therefore, its state is forcibly locked to block subsequent splitting operations.

[0227] For example, the preset scale threshold can be set to 1 / 5000. This means that if the projected area of ​​a 3D rendering primitive in the current rendered image exceeds one five-thousandth of the total area, the 3D rendering primitive will be marked as prohibited from splitting.

[0228] It should be understood that 1 / 5000 is only an example value. In actual engineering applications, it can be flexibly configured according to the terrain complexity of the target scene: for pure desert or snowfield scenes with extremely simple terrain, the preset ratio threshold can be lowered to 1 / 8000 to more aggressively suppress redundancy; while for mountain scenes with fragmented terrain and few large flat areas, the preset ratio threshold can be relaxed to 1 / 3000 to avoid underfitting caused by excessive constraints.

[0229] This embodiment transforms the suppression condition of the splitting operation from projected area to a proportional relationship, achieving scene independence and scale adaptation of the source suppression mechanism. On one hand, this embodiment completely decouples the binding relationship between algorithm parameters and hardware resolution, enabling the same modeling method to be seamlessly transferred to different acquisition platforms such as UAV aerial photography, vehicle surround view, and handheld scanning, without needing to recalibrate the preset proportional threshold for each resolution. On the other hand, the determination based on the proportional relationship more accurately reflects the marginal contribution rate of 3D rendering primitives to the representation of the target scene, fundamentally blocking the path of exponential explosive growth of 3D rendering primitives in flat areas during training iterations. Compared to post-pruning strategies, this significantly saves the gradient calculation and memory allocation overhead caused by invalid splitting, significantly improving overall training efficiency.

[0230] In one alternative implementation, a composite photometric loss function is used as a constraint during the iterative optimization of the candidate scene model. The formula for calculating the composite photometric loss function is L=(1-λ)L1+λL. SSIM Where L1 is the pixel-by-pixel loss; L SSIM λ represents the structural similarity loss; λ is the balance coefficient.

[0231] For example, λ=0.2.

[0232] The embodiments of this application will be described below in conjunction with the training phase of the candidate scene model.

[0233] The three-dimensional scene reconstruction method provided in this application adopts a multi-stage closed-loop process, including pre-training and densification, a first pruning stage, a depth-guided probability resampling stage, a second pruning stage, and iterative convergence optimization. It includes steps S201 to S206, which are described in detail below.

[0234] After obtaining multi-view images of the target scene, the pre-training and densification stage can be entered. The pre-training and densification stage consists of step S202, and steps S203, S204, and S2051, which are executed cyclically. Step S202 is used to initialize the 3D rendering primitives. During the iteration process, step S203 determines whether the 3D rendering primitives are prohibited from splitting, step S204 performs splitting and cloning operations, and step S2051 completes forward rendering. For example, the pre-training and densification phase performs a total of 7000 iterations. During these 7000 iterations, steps S203, S204, and S205 are continuously executed in a loop to complete each round of routine optimization. After every 100 routine iterations, an additional round of densification is performed. Based on step S204, 3D rendering primitives with large gradient magnitudes are split, 3D rendering primitives with positional deviations are cloned, and invalid 3D rendering primitives with an opacity less than 0.005 are simultaneously removed. After 7000 iterations, the pre-training and densification phase ends, and the first pruning phase begins.

[0235] The first pruning stage includes step S205, which uses a sampling-without-replacement method to simplify the 3D rendering primitives. After the first pruning stage is completed, the spatial position of the retained 3D rendering primitives remains unchanged, their color attributes are set to the color information derived from the low-order spherical harmonic function, and their opacity parameters are set to the initial opacity value. Then, using the target 3D rendering primitive as the 3D rendering primitive in step S203, 3000 iterations of training are performed on steps S203, S204, S2051, S205, and S2052 as a whole.

[0236] After the first pruning phase and subsequent 3000 iterations, a depth-guided probabilistic resampling phase is executed. This phase includes step S2052, which compensates for the erroneously removed 3D rendering primitives in the preceding pruning operation. Following the depth-guided probabilistic resampling phase, the original and new 3D rendering primitives are used as the 3D rendering primitives in step S203. For steps S203, S204, S2051, S205, and S2052, 2000 iterations of training are performed to gradually fit the local geometric and texture features to the new 3D rendering primitives.

[0237] After 2000 iterations of training, accumulating to the 12000th iteration, a second pruning phase is executed, which includes step S206. After the second pruning phase, using the retained 3D rendering primitives as the 3D rendering primitives from step S203, 1000 iterations of training are performed for steps S203, S204, S2051, S205, S2052, and S206 to obtain the 3D scene model of the target scene.

[0238] The above describes a three-dimensional scene reconstruction method provided by the embodiments of this application. The following describes the apparatus for performing the above three-dimensional scene reconstruction method.

[0239] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of a three-dimensional scene reconstruction device provided in an embodiment of this application. Figure 3 As shown, the 3D scene reconstruction device includes:

[0240] Image acquisition module 301 is used to acquire multi-view images of a target scene, the target scene including flat terrain and locally uneven terrain;

[0241] Primitive generation module 302 is used to initialize multiple three-dimensional rendering primitives based on the multi-view image;

[0242] The split suppression module 303 is used to determine, for each of the three-dimensional rendering primitives, whether to perform a split suppression operation on the three-dimensional rendering primitive based on the projected area of ​​the three-dimensional rendering primitive in the rendering view.

[0243] The split-clone execution module 304 is used to perform a split or clone operation based on the unsuppressed state of the three-dimensional rendering primitive to obtain multiple split or cloned three-dimensional rendering primitives.

[0244] The primitive filtering module 305 is used to filter out target three-dimensional rendering primitives from the three-dimensional rendering primitive set based on the projected area of ​​each three-dimensional rendering primitive in the three-dimensional rendering primitive set under the rendering view. The three-dimensional rendering primitive set includes multiple three-dimensional rendering primitives initialized and multiple three-dimensional rendering primitives split or cloned.

[0245] The model generation module 306 is used to obtain a three-dimensional scene model of the target scene based on the target three-dimensional rendering primitive.

[0246] In one alternative implementation, it also includes:

[0247] The forward rendering module is used to render a rendered image by forward rendering along the rendering viewpoint based on all the three-dimensional rendering primitives in the set of three-dimensional rendering primitives.

[0248] The detail completion module is used to identify areas of missing details based on the rendered image and generate new 3D rendering primitives in the areas of missing details.

[0249] In one alternative implementation, the model generation module includes:

[0250] The scene model acquisition unit is used to obtain a three-dimensional scene model of the target scene based on the new three-dimensional rendering primitive and the target three-dimensional rendering primitive.

[0251] In one alternative implementation, it also includes:

[0252] The first parameter setting module is used to set the color attribute of the new 3D rendering primitive to the color information derived from the low-order spherical harmonic function;

[0253] The second parameter setting module is used to set the opacity parameter of the new 3D rendering primitive to the initial opacity value.

[0254] In one alternative implementation, the detail completion module includes:

[0255] The pixel identification unit is used to identify target pixels with geometric defects based on the cumulative opacity or depth difference corresponding to the rendered image. The cumulative opacity refers to the sum of the opacities of multiple three-dimensional rendering primitives under the rendering viewpoint, and the depth difference refers to the difference between the rendering depth value corresponding to the pixel in the rendered image and the actual depth value corresponding to the pixel in the actual scene depth map.

[0256] A region delineation unit is used to delineate the region lacking detail based on the target pixel;

[0257] An information extraction unit is used to extract the three-dimensional spatial coordinates and real texture color corresponding to the target pixel;

[0258] Generate primitive units, which are used to generate the new 3D rendering primitives in the areas where details are missing, based on the 3D spatial coordinates and the real texture colors.

[0259] In one optional implementation, the primitive filtering module includes:

[0260] A contribution determination unit is used to determine the rendering contribution of each of the three-dimensional rendering primitives in the set of three-dimensional rendering primitives to the rendering effect of the target scene based on the projected area of ​​the three-dimensional rendering primitive under the rendering view.

[0261] The primitive unit is reserved to retain the target 3D rendering primitive in the 3D rendering primitive set whose rendering contribution meets the preset conditions.

[0262] In one alternative implementation, determining the contribution unit includes:

[0263] The parameter acquisition subunit is used to acquire the opacity and view contribution of the 3D rendering primitive under the rendering viewpoint.

[0264] A contribution calculation subunit is used to calculate the rendering contribution of the 3D rendering primitive based on the projected area, the opacity, and the viewpoint contribution.

[0265] In one alternative implementation, the reserved primitive unit includes at least one of the following:

[0266] The retention probability processing subunit is used to normalize the rendering contribution of each 3D rendering primitive in the set of 3D rendering primitives to obtain the retention probability of each 3D rendering primitive; and to retain the target 3D rendering primitive among the multiple 3D rendering primitives whose retention probability is greater than or equal to a preset retention threshold.

[0267] The sampling subunit without replacement is used to obtain a second number based on a first number of three-dimensional rendering primitives in the set of three-dimensional rendering primitives whose rendering contribution is non-zero, a preset sampling factor, and the total number of three-dimensional rendering primitives in the set of three-dimensional rendering primitives; and to retain the second number of target three-dimensional rendering primitives by sampling without replacement.

[0268] The distribution function calculation subunit is used to sort all the three-dimensional rendering primitives in the set of three-dimensional rendering primitives in descending order of their rendering contribution to obtain a sorting result; calculate the cumulative distribution function based on the sorting result; and retain the minimum number of target three-dimensional rendering primitives that make the cumulative distribution function reach a preset cumulative threshold.

[0269] In one alternative implementation, the model generation module includes:

[0270] The contribution determination unit is used to determine the rendering contribution of each three-dimensional rendering primitive to the rendering effect of the target scene based on the projected area of ​​each three-dimensional rendering primitive in the primitive set under the rendering view; the primitive set includes at least the target three-dimensional rendering primitive.

[0271] Retain rendering primitive units, used to retain 3D rendering primitives in the primitive set whose rendering contribution meets preset conditions;

[0272] The convergence detection module is used to detect whether the candidate scene model composed of the retained 3D rendering primitives meets the preset convergence conditions.

[0273] A model determination module is used to determine the candidate scene model as the 3D scene model if the preset convergence condition is met.

[0274] The triggering unit is used to trigger the re-execution of the split suppression module if the preset convergence condition is not met.

[0275] In one alternative implementation, the split suppression module includes:

[0276] A proportional unit is used to determine the proportional relationship between the projected area of ​​the 3D rendering primitive under the rendering viewpoint and the total area of ​​the rendered image.

[0277] A split-prohibition unit is used to prevent the 3D rendering primitive from performing a split operation if the ratio is greater than or equal to a preset ratio threshold.

[0278] This application also provides an electronic device in its embodiments. (See reference...) Figure 4 The diagram illustrates a structural schematic of an electronic device suitable for implementing the 3D scene reconstruction method in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, in-vehicle terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 4 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0279] like Figure 4 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 into a random access memory (RAM) 403. When the electronic device is powered on, the RAM 403 also stores various programs and data required for the operation of the electronic device. The processing unit 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0280] Typically, the following devices can be connected to I / O interface 405: input devices 406 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 407 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 408 including, for example, memory cards, hard drives, etc.; and communication devices 409. Communication device 409 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 4Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have instead.

[0281] This application also provides a computer program product, including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the three-dimensional scene reconstruction methods provided in this application.

[0282] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the three-dimensional scene reconstruction methods provided in this application.

[0283] This application also provides a vehicle, including the three-dimensional scene reconstruction device as described in the above embodiments, or including the electronic device as described in the above embodiments.

[0284] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0285] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0286] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0287] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A method for reconstructing a three-dimensional scene, characterized in that, include: Acquire multi-view images of a target scene, the target scene including flat terrain and locally uneven terrain; Multiple 3D rendering primitives are obtained based on the multi-view image initialization; For each of the three-dimensional rendering primitives, based on the projected area of ​​the three-dimensional rendering primitive in the rendering view, it is determined whether to perform the operation of suppressing the splitting of the three-dimensional rendering primitive. Perform a splitting or cloning operation based on the unsuppressed state of the 3D rendering primitive to obtain multiple split or cloned 3D rendering primitives. Based on the projected area of ​​each of the three-dimensional rendering primitives in the set of three-dimensional rendering primitives under the rendering view, a target three-dimensional rendering primitive is obtained by filtering from the set of three-dimensional rendering primitives. The set of three-dimensional rendering primitives includes multiple three-dimensional rendering primitives obtained by initialization and multiple three-dimensional rendering primitives that are split or cloned. Based on the target 3D rendering primitives, a 3D scene model of the target scene is obtained.

2. The three-dimensional scene reconstruction method according to claim 1, characterized in that, Before obtaining the 3D scene model of the target scene based on the target 3D rendering primitives, the process further includes: A rendered image is obtained by rendering all the three-dimensional rendering primitives in the set of three-dimensional rendering primitives along the rendering viewpoint in the forward direction. Based on the rendered image, identify areas lacking detail, and generate new 3D rendering primitives in those areas.

3. The three-dimensional scene reconstruction method according to claim 2, characterized in that, The process of obtaining a 3D scene model of the target scene based on the target 3D rendering primitives includes: Based on the new 3D rendering primitive and the target 3D rendering primitive, a 3D scene model of the target scene is obtained.

4. The three-dimensional scene reconstruction method according to claim 3, characterized in that, After generating new 3D rendering primitives in the areas lacking detail, and before obtaining the 3D scene model of the target scene based on the new 3D rendering primitives and the target 3D rendering primitives, the process further includes: Set the color attribute of the new 3D rendering primitive to the color information derived from the low-order spherical harmonic function; Set the opacity parameter of the new 3D rendering primitive to the initial opacity value.

5. The three-dimensional scene reconstruction method according to claim 2, characterized in that, The step of identifying regions of missing detail in the rendered image and generating new 3D rendering primitives in these regions includes: Based on the cumulative opacity or depth difference corresponding to the rendered image, target pixels with geometric defects are identified. The cumulative opacity refers to the sum of the opacities of multiple three-dimensional rendering primitives under the rendering viewpoint. The depth difference refers to the difference between the rendering depth value corresponding to the pixel in the rendered image and the actual depth value corresponding to the pixel in the actual scene depth map. The region of missing detail is defined based on the target pixel; Extract the three-dimensional spatial coordinates and real texture color corresponding to the target pixel; Based on the three-dimensional spatial coordinates and the real texture colors, the new three-dimensional rendering primitives are generated in the areas where details are missing.

6. The three-dimensional scene reconstruction method according to claim 1, characterized in that, The process of selecting target 3D rendering primitives from the 3D rendering primitive set based on the projected area of ​​each 3D rendering primitive in the rendering viewpoint includes: For each of the three-dimensional rendering primitives in the set of three-dimensional rendering primitives, the rendering contribution of the three-dimensional rendering primitive to the rendering effect of the target scene is determined based on the projected area of ​​the three-dimensional rendering primitive under the rendering view. Retain the target 3D rendering primitives in the set of 3D rendering primitives whose rendering contribution meets the preset conditions.

7. The three-dimensional scene reconstruction method according to claim 6, characterized in that, The step of determining the rendering contribution of the 3D rendering primitive to the rendering effect of the target scene based on the projected area of ​​the 3D rendering primitive under the rendering view includes: Obtain the opacity and view contribution of the 3D rendering primitive under the rendering viewpoint; The rendering contribution of the 3D rendering primitive is calculated based on the projected area, the opacity, and the viewpoint contribution.

8. The three-dimensional scene reconstruction method according to claim 6, characterized in that, The target 3D rendering primitives in the set of 3D rendering primitives whose rendering contribution meets the preset conditions include at least one of the following: Normalize the rendering contribution of each 3D rendering primitive in the set of 3D rendering primitives to obtain the retention probability of each 3D rendering primitive; retain the target 3D rendering primitives whose retention probability is greater than or equal to a preset retention threshold among the multiple 3D rendering primitives. A second number is obtained based on the first number of 3D rendering primitives in the set of 3D rendering primitives whose rendering contribution is non-zero, a preset sampling factor, and the total number of 3D rendering primitives in the set of 3D rendering primitives; The second number of target 3D rendering primitives are retained by sampling without replacement; The sorting result is obtained by arranging all the three-dimensional rendering primitives in the set of three-dimensional rendering primitives in descending order of their rendering contribution. Calculate the cumulative distribution function based on the sorting results; retain the minimum number of target 3D rendering primitives that make the cumulative distribution function reach a preset cumulative threshold.

9. The three-dimensional scene reconstruction method according to any one of claims 1 to 8, characterized in that, The process of obtaining a 3D scene model of the target scene based on the target 3D rendering primitives includes: Based on the projected area of ​​each 3D rendering primitive in the primitive set under the rendering viewpoint, the rendering contribution of each 3D rendering primitive to the rendering effect of the target scene is determined; the primitive set includes at least the target 3D rendering primitive. Retain the 3D rendering primitives in the primitive set whose rendering contribution meets the preset conditions; Detect whether the candidate scene model composed of the retained 3D rendering primitives satisfies the preset convergence condition; If the preset convergence condition is met, the candidate scene model is determined to be the 3D scene model; If the preset convergence condition is not met, return to the execution step and determine whether to perform the operation to suppress the splitting of the 3D rendering primitive based on the projected area of ​​the 3D rendering primitive in the rendering view.

10. The three-dimensional scene reconstruction method according to any one of claims 1 to 8, characterized in that, The step of determining whether to perform the operation to suppress the splitting of the 3D rendering primitive based on the projected area of ​​the 3D rendering primitive in the rendering view includes: Determine the ratio between the projected area of ​​the 3D rendering primitive under the rendering viewpoint and the total area of ​​the rendered image; If the ratio is greater than or equal to a preset ratio threshold, the 3D rendering primitive is prohibited from performing a splitting operation.

Images