A three-dimensional scene reconstruction method for an automatic driving vehicle scene

Abstract

The application relates to the technical field of three-dimensional reconstruction, and relates to a three-dimensional scene reconstruction method for an automatic driving vehicle-mounted scene, which comprises the following steps: acquiring multi-source sensor data collected by an automatic driving vehicle; performing spatial initialization on a three-dimensional Gaussian primitive set, and taking points in a point cloud as initial three-dimensional Gaussian primitive centers; associating preset key attributes of the three-dimensional Gaussian primitive with random variables, and calculating internal uncertainty signals of each primitive; performing parameter updating and adaptive life cycle management on initial geometric parameters and initial appearance parameters of the three-dimensional Gaussian primitive set until updated three-dimensional Gaussian primitive sets are obtained after the parameter updating meets preset convergence conditions; and projecting the updated three-dimensional Gaussian primitive sets to a target visual angle plane through a differentiable rasterization rendering pipeline to generate and output a reconstructed three-dimensional scene image under the target visual angle by using the updated three-dimensional Gaussian primitive sets after iterative optimization. The application can improve the timeliness and accuracy of three-dimensional scene reconstruction.

Description

Technical Field

[0001] This invention belongs to the field of 3D reconstruction technology, and in particular relates to a 3D scene reconstruction method for autonomous driving vehicle scenarios. Background Technology

[0002] With the rapid development of autonomous driving technology, onboard perception systems (such as cameras and LiDAR) need to collect multi-view environmental data in real time and quickly and accurately reconstruct the surrounding 3D scene to provide reliable spatial support for vehicle decision-making such as obstacle avoidance, lane keeping, and response to emergencies. Currently, 3D scene reconstruction technology used for Novel View Synthesis (NVS) is undergoing a paradigm shift from "implicit" representation to "explicit" representation.

[0003] Early implicit representation techniques, such as Neural Radiation Fields (NeRF), learned continuous five-dimensional functions through multilayer perceptrons (MLPs) to map spatial coordinates and viewpoint orientation into volume density and color. Although NeRF can synthesize realistic new viewpoint images, its training and rendering speeds are extremely slow, and it is difficult to perform real-time scene editing, which cannot meet the stringent real-time requirements of autonomous driving.

[0004] To address the aforementioned issues, explicit representation techniques, such as 3D Gaussian Splatting (3DGS), have emerged. 3DGS deconstructs a 3D scene into millions of discrete explicit primitives (i.e., 3D Gaussian ellipsoids), each defined by geometric parameters (position, rotation, scale) and appearance parameters (color, opacity). During rendering, 3DGS employs a differentiable rasterization rendering pipeline to project the 3D Gaussian primitives onto a 2D image plane. Pixel colors are drawn using GPU-based fast sorting and alpha blending algorithms, achieving real-time rendering. During optimization (training), the system continuously adjusts the parameters of all primitives using gradient descent, calculates gradients using photometric losses (such as L1 Loss and D-SSIM Loss), and updates parameters using optimizers (such as Adam). In addition, to adaptively adjust scene details, 3DGS also includes a primitive lifecycle management mechanism based on heuristics, such as densification (splitting or copying) when the primitive's position gradient exceeds a threshold, or pruning when the opacity is below a threshold.

[0005] While 3DGS achieves real-time rendering in general scenarios, its application in autonomous driving scenarios presents significant drawbacks due to the unique characteristics of high-speed vehicle movement, noisy sensor data, complex and variable road conditions (such as rain, backlighting, and occlusion), and limited onboard computing power. Existing technologies employ an indiscriminate gradient update strategy for all primitives during optimization, failing to differentiate gradient reliability. In autonomous driving scenarios, unreliable gradients from "ambiguous regions" (such as road surface reflections in rain or obstacle edges under backlighting) directly contaminate the geometric parameters of core obstacles (such as children or bicycles). This indiscriminate update method leads to significant deviations in the reconstructed positions of core obstacles (often exceeding 0.3 meters), exceeding the industry accuracy threshold for obstacle recognition in autonomous driving, increasing the risk of misjudgment. Furthermore, existing technologies are prone to "floating object" artifacts (such as misreconstructing tree shadows or road surface water stains as obstacles) and "fogging" artifacts (such as obscuring real lane lines or curbs). These artifacts can cause autonomous driving systems to fail to accurately distinguish between real obstacles and invalid noise, leading to dangerous situations such as "false braking" (mistaking shadows for obstacles) or "missed detection" (missing the detection of guardrails obscured by fogging artifacts). Furthermore, the core requirement for 3D reconstruction in autonomous driving is "high-density reconstruction of near-distance obstacles and lightweight simplification of distant backgrounds." However, existing technologies based on heuristic primitive management mechanisms with fixed thresholds suffer from two major problems: first, "blind pruning," which easily misjudges key primitives of thin-structure obstacles (such as guardrails and bicycle frames) as useless and removes them, resulting in missing obstacle shapes; second, "over-densification," which indiscriminately adds primitives to blurred areas in the distance (such as the horizon background), not only ineffectively consuming a large amount of onboard GPU computing power (typically more than 30%), but also leading to insufficient optimization resources in core areas, making it difficult to achieve optimal performance with limited onboard hardware resources. Finally, autonomous driving requires a millisecond-level closed-loop response for "data acquisition-reconstruction-decision." Existing technologies continue to allocate computing power for fine-tuning in already stable areas, resulting in slow convergence of optimizations for sudden dynamic targets (such as pedestrians crossing the road), and high single-frame reconstruction latency (often exceeding 500ms), which cannot support real-time avoidance decisions for vehicles traveling at high speeds. Therefore, existing technologies lead to low timeliness and accuracy in 3D scene reconstruction for autonomous driving scenarios. Summary of the Invention

[0006] This invention provides a method for 3D scene reconstruction for autonomous driving vehicle scenarios, which can improve the timeliness and accuracy of 3D scene reconstruction for autonomous driving vehicle scenarios.

[0007] To achieve the above objectives, the present invention provides a three-dimensional scene reconstruction method for autonomous driving vehicle scenarios, comprising: Acquire multi-source sensor data collected by autonomous vehicles, including multi-view image data and LiDAR point cloud data; The spatial initialization of the three-dimensional Gaussian tuple is performed using lidar point cloud data. The points in the point cloud are used as the initial centers of the three-dimensional Gaussian tuples, and the initial geometric parameters and initial appearance parameters of the centers of the three-dimensional Gaussian tuples are set. The key attributes of the three-dimensional Gaussian primitives are associated with random variables that follow a specific probability distribution, and the inherent uncertainty signal of each primitive is calculated based on the statistical characteristics of the probability distribution. Based on the inherent uncertainty signal, multi-view image data is used as a supervision signal to perform parameter update and adaptive lifecycle management on the initial geometric parameters and initial appearance parameters of the three-dimensional Gaussian element set until the parameter update meets the preset convergence condition to obtain the updated three-dimensional Gaussian element set. Using the iteratively optimized updated 3D Gaussian primitive set, the updated 3D Gaussian primitive set is projected onto the target view plane through a differentiable rasterization rendering pipeline, generating and outputting a reconstructed 3D scene image from the target viewpoint.

[0008] To address the aforementioned problems, the present invention also provides a three-dimensional scene reconstruction device for autonomous driving vehicle scenarios, the device comprising: The data acquisition module is used to acquire multi-source sensor data collected by autonomous vehicles, including multi-view image data and lidar point cloud data. The parameter update module is used to spatially initialize a 3D Gaussian primitive set using LiDAR point cloud data. Points in the point cloud are used as the initial centers of the 3D Gaussian primitives, and initial geometric and appearance parameters are set for these centers. Preset key attributes of the 3D Gaussian primitives are associated with random variables following a specific probability distribution, and the inherent uncertainty signal of each primitive is calculated based on the statistical characteristics of the probability distribution. Based on the inherent uncertainty signal, multi-view image data is used as a monitoring signal to perform parameter updates and adaptive lifecycle management on the initial geometric and appearance parameters of the 3D Gaussian primitive set until the parameter updates meet preset convergence conditions, resulting in an updated 3D Gaussian primitive set. The 3D scene reconstruction module utilizes an iteratively optimized updated 3D Gaussian primitive set, which is then projected onto the target view plane via a differentiable rasterization rendering pipeline to generate and output a reconstructed 3D scene image from the target viewpoint.

[0009] To address the above problems, the present invention also provides an electronic device, the electronic device comprising: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the three-dimensional scene reconstruction method for autonomous driving vehicle scenarios described above.

[0010] To address the aforementioned problems, the present invention also provides a computer-readable storage medium storing at least one computer program, which is executed by a processor in an electronic device to implement the aforementioned method for three-dimensional scene reconstruction for autonomous driving vehicle scenarios.

[0011] This invention utilizes LiDAR point cloud data to spatially initialize a 3D Gaussian primitive set, using points in the point cloud as initial 3D Gaussian primitive centers and setting initial geometric and appearance parameters for these centers. This shortens the time required for 3D reconstruction and improves the efficiency of 3D scene reconstruction. Furthermore, by associating preset key attributes of the 3D Gaussian primitives with random variables following a specific probability distribution, and calculating the inherent uncertainty signal of each primitive based on the statistical characteristics of the probability distribution, obstacles in the physical world can be transformed into mathematically measurable indicators, providing a basis for subsequent differentiated optimization. Moreover, based on inherent uncertainty... The system utilizes multi-view image data as a supervisory signal to perform parameter updates and adaptive lifecycle management on the initial geometric and appearance parameters of the 3D Gaussian primitive set until the parameter updates meet the preset convergence conditions, thus obtaining an updated 3D Gaussian primitive set. This provides more accurate parameters and reduces the computational power consumption on the autonomous vehicle side. Finally, using the iteratively optimized updated 3D Gaussian primitive set, the system projects the updated 3D Gaussian primitive set onto the target view plane through a differentiable rasterization rendering pipeline, generating and outputting a reconstructed 3D scene image from the target viewpoint. This improves the timeliness and accuracy of 3D scene reconstruction in autonomous vehicle scenarios. Attached Figure Description

[0012] Figure 1 This is a flowchart illustrating a three-dimensional scene reconstruction method for autonomous driving vehicle scenarios according to an embodiment of the present invention. Figure 2 A functional block diagram of a three-dimensional scene reconstruction device for autonomous driving vehicle scenarios provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of an electronic device that implements the three-dimensional scene reconstruction method for autonomous driving vehicle scenarios, according to an embodiment of the present invention.

[0013] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0014] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0015] This application provides a method for 3D scene reconstruction in autonomous driving vehicle scenarios. The execution entity of this method includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this application: a server, a terminal, etc. In other words, the method for 3D scene reconstruction in autonomous driving vehicle scenarios can be executed by software or hardware installed on a terminal device or a server device. The software can be a blockchain platform. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster. The server can be an independent server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0016] Reference Figure 1 The diagram shown is a flowchart illustrating a three-dimensional scene reconstruction method for autonomous driving vehicle scenarios according to an embodiment of the present invention. In this embodiment, the three-dimensional scene reconstruction method for autonomous driving vehicle scenarios includes: S1. Acquire multi-source sensor data collected by autonomous vehicles, including multi-view image data and LiDAR point cloud data.

[0017] Understandably, autonomous vehicles refer to motor vehicles that autonomously complete driving tasks; they are the hardware carriers for data collection and the terminal platforms for algorithm execution.

[0018] Understandably, multi-source sensor data refers to environmental perception data acquired from two or more sensors with different physical mechanisms.

[0019] Understandably, multi-view image data refers to a sequence of two-dimensional RGB images continuously captured by cameras installed at different locations on a vehicle (front view, side view, rear view, etc.) during the vehicle's movement, covering the same scene from different angles.

[0020] Understandably, lidar point cloud data refers to a set of three-dimensional spatial coordinate points generated by a lidar (LiDAR) emitting a laser beam and receiving the echo, and includes reflection intensity information.

[0021] S2. Spatial initialization of the 3D Gaussian primitive set is performed using LiDAR point cloud data. Points in the point cloud are used as the initial 3D Gaussian primitive centers, and the initial geometric parameters and initial appearance parameters of the 3D Gaussian primitive centers are set.

[0022] Understandably, a 3D Gaussian element refers to an ellipsoid with a 3D volume, which is the basic unit that constitutes a 3D scene.

[0023] Understandably, spatial initialization refers to the process of setting a reasonable starting point for the 3D scene reconstruction model. For example, if a lidar detects a tree 10 meters ahead, the 3D scene reconstruction model directly generates a batch of Gaussian elements at the coordinates corresponding to the tree, rather than guessing from scratch.

[0024] Understandably, the center of a three-dimensional Gaussian element refers to the coordinates of the geometric center of the three-dimensional Gaussian element.

[0025] Understandably, the initial geometric parameters refer to the initial shape-related parameters of the Gaussian ellipsoid, such as: scale: the size of the Gaussian ellipsoid; covariance matrix: describing the shape and orientation of the ellipsoid; rotation: the orientation of the ellipsoid.

[0026] Understandably, the initial appearance parameters refer to the color and lighting-related properties of Gaussian elements, for example: RGB color; opacity; spherical harmonic coefficients (SH coefficients): used to represent direction-dependent light reflection.

[0027] S3. Associate the preset key attributes of the three-dimensional Gaussian primitives with random variables that follow a specific probability distribution, and calculate the inherent uncertainty signal of each primitive based on the statistical characteristics of the probability distribution.

[0028] It is understood that the preset key attribute refers to a certain feature parameter carried by the three-dimensional Gaussian primitive itself, which in this embodiment of the invention refers to the opacity of the three-dimensional Gaussian primitive.

[0029] For example, in autonomous driving scenarios, primitive opacity The "presence" of obstacles (e.g., high opacity for pedestrians and vehicles, low opacity for shadows and road surface reflections) directly affects the accuracy of obstacle recognition. This invention models opacity as a beta distribution and extracts variance as an uncertainty signal. Among them, high values Corresponding to "blurred obstacles in rainy weather" and "ambiguous targets at a distance" (such as pedestrians and road signs that are difficult to distinguish at a distance), low values This corresponds to "clearly defined core obstacles at close range" (such as vehicles ahead or lane lines). This signal provides a "priority basis specific to the autonomous driving scenario" for subsequent gradient control and primitive management, ensuring that modeling resources for core obstacles are allocated preferentially.

[0030] It is understood that a random variable that follows a specific probability distribution refers to a variable described using statistical methods. In this embodiment of the invention, the beta distribution is used.

[0031] Understandably, the intrinsic uncertainty signal is a key concept in probabilistic modeling, representing the "degree of uncertainty" of the model regarding a certain primitive attribute.

[0032] Specifically, the key attributes of a three-dimensional Gaussian unit are associated with random variables that follow a specific probability distribution, including: The opacity of the 3D Gaussian primitive is selected as the preset key attribute; Opacity is modeled as a random variable following a beta distribution, which is defined by two shape parameters that are updated as learnable parameters during iterative optimization.

[0033] Furthermore, the inherent uncertainty signal of each primitive is calculated based on the statistical properties of the probability distribution, including: The inherent uncertainty signal of each primitive is calculated using the following formula. : ; in, For the first random variable that satisfies a beta distribution, characterize the support for the... The amount of evidence per elementary element For the second random variable that satisfies a beta distribution, characterize the opposition to the first random variable. The amount of evidence per primitive.

[0034] Furthermore, after calculating the inherent uncertainty signal of each primitive, the process further includes smoothing and updating the inherent uncertainty signal, wherein the smoothing and updating of the inherent uncertainty signal includes: Determine the K nearest neighbor primitives of each 3D Gaussian primitive in 3D space, where K is a positive integer; Calculate the weighted average of the intrinsic uncertainty signals of the K nearest neighbor 3D Gaussian elements; The inherent uncertainty signal of the current three-dimensional Gaussian element is smoothly updated using the weighted average value.

[0035] For example, smoothing updates to signals with inherent uncertainty can be implemented using the following steps: For each primitive Find its spatial K-nearest neighbors (K-NN). Use its neighborhood A weighted average (e.g., Gaussian weighted) is used to update (smooth) the uncertainty: ;in, It is a three-dimensional Gaussian unit The final uncertainty after smoothing reflects the model's confidence in whether the physical location is a real obstacle; It is a three-dimensional Gaussian unit In three-dimensional space, the "K nearest neighbor set" corresponds to physically adjacent road regions; It is a spatial correlation weight, reflecting the three-dimensional Gaussian element. With three-dimensional Gaussian elements Spatial correlation strength; It is a nearest-neighbor 3D Gaussian element The initial uncertainty signal is calculated solely based on the raw data from the sensors (camera / LiDAR) and is related to the primitives. The confidence level corresponding to the physical location may include sensor noise; This is a weight normalization factor that maps the weighted summation result to a range consistent with the uncertain signal, ensuring that the final uncertainty value is reasonable and the confidence levels of 3D Gaussian elements at different physical locations are comparable. Example values ​​are as follows: K-NN neighborhood size K=16, Gaussian weighting coefficients... =0.5 meters.

[0036] S4. Based on the inherent uncertainty signal, multi-view image data is used as a supervision signal to perform parameter update and adaptive lifecycle management on the initial geometric parameters and initial appearance parameters of the three-dimensional Gaussian element set until the parameter update meets the preset convergence condition to obtain the updated three-dimensional Gaussian element set.

[0037] Understandably, supervision signals refer to the standard answers used to guide the model in learning, directing the parameters to adjust in the right direction.

[0038] Understandably, parameter updates refer to the process of modifying the internal values ​​of a model using optimization algorithms such as gradient descent.

[0039] Understandably, adaptive lifecycle management refers to the full-process monitoring of splitting and pruning in traditional 3D Gaussian sputtering. Splitting (densification) means that when a certain area is found to be "both blurry (high uncertainty) and seemingly solid (high opacity expectation)," it is judged as underfitting, and a large primitive is split into two small primitives to fill in the details. Pruning (removal) means that when a certain area is found to be "very blurry (high uncertainty) and almost transparent (low opacity expectation)," it is judged as noise or artifact (such as floating noise in the air), and it is directly removed to save computing power.

[0040] Understandably, the preset convergence condition refers to the criterion for determining when the algorithm stops iterating and outputs the final result. For example, when the difference (Loss) between the rendered image and the real image is very small and no longer decreases significantly after multiple iterations, it is considered that the optimal solution has been reached.

[0041] Specifically, parameter updates are performed on the initial geometric and appearance parameters of the three-dimensional Gaussian element set based on the inherent uncertainty signal, including: The steps of updating the initial geometric parameters using gradient modulation of the intrinsic uncertainty signal are as follows: constructing a suppression factor and an acceleration factor using the intrinsic uncertainty signal, wherein both the suppression factor and the acceleration factor are negatively correlated with the intrinsic uncertainty signal; during backpropagation, the gradient of the geometric parameters is weighted and suppressed using the suppression factor, and the initial geometric parameters are updated using the gradient after weighted suppression. The gradient of the shape parameter of a specific probability distribution is weighted and amplified using an acceleration factor, and the initial appearance parameter is updated using the weighted and amplified gradient.

[0042] For example, updating the initial geometric and appearance parameters of a three-dimensional Gaussian element set based on an inherent uncertainty signal can be achieved using the following steps: Step 1, Geometric Gradient Suppression (for unstable primitives): Target: Geometric parameters (such as position) ,scale Rotation gradient of ) High uncertainty region ( ) generated It is unreliable, and its impact should be minimized to prevent the formation of "suspended matter." To address this, an inhibitory factor should be designed. This factor is related to uncertainty. Positive correlation (i.e., the greater the uncertainty, the stronger the inhibition). ;in Yes The normalization function (e.g., Min-Max normalization). These are hyperparameters, generally speaking. ∈[0.6,0.8], 0.8 is used for complex conditions such as rain / backlight (to enhance suppression), and 0.6 is used for sunny days (to balance suppression and convergence). The adjusted gradient .

[0043] Step 2, Accelerating Distributed Gradients (for Stable Primitives): Objective: Parameters related to the probability distribution (e.g., ...) The gradient of the parameter that controls opacity. For models that are "confident" (low numerical values) We want the primitives of a given scenario to converge as quickly as possible to a stable, sharp shape ("locking" the correct part of the scenario). Define the "confidence level": Design an acceleration factor This factor is related to the confidence level. Positive correlation. ; =0.2, to avoid excessive acceleration of the gradient of the distributed parameters, which could lead to convergence oscillations. The adjusted gradient. .

[0044] Furthermore, the parameter update of the initial geometric parameters and initial appearance parameters of the three-dimensional Gaussian element set based on the inherent uncertainty signal also includes: Calculate the uncertainty of lidar based on the local point cloud density or reflection intensity variance of lidar point cloud data; Based on the local differences between the rendered image and the real image from the current perspective, the view space uncertainty is calculated; By combining the inherent uncertainty signal, the LiDAR uncertainty, and the view space uncertainty, a joint confidence weight is generated. The final parameter gradient is obtained by weighting the parameter gradient of the three-dimensional Gaussian unit using joint confidence weights.

[0045] For example, the joint confidence weights can be generated by combining the inherent uncertainty signal, the lidar uncertainty, and the view space uncertainty, which can be achieved by the following implementation steps: A gradient (basic element) In view The final regulating factor of the gradient generated in the process It should be and Functions: For example, a simple implementation is: This means that even if a primitive itself has a high confidence level ( Low, therefore (Close to 1), if from a "bad view" (high) If a gradient is received in the input, this gradient will also be suppressed separately.

[0046] Furthermore, based on the inherent uncertainty signal, parameter updates and adaptive lifecycle management are performed on the initial geometric and appearance parameters of the 3D Gaussian element set, wherein the adaptive lifecycle management includes: The expected value of the probability distribution is calculated as the expected opacity of the three-dimensional Gaussian element; When the intrinsic uncertainty signal of the 3D Gaussian element is greater than or equal to the preset uncertainty threshold, and the expected opacity is greater than or equal to the preset opacity threshold, the 3D Gaussian element is determined to be in an underfitting state, and a splitting operation is performed on the 3D Gaussian element. When the inherent uncertainty signal of a 3D Gaussian element is greater than or equal to a preset uncertainty threshold, and the expected opacity is less than a preset opacity threshold, the 3D Gaussian element is determined to be transient noise, and a pruning operation is performed on the 3D Gaussian element.

[0047] S5. Using the iteratively optimized updated 3D Gaussian primitive set, the updated 3D Gaussian primitive set is projected onto the target view plane through a differentiable rasterization rendering pipeline to generate and output the reconstructed 3D scene image under the target view.

[0048] Understandably, a differentiable rasterization rendering pipeline refers to a rendering method that can differentiate the rendering process, allowing the rendering process to both generate images and differentiate the rendering results with respect to the parameters of the 3D scene, thereby supporting gradient-based learning and optimization.

[0049] Understandably, projecting onto the target view plane means mapping three-dimensional Gaussian primitives from three-dimensional space onto a two-dimensional image plane corresponding to a certain camera viewpoint.

[0050] This invention utilizes LiDAR point cloud data to spatially initialize a 3D Gaussian primitive set, using points in the point cloud as initial 3D Gaussian primitive centers and setting initial geometric and appearance parameters for these centers. This shortens the time required for 3D reconstruction and improves the efficiency of 3D scene reconstruction. Furthermore, by associating preset key attributes of the 3D Gaussian primitives with random variables following a specific probability distribution, and calculating the inherent uncertainty signal of each primitive based on the statistical characteristics of the probability distribution, obstacles in the physical world can be transformed into mathematically measurable indicators, providing a basis for subsequent differentiated optimization. Moreover, based on inherent uncertainty... The system utilizes multi-view image data as a supervisory signal to perform parameter updates and adaptive lifecycle management on the initial geometric and appearance parameters of the 3D Gaussian primitive set until the parameter updates meet the preset convergence conditions, thus obtaining an updated 3D Gaussian primitive set. This provides more accurate parameters and reduces the computational power consumption on the autonomous vehicle side. Finally, using the iteratively optimized updated 3D Gaussian primitive set, the system projects the updated 3D Gaussian primitive set onto the target view plane through a differentiable rasterization rendering pipeline, generating and outputting a reconstructed 3D scene image from the target viewpoint. This improves the timeliness and accuracy of 3D scene reconstruction in autonomous vehicle scenarios.

[0051] like Figure 2 The diagram shown is a functional block diagram of a three-dimensional scene reconstruction device for autonomous driving vehicle scenarios provided in an embodiment of the present invention.

[0052] The 3D scene reconstruction device 100 for autonomous driving vehicle scenarios described in this invention can be installed in an electronic device. Depending on the functions implemented, the 3D scene reconstruction device 100 for autonomous driving vehicle scenarios may include a data acquisition module 101, a parameter update module 102, and a 3D scene reconstruction module 103.

[0053] The module described in this invention can also be called a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and are stored in the memory of the electronic device.

[0054] In this embodiment, the functions of each module / unit are as follows: The data acquisition module 101 is used to acquire multi-source sensor data collected by autonomous vehicles, including multi-view image data and lidar point cloud data.

[0055] The parameter update module 102 is used to spatially initialize the three-dimensional Gaussian primitive set using lidar point cloud data. Points in the point cloud are used as the initial centers of the three-dimensional Gaussian primitives, and initial geometric and appearance parameters of these centers are set. Preset key attributes of the three-dimensional Gaussian primitives are associated with random variables following a specific probability distribution, and the inherent uncertainty signal of each primitive is calculated based on the statistical characteristics of the probability distribution. Based on the inherent uncertainty signal, multi-view image data is used as a monitoring signal to perform parameter updates and adaptive lifecycle management on the initial geometric and appearance parameters of the three-dimensional Gaussian primitive set until the parameter updates meet preset convergence conditions, resulting in an updated three-dimensional Gaussian primitive set.

[0056] The 3D scene reconstruction module 103 is used to utilize the iteratively optimized updated 3D Gaussian primitive set, and project the updated 3D Gaussian primitive set onto the target view plane through a differentiable rasterization rendering pipeline, thereby generating and outputting a reconstructed 3D scene image from the target viewpoint.

[0057] like Figure 3 The diagram shown is a structural schematic of an electronic device that implements a three-dimensional scene reconstruction method for autonomous driving vehicle scenarios, according to an embodiment of the present invention.

[0058] The electronic device may include a processor 10, a memory 11, a communication bus 12, and a communication interface 13. It may also include a computer program stored in the memory 11 and capable of running on the processor 10, such as a three-dimensional scene reconstruction method program for autonomous driving vehicle scenarios.

[0059] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the memory 11 (e.g., executing a 3D scene reconstruction method program for autonomous driving vehicle scenarios) and calls data stored in the memory 11 to perform various functions of the electronic device and process data.

[0060] The memory 11 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of an electronic device, such as a portable hard drive. In other embodiments, the memory 11 can be an external storage device of the electronic device, such as a plug-in portable hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc. Furthermore, the memory 11 can include both internal and external storage units of the electronic device. The memory 11 can be used not only to store application software and various types of data installed on the electronic device, such as the code of a 3D scene reconstruction method program for autonomous driving vehicle scenarios, but also to temporarily store data that has been output or will be output.

[0061] The communication bus 12 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory 11 and at least one processor 10, etc.

[0062] The communication interface 13 is used for communication between the aforementioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, Bluetooth interface, etc.), typically used to establish communication connections between the electronic device and other electronic devices. The user interface may be a display, an input unit (such as a keyboard), or optionally, a standard wired or wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device and to display a visual user interface.

[0063] Figure 3 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 3The structure shown does not constitute a limitation on the electronic device and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0064] For example, although not shown, the electronic device may also include a power supply (such as a battery) to power the various components. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0065] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0066] The memory 11 in the electronic device stores a 3D scene reconstruction method program for autonomous driving vehicle scenarios. This program is a combination of multiple instructions, which, when run in the processor 10, can achieve the following: Acquire multi-source sensor data collected by autonomous vehicles, including multi-view image data and LiDAR point cloud data; The spatial initialization of the three-dimensional Gaussian tuple is performed using lidar point cloud data. The points in the point cloud are used as the initial centers of the three-dimensional Gaussian tuples, and the initial geometric parameters and initial appearance parameters of the centers of the three-dimensional Gaussian tuples are set. The key attributes of the three-dimensional Gaussian primitives are associated with random variables that follow a specific probability distribution, and the inherent uncertainty signal of each primitive is calculated based on the statistical characteristics of the probability distribution. Based on the inherent uncertainty signal, multi-view image data is used as a supervision signal to perform parameter update and adaptive lifecycle management on the initial geometric parameters and initial appearance parameters of the three-dimensional Gaussian element set until the parameter update meets the preset convergence condition to obtain the updated three-dimensional Gaussian element set. Using the iteratively optimized updated 3D Gaussian primitive set, the updated 3D Gaussian primitive set is projected onto the target view plane through a differentiable rasterization rendering pipeline, generating and outputting a reconstructed 3D scene image from the target viewpoint.

[0067] Specifically, the specific implementation method of the processor 10 for the above instructions can be referred to the description of the relevant steps in the corresponding embodiment of the accompanying drawings, and will not be repeated here.

[0068] Furthermore, if the modules / units integrated in the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0069] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor of an electronic device, can perform the following: Acquire multi-source sensor data collected by autonomous vehicles, including multi-view image data and LiDAR point cloud data; The spatial initialization of the three-dimensional Gaussian tuple is performed using lidar point cloud data. The points in the point cloud are used as the initial centers of the three-dimensional Gaussian tuples, and the initial geometric parameters and initial appearance parameters of the centers of the three-dimensional Gaussian tuples are set. The key attributes of the three-dimensional Gaussian primitives are associated with random variables that follow a specific probability distribution, and the inherent uncertainty signal of each primitive is calculated based on the statistical characteristics of the probability distribution. Based on the inherent uncertainty signal, multi-view image data is used as a supervision signal to perform parameter update and adaptive lifecycle management on the initial geometric parameters and initial appearance parameters of the three-dimensional Gaussian element set until the parameter update meets the preset convergence condition to obtain the updated three-dimensional Gaussian element set. Using the iteratively optimized updated 3D Gaussian primitive set, the updated 3D Gaussian primitive set is projected onto the target view plane through a differentiable rasterization rendering pipeline, generating and outputting a reconstructed 3D scene image from the target viewpoint.

[0070] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0071] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0072] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0073] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0074] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0075] The blockchain referred to in this invention is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0076] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0077] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a system claim may also be implemented by a single unit or device through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for reconstructing a 3D scene for autonomous driving vehicle scenarios, characterized in that, The method includes: Acquire multi-source sensor data collected by autonomous vehicles, including multi-view image data and LiDAR point cloud data; The spatial initialization of the three-dimensional Gaussian tuple is performed using lidar point cloud data. The points in the point cloud are used as the initial centers of the three-dimensional Gaussian tuples, and the initial geometric parameters and initial appearance parameters of the centers of the three-dimensional Gaussian tuples are set. The key attributes of the three-dimensional Gaussian primitives are associated with random variables that follow a specific probability distribution, and the inherent uncertainty signal of each primitive is calculated based on the statistical characteristics of the probability distribution. Based on the inherent uncertainty signal, multi-view image data is used as a supervision signal to perform parameter update and adaptive lifecycle management on the initial geometric parameters and initial appearance parameters of the three-dimensional Gaussian element set until the parameter update meets the preset convergence condition to obtain the updated three-dimensional Gaussian element set. Using the iteratively optimized updated 3D Gaussian primitive set, the updated 3D Gaussian primitive set is projected onto the target view plane through a differentiable rasterization rendering pipeline, generating and outputting a reconstructed 3D scene image from the target viewpoint.

2. The 3D scene reconstruction method for autonomous driving vehicle scenarios as described in claim 1, characterized in that, The association of preset key attributes of three-dimensional Gaussian elements with random variables following a specific probability distribution includes: The opacity of the 3D Gaussian primitive is selected as the preset key attribute; Opacity is modeled as a random variable following a beta distribution, which is defined by two shape parameters that are updated as learnable parameters during iterative optimization.

3. The 3D scene reconstruction method for autonomous driving vehicle scenarios as described in claim 2, characterized in that, The calculation of the inherent uncertainty signal of each primitive based on the statistical properties of the probability distribution includes: The inherent uncertainty signal of each primitive is calculated using the following formula. : ； in, For the first random variable that satisfies a beta distribution, characterize the support for the... The amount of evidence per elementary element For the second random variable that satisfies a beta distribution, characterize the opposition to the first random variable. The amount of evidence per primitive.

4. The 3D scene reconstruction method for autonomous driving vehicle scenarios as described in claim 1, characterized in that, The parameter update of the initial geometric and appearance parameters of the three-dimensional Gaussian element set based on the inherent uncertainty signal includes: The steps of updating the initial geometric parameters using gradient modulation of the intrinsic uncertainty signal are as follows: constructing a suppression factor and an acceleration factor using the intrinsic uncertainty signal, wherein both the suppression factor and the acceleration factor are negatively correlated with the intrinsic uncertainty signal; during backpropagation, the gradient of the geometric parameters is weighted and suppressed using the suppression factor, and the initial geometric parameters are updated using the gradient after weighted suppression. The gradient of the shape parameter of a specific probability distribution is weighted and amplified using an acceleration factor, and the initial appearance parameter is updated using the weighted and amplified gradient.

5. The 3D scene reconstruction method for autonomous driving vehicle scenarios as described in claim 4, characterized in that, The parameter update of the initial geometric parameters and initial appearance parameters of the three-dimensional Gaussian element set based on the inherent uncertainty signal also includes: Calculate the uncertainty of lidar based on the local point cloud density or reflection intensity variance of lidar point cloud data; Based on the local differences between the rendered image and the real image from the current perspective, the view space uncertainty is calculated; By combining the inherent uncertainty signal, the LiDAR uncertainty, and the view space uncertainty, a joint confidence weight is generated. The final parameter gradient is obtained by weighting the parameter gradient of the three-dimensional Gaussian unit using joint confidence weights.

6. The 3D scene reconstruction method for autonomous driving vehicle scenarios as described in claim 4, characterized in that, The initial geometric and appearance parameters of the three-dimensional Gaussian element set are updated and adaptively managed based on the inherent uncertainty signal. The adaptive lifecycle management includes: The expected value of the probability distribution is calculated as the expected opacity of the three-dimensional Gaussian element; When the intrinsic uncertainty signal of the 3D Gaussian element is greater than or equal to the preset uncertainty threshold, and the expected opacity is greater than or equal to the preset opacity threshold, the 3D Gaussian element is determined to be in an underfitting state, and a splitting operation is performed on the 3D Gaussian element. When the inherent uncertainty signal of a 3D Gaussian element is greater than or equal to a preset uncertainty threshold, and the expected opacity is less than a preset opacity threshold, the 3D Gaussian element is determined to be transient noise, and a pruning operation is performed on the 3D Gaussian element.

7. The 3D scene reconstruction method for autonomous driving vehicle scenarios as described in claim 1, characterized in that, After calculating the inherent uncertainty signal of each primitive, the method further includes smoothing and updating the inherent uncertainty signal, wherein the smoothing and updating of the inherent uncertainty signal includes: Determine the K nearest neighbor primitives of each 3D Gaussian primitive in 3D space, where K is a positive integer; Calculate the weighted average of the intrinsic uncertainty signals of the K nearest neighbor 3D Gaussian elements; The inherent uncertainty signal of the current three-dimensional Gaussian element is smoothly updated using the weighted average value.

8. A three-dimensional scene reconstruction device for autonomous driving vehicle scenarios, characterized in that, The apparatus is used to implement the three-dimensional scene reconstruction method for autonomous driving vehicle scenarios as described in any one of claims 1 to 7, the apparatus comprising: The data acquisition module is used to acquire multi-source sensor data collected by autonomous vehicles, including multi-view image data and lidar point cloud data. The parameter update module is used to spatially initialize a 3D Gaussian primitive set using LiDAR point cloud data. Points in the point cloud are used as the initial centers of the 3D Gaussian primitives, and initial geometric and appearance parameters are set for these centers. Preset key attributes of the 3D Gaussian primitives are associated with random variables following a specific probability distribution, and the inherent uncertainty signal of each primitive is calculated based on the statistical characteristics of the probability distribution. Based on the inherent uncertainty signal, multi-view image data is used as a monitoring signal to perform parameter updates and adaptive lifecycle management on the initial geometric and appearance parameters of the 3D Gaussian primitive set until the parameter updates meet preset convergence conditions, resulting in an updated 3D Gaussian primitive set. The 3D scene reconstruction module utilizes an iteratively optimized updated 3D Gaussian primitive set, which is then projected onto the target view plane via a differentiable rasterization rendering pipeline to generate and output a reconstructed 3D scene image from the target viewpoint.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the three-dimensional scene reconstruction method for autonomous driving vehicle scenarios as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the three-dimensional scene reconstruction method for autonomous driving vehicle scenarios as described in any one of claims 1 to 7.

Images