A Novel Radar View Synthesis Method Based on Scene Graph

Abstract

This invention discloses a novel radar view synthesis method based on scene graphs. The invention applies 3D Gaussian technology to radar novel view synthesis, using multi-frame LiDAR point clouds as input to construct a hierarchical scene graph to decouple the static background Gaussian and dynamic object Gaussian. Each component is represented by different Gaussian primitives and is jointly optimized with four lightweight neural MLPs (neural Gaussian fields). This successfully predicts the covariance, intensity, ray loss characteristics, and opacity of each Gaussian. Finally, it is further optimized using a trained U-Net to improve the point cloud quality. During rendering, given the LiDAR pose and corresponding timestamp, the system queries the scene graph and obtains active object nodes, which are then aggregated with the static background Gaussian. The aggregated representation is then rendered along the ray execution volume using LBS projection and neural attribute prediction to generate simulated distance, intensity, and ray loss probability images.

Description

Technical Field

[0001] This invention relates to 3D reconstruction, which is particularly crucial for fields such as autonomous driving and robotics. This invention applies 3D Gaussian technology to novel radar view synthesis. Using multi-frame LiDAR point clouds as input, a hierarchical scene graph is constructed to decouple the static background Gaussian and dynamic object Gaussian. Each component is represented by different Gaussian primitives and is jointly optimized with four lightweight neural MLPs (neural Gaussian fields). This successfully predicts the covariance, intensity, ray loss characteristics, and opacity of each Gaussian. Finally, it is further optimized using a trained U-Net to improve the point cloud quality. During rendering, given the LiDAR pose and corresponding timestamp, the system queries the scene graph and obtains active object nodes, which are then aggregated with the static background Gaussian. The aggregated representation is then projected using LBS and predicted by neural attributes, and rendered along the ray actuator to generate simulated distance, intensity, and ray loss probability images. Background Technology

[0002] Novel View Synthesis (NVS) aims to generate scene content from arbitrary new perspectives from observation data with limited viewpoints, and is a core research problem in the intersection of computer vision and graphics. This technology has significant application value in areas such as autonomous driving virtual simulation, 3D digital twin construction, and robot autonomous navigation. Traditional NVS research mainly focuses on optical images, with methods such as Neural Radiation Field (NeRF) achieving photorealistic rendering through implicit scene representation. However, image-based methods are susceptible to changes in lighting and inclement weather, and lack precise geometric measurement information, making it difficult to meet the stringent requirements of high precision and robustness for safety-critical applications such as autonomous driving.

[0003] In recent years, novel view synthesis techniques based on neural radiation fields have provided a new implicit modeling paradigm for LiDAR data simulation. This technology, by learning the continuous volume density and attribute distribution of a scene, can theoretically achieve higher fidelity data generation than traditional geometric simulation. However, applying NVS technology to large-scale urban LiDAR scenes still faces several scientific challenges: First, LiDAR data has extremely high spatial sparsity, and traditional methods are prone to geometric distortion and overfitting when processing sparse inputs; second, urban scenes contain a large number of dynamic targets, and how to achieve accurate separation and distortion-free reconstruction of dynamic objects from static backgrounds at high rendering frequencies remains a current research hotspot and challenge in academia.

[0004] Therefore, researching novel LiDAR view synthesis methods that possess physical consistency, geometric accuracy, and support efficient dynamic updates has profound academic and engineering value for building digital twin platforms for autonomous driving. This research aims to explore the adaptation of 3D Gaussian sputtering (3DGS) representation in the field of LiDAR data synthesis by introducing a global optimization mechanism and hierarchical scene representation. This not only significantly reduces the testing and verification costs of autonomous driving systems but also provides core technical support and data assurance for expanding the safety boundaries of autonomous driving perception by generating physically realistic and spatiotemporally consistent synthetic point cloud data. Summary of the Invention

[0005] The specific content of the invention is as follows:

[0006] 1. A novel radar view synthesis method based on scene graphs, comprising the following steps:

[0007] Step 1) The problem studied in this paper involves input data including a set of LiDAR scans collected by moving sensors in a driving scenario, as well as the corresponding calibration sensor pose P and timestamp T. Additionally, a set of bounding boxes B is provided for tracking moving vehicles in the scene. The research objective is to construct an explicit representation of this dynamic scene and, from any given new time... And New Perspective Render a highly realistic LiDAR view, while maintaining high fidelity under various scene operations and sensor configurations;

[0008] Step 2) Map the lidar point cloud into a two-dimensional distance image. For any three-dimensional point in the Cartesian coordinate system... The transition to the spherical coordinate system can be achieved using the following formula. The conversion is as follows:

[0009]

[0010] Step 3.1) Decompose the scene into a static background and moving objects, each component represented by a different Gaussian meta-node. For rigid structures such as roads, the Gaussian element is defined as follows: All static Gaussians are defined in the world coordinate system, including the center position x. Rotation quaternion, 2D scale vector echo intensity ray loss probability Opacity and 32-dimensional optimizable feature labels ;

[0011] Step 3.2) For dynamic object nodes Define a set of shared primitives in the local coordinate system.

[0012]

[0013] in This represents the Gaussian value corresponding to the i-th object. This represents the number of Gaussians contained in the i-th object;

[0014] Step 3.3) To handle foreground moving vehicles, this paper uses tracking bounding boxes to track their trajectories. Since dynamic objects are defined in a local coordinate system, to transform them into a world coordinate system, this paper defines the tracking pose of each rigid object as a series of rotation matrices. Translation vector The transformed form is as follows:

[0015]

[0016]

[0017] in It is 3D coordinates in the world coordinate system. O represents the local coordinate system and w represents the world coordinate system. With this representation, we can combine independent models together to reconstruct and render dynamic scenes, thereby improving reconstruction accuracy.

[0018] Step 4.1) Use four lightweight multilayer perceptrons to acquire the required attributes during the rendering process, employing... Learning covariance and the offset of the initial position x The input parameters are simultaneously associated with the observation direction. As shown in the following formula:

[0019]

[0020] Intensity sensor The definition is as follows:

[0021]

[0022] This article uses To learn the radar ray loss probability, we need to understand that the laser beam will not return a value. Ray loss typically occurs in close-range detection due to blind spots caused by sensor field of view limitations, and in long-range detection where it is impossible to measure. Therefore, the network input includes the viewpoint direction. eigenvectors of each Gaussian point The information about the distance d is given by the following formula:

[0023]

[0024] Opaque mesh The definition is as follows:

[0025]

[0026] Where 'a' explains the occlusion and invisibility between each Gaussian;

[0027] Step 4.2) Perform distance-view rasterization on each pixel P, and perform relevant high-order pixel rasterization based on the ray direction d. Perform microsection projection (LBS) to calculate the 2D Gaussian cross-sectional contribution in the plane perpendicular to the ray, and the cross-sectional parameters are related to the transformed ray direction. The flight distance d is input into the MLP, and each Gaussian attribute is predicted. The volume rendering accumulates the contribution along the ray to synthesize the final measurement, as shown in the following formula:

[0028]

[0029] in, Transparency weights are assigned based on depth, and laser beam loss events are finally evaluated using a ray loss determination mechanism: if the cumulative probability... If so, then all attributes of that pixel are cleared;

[0030] The ray loss probability map is obtained from the Gaussian distribution through alpha mixing:

[0031]

[0032] Given that LiDAR ray loss is also affected by the characteristics of LiDAR itself, a learnable prior probability of ray loss is introduced. The final ray loss probability diagram is shown in the following equation:

[0033]

[0034] The probabilistic graph is supervised by the U-Net binary cross-entropy loss function with residuals and the ground truth mask, as shown in the formula:

[0035]

[0036] Step 5) Use the following objective loss function to optimize the feature vector of each Gaussian. And four MLPs:

[0037]

[0038]

[0039]

[0040] Among them depth loss use Loss, intensity loss weight Set to 0.2, CD loss This paper uses the chamfer distance between the rendered point cloud and the ground truth, and also employs regularization loss. To suppress the increase in scale, This is the time-series consistency loss function;

[0041] By optimizing the loss function, a three-dimensional radar point cloud is finally obtained, and a radar point cloud under a new pose is generated.

[0042] The present invention has the following advantages and beneficial effects:

[0043] To address the issues of geometric distortion and overfitting in large-scale sparse scenes, as well as insufficient independent modeling capability for dynamic targets and difficulty in accurately capturing the boundaries of moving objects, this invention proposes a novel radar view synthesis method based on scene graphs. This method decouples the static background and dynamic object Gaussian by hierarchical scene graphs and combines them with lightweight neural MLP joint optimization to effectively alleviate geometric distortion and overfitting caused by sparse LiDAR data. At the same time, the modeling of the local coordinate system of dynamic objects and the design of attitude transformation can accurately capture motion boundaries. Combined with U-Net optimization and multiple loss function constraints, the rendering fidelity is improved, adapting to the needs of autonomous driving and other scenarios, and possessing both academic value and engineering practicality.

[0044] This invention has significant application and implementation value, which can greatly reduce the time and economic cost of testing and verification of autonomous driving systems, provide high-fidelity point cloud data support for fields such as autonomous driving virtual simulation, 3D digital twin construction, and robot autonomous navigation, promote the relevant technologies from the laboratory to industrial applications, and empower the industrial upgrading and high-quality development of the autonomous driving and robotics fields. Attached Figure Description

[0045] Figure 1 This is a flowchart of a novel radar view synthesis method based on global joint optimization. Detailed Implementation

[0046] The advantages and objectives of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the description herein is only for explaining the present invention and is not intended to limit the present invention.

[0047] In a large-scale scenario, the input consists of a set of LiDAR scans collected by mobile sensors during a driving scene, along with the corresponding calibration sensor poses P and timestamps T. Additionally, a set of bounding boxes B is provided for tracking moving vehicles in the scene.

[0048] Appendix Figure 1Flowchart of a novel radar view synthesis method based on global joint optimization.

[0049] Includes the following steps:

[0050] Step 1) The problem studied in this paper involves input data including a set of LiDAR scans collected by moving sensors in a driving scenario, as well as the corresponding calibration sensor pose P and timestamp T. Additionally, a set of bounding boxes B is provided for tracking moving vehicles in the scene. The research objective is to construct an explicit representation of this dynamic scene and, from any given new time... And New Perspective Render a highly realistic LiDAR view, while maintaining high fidelity under various scene operations and sensor configurations;

[0051] Step 2) Map the lidar point cloud into a two-dimensional distance image. For any three-dimensional point in the Cartesian coordinate system... The transition to the spherical coordinate system can be achieved using the following formula. The conversion is as follows:

[0052]

[0053] Step 3.1) Decompose the scene into a static background and moving objects, each component represented by a different Gaussian meta-node. For rigid structures such as roads, the Gaussian element is defined as follows: All static Gaussians are defined in the world coordinate system, including the center position x. Rotation quaternion, 2D scale vector echo intensity ray loss probability Opacity and 32-dimensional optimizable feature labels ;

[0054] Step 3.2) For dynamic object nodes Define a set of shared primitives in the local coordinate system.

[0055]

[0056] in This represents the Gaussian value corresponding to the i-th object. This represents the number of Gaussians contained in the i-th object;

[0057] Step 3.3) To handle foreground moving vehicles, this paper uses tracking bounding boxes to track their trajectories. Since dynamic objects are defined in a local coordinate system, to transform them into a world coordinate system, this paper defines the tracking pose of each rigid object as a series of rotation matrices. Translation vector The transformation form is as follows:

[0058]

[0059]

[0060] in It is 3D coordinates in the world coordinate system. O represents the local coordinate system and w represents the world coordinate system. With this representation, we can combine independent models together to reconstruct and render dynamic scenes, thereby improving reconstruction accuracy.

[0061] Step 4.1) Use four lightweight multilayer perceptrons to acquire the required attributes during the rendering process, employing... Learning covariance and the offset of the initial position x The input parameters are simultaneously associated with the observation direction. As shown in the following formula:

[0062]

[0063] Intensity sensor The definition is as follows:

[0064]

[0065] This article uses To learn the radar ray loss probability, we need to understand that the laser beam will not return a value. Ray loss typically occurs in close-range detection due to blind spots caused by sensor field of view limitations, and in long-range detection where it is impossible to measure. Therefore, the network input includes the viewpoint direction. eigenvectors of each Gaussian point The information about the distance d is given by the following formula:

[0066]

[0067] Opaque mesh The definition is as follows:

[0068]

[0069] Where 'a' explains the occlusion and invisibility between each Gaussian;

[0070] Step 4.2) Perform distance-view rasterization on each pixel P, and perform relevant high-order pixel rasterization based on the ray direction d. Perform microsection projection (LBS) to calculate the 2D Gaussian cross-sectional contribution in the plane perpendicular to the ray, and the cross-sectional parameters are related to the transformed ray direction. The flight distance d is input into the MLP, and each Gaussian attribute is predicted. The volume rendering accumulates the contribution along the ray to synthesize the final measurement, as shown in the following formula:

[0071]

[0072] in, Transparency weights are assigned based on depth, and laser beam loss events are finally evaluated using a ray loss determination mechanism: if the cumulative probability... If so, then all attributes of that pixel are cleared;

[0073] The ray loss probability map is obtained from the Gaussian distribution through alpha mixing:

[0074]

[0075] Given that LiDAR ray loss is also affected by the characteristics of LiDAR itself, a learnable prior probability of ray loss is introduced. The final ray loss probability diagram is shown in the following equation:

[0076]

[0077] The probabilistic graph is supervised by the U-Net binary cross-entropy loss function with residuals and the ground truth mask, as shown in the formula:

[0078]

[0079] Step 5) Use the following objective loss function to optimize the feature vector of each Gaussian. And four MLPs:

[0080]

[0081]

[0082]

[0083] Among them depth loss use Loss, intensity loss weight Set to 0.2, CD loss This paper uses the chamfer distance between the rendered point cloud and the ground truth, and also employs regularization loss. To suppress the increase in scale, This is the time-series consistency loss function;

[0084] By optimizing the loss function, a three-dimensional radar point cloud is finally obtained, and a radar point cloud under a new pose is generated.

Claims

1. A novel radar view synthesis method based on scene graphs, comprising the following steps. Step 1) The problem studied in this paper involves input data including a set of LiDAR scans collected by moving sensors in a driving scenario, as well as the corresponding calibration sensor pose P and timestamp T. Additionally, a set of bounding boxes B is provided for tracking moving vehicles in the scene. The research objective is to construct an explicit representation of this dynamic scene and, from any given new time... And New Perspective Render a highly realistic LiDAR view, while maintaining high fidelity under various scene operations and sensor configurations; Step 2) Map the lidar point cloud into a two-dimensional distance image. For any three-dimensional point in the Cartesian coordinate system... The transition to the spherical coordinate system can be achieved using the following formula. The conversion is as follows: Step 3.1) Decompose the scene into a static background and moving objects, each component represented by a different Gaussian meta-node. For rigid structures such as roads, the Gaussian element is defined as follows: All static Gaussians are defined in the world coordinate system, including the center position x. Rotation quaternion, 2D scale vector echo intensity ray loss probability Opacity and 32-dimensional optimizable feature labels ; Step 3.2) For dynamic object nodes Define a set of shared primitives in the local coordinate system. in This represents the Gaussian value corresponding to the i-th object. This represents the number of Gaussians contained in the i-th object; Step 3.3) To handle foreground moving vehicles, this paper uses tracking bounding boxes to track their trajectories. Since dynamic objects are defined in a local coordinate system, to transform them into a world coordinate system, this paper defines the tracking pose of each rigid object as a series of rotation matrices. Translation vector The transformed form is as follows: in It is 3D coordinates in the world coordinate system. O represents the local coordinate system and w represents the world coordinate system. With this representation, we can combine independent models together to reconstruct and render dynamic scenes, thereby improving reconstruction accuracy. Step 4.1) Use four lightweight multilayer perceptrons to acquire the required attributes during the rendering process, employing... Learning covariance and the offset of the initial position x The input parameters are simultaneously associated with the observation direction. As shown in the following formula: Intensity sensor The definition is as follows: This article uses To learn the radar ray loss probability, we need to understand that the laser beam will not return a value. Ray loss typically occurs in close-range detection due to blind spots caused by sensor field of view limitations, and in long-range detection where it is impossible to measure. Therefore, the network input includes the viewpoint direction. eigenvectors of each Gaussian point The information about the distance d is given by the following formula: Opaque mesh The definition is as follows: Where 'a' explains the occlusion and invisibility between each Gaussian. Step 4.2) Perform distance-view rasterization on each pixel P, and perform relevant high-order pixel rasterization based on the ray direction d. Perform microsection projection (LBS) to calculate the 2D Gaussian cross-sectional contribution in the plane perpendicular to the ray, and the cross-sectional parameters are related to the transformed ray direction. The flight distance d is input into the MLP, and each Gaussian attribute is predicted. The volume rendering accumulates the contribution along the ray to synthesize the final measurement, as shown in the following formula: in, Transparency weights are assigned based on depth, and laser beam loss events are finally evaluated using a ray loss determination mechanism: if the cumulative probability... If so, then all attributes of that pixel are cleared; The ray loss probability map is obtained from the Gaussian distribution through alpha mixing: Given that LiDAR ray loss is also affected by the characteristics of LiDAR itself, a learnable prior probability of ray loss is introduced. The final ray loss probability diagram is shown in the following equation: The probabilistic graph is supervised by the U-Net binary cross-entropy loss function with residuals and the ground truth mask, as shown in the formula: Step 5) Use the following objective loss function to optimize the feature vector of each Gaussian. And four MLPs: Among them depth loss use Loss, intensity loss weight Set to 0.2, CD loss This paper uses the chamfer distance between the rendered point cloud and the ground truth, and also employs regularization loss. To suppress the increase in scale, This is the time-series consistency loss function; By optimizing the loss function, a three-dimensional radar point cloud is finally obtained, and a radar point cloud under a new pose is generated.

Images