A multi-view consistent three-dimensional scene construction method and system

Abstract

The application discloses a multi-view consistent three-dimensional scene construction method and system, and belongs to the technical field of three-dimensional perception and unmanned driving. In order to solve the technical problems of high labeling cost and poor view consistency, the application obtains synchronous images and point clouds, extracts features by using a double-flow network, and performs spatial mapping and fusion to obtain fused sensing features. The features are processed by using a volume rendering module to obtain a composite image and depth. Based on the composite result and original observation data, a joint loss containing geometric consistency loss is constructed, the network is iteratively optimized and trained to obtain a perception model, and then a three-dimensional scene reconstruction result is generated. The application introduces multi-view geometric consistency closed-loop constraints, realizes zero-labeling cost training through a self-supervision mechanism, and significantly improves the description accuracy of the model on the edges of complex objects and the perception stability in a shielding environment.

Description

Technical Field

[0001] This invention belongs to the field of 3D perception and autonomous driving technology, specifically relating to a method and system for constructing a 3D scene with consistent multi-view perspectives. Background Technology

[0002] In the environmental perception tasks of autonomous driving systems, the detection and representation of 3D obstacles is a core component. Traditional methods often rely on 3D object detection models, simplifying objects in the environment into 3D cuboid bounding boxes. However, this representation method can only cover a limited number of known object categories, and it struggles to provide detailed geometric information when dealing with complex, irregularly shaped obstacles, thus affecting the safety of obstacle avoidance path planning.

[0003] Furthermore, existing deep learning perception models heavily rely on large-scale manually labeled data, a process that is time-consuming, labor-intensive, and costly. While implicit representation techniques based on Neural Radiation Fields (NeRF) have demonstrated powerful 3D reconstruction capabilities, their original frameworks are mostly applicable to static offline scenarios and lack explicit constraints for cross-view geometric consistency. This results in models often failing to guarantee logical consistency in depth estimation across different observation perspectives in dynamic driving environments. Summary of the Invention

[0004] The purpose of this invention is to address the problems of existing perception models being highly dependent on manual annotation, having low accuracy in modeling irregular obstacles, and having inconsistent perception results from multiple perspectives. This invention proposes a method and system for constructing 3D scenes with consistent multiple perspectives. Through a self-supervised training mechanism and geometric consistency constraints, this invention achieves high-precision and high-stability 3D scene modeling and obstacle avoidance representation for complex environments without relying on manual annotation.

[0005] To achieve the above objectives, the present invention adopts the following technical solution.

[0006] A method for constructing a consistent 3D scene from multiple perspectives includes the following steps: Acquire synchronized raw camera images and LiDAR point clouds; A dual-stream feature extraction network is used to extract features from the original camera image and the lidar point cloud, and the extracted features are fused to obtain fused sensing features; The volume rendering module performs volume rendering processing based on the fused sensing features to obtain the synthesized image and synthesized depth; Based on the synthesized image and the synthesized depth, as well as the original camera image and the lidar point cloud, a joint loss is constructed, and the joint loss is used to iteratively optimize and train the dual-stream feature extraction network and the volume rendering module to obtain a 3D scene perception model. The 3D scene perception model is used to process the camera image and LiDAR point cloud to generate a 3D scene reconstruction result.

[0007] Further, feature extraction is performed on the original camera image and the lidar point cloud using a two-stream feature extraction network, including: The original camera image is downsampled using a deep convolutional neural network to extract semantic features of the image from a perspective view. The point cloud of the lidar is voxelized and mapped to a bird's-eye view plane to extract the geometric structure features of the point cloud from the bird's-eye view.

[0008] Furthermore, the extracted features are fused to obtain fused sensing features, including: Obtain the three-dimensional spatial points to be sampled, and use the projection matrix to project the three-dimensional spatial points to be sampled onto the perspective view plane and the bird's-eye view plane respectively; Image semantic features are obtained in the perspective plane by bilinear interpolation, and point cloud geometric features are obtained in the bird's-eye view plane. The obtained semantic features of the image and the geometric features of the point cloud are vector-concatenated to obtain fused sensing features.

[0009] Further, volume rendering processing is performed based on the fused sensing features to obtain a synthesized image and synthesized depth, including: Discrete sampling is performed along the ray from the camera focus to the pixel to determine the volume density and color value of each sampling point; The transmittance and opacity weights corresponding to each sampling point are calculated based on the volume density. The composite image is obtained by weighted integral of the color values ​​of the sampling points according to the opacity weight, and the composite depth is obtained by weighted integral of the sampling distance of the sampling points according to the opacity weight.

[0010] Furthermore, discrete sampling is performed along the ray from the camera focus to the pixel to determine the volume density and color value of each sampling point, including: The ray beam to be rendered is selected within the pixel area based on a random sampling strategy, and the ray beam corresponding to the lidar point cloud coverage area is selected first. The volume density and color value are obtained by using a multilayer perceptron to analyze the sampling points on the ray beam to be rendered.

[0011] Further, a joint loss is constructed based on the synthesized image and the synthesized depth, as well as the original camera image and the LiDAR point cloud, including: Calculate the color deviation between the pixels of the synthesized image and the original camera image to obtain the photometric loss; The distance error between the synthesized depth and the projection depth of the lidar point cloud is calculated to obtain the depth supervision loss; The geometric consistency loss is obtained by calculating the depth matching error between multiple observation viewpoints based on the cyclic reprojection logic. The combined loss is obtained by weighted summation of the photometric loss, the depth supervision loss, and the geometric consistency loss.

[0012] Furthermore, based on the cyclic reprojection logic, the depth matching error between multiple observation views is calculated to obtain the geometric consistency loss, including: By combining the pixel coordinates from the first-person perspective with the estimated depth, the first three-dimensional spatial point is obtained through mapping. Project the first three-dimensional spatial point onto the imaging plane of the second viewpoint to obtain the second viewpoint projection coordinates; The estimated depth corresponding to the projection coordinates of the second viewpoint is obtained and restored to the second three-dimensional space point. The second three-dimensional space point is then projected back to the first viewpoint to obtain the reprojected pixel coordinates. The geometric consistency loss is obtained by measuring the Euclidean distance between the reprojected pixel coordinates and the pixel coordinates under the first viewpoint.

[0013] Furthermore, the dual-stream feature extraction network and the volume rendering module are iteratively optimized and trained using the joint loss, including: The network parameters of the dual-stream feature extraction network and the volume rendering module are updated using gradients based on the joint loss. After the joint loss converges, the zero isosurface is extracted to generate a three-dimensional mesh, thus completing the training of the perception model.

[0014] Furthermore, the 3D scene perception model is used to process the camera image and LiDAR point cloud to generate a 3D scene reconstruction result, including: Observation frames are selected based on the continuous observation sequence of the scene to be reconstructed, and used as rendering supervision references; The trained 3D scene perception model extracts volumetric features and synthesizes scenes from real-time sensor data to generate consistent 3D scene reconstruction results from multiple perspectives.

[0015] A multi-view consistent 3D scene construction system includes: The data acquisition module is used to acquire synchronized raw camera images and LiDAR point clouds; A dual-stream feature extraction network is used to extract features from the original camera image and the lidar point cloud, and to fuse the extracted features to obtain fused sensing features; The volume rendering module is used to perform volume rendering processing based on the fused sensing features to obtain a synthesized image and a synthesized depth. The training module is used to construct a joint loss based on the synthesized image and the synthesized depth, as well as the original camera image and the lidar point cloud, and to use the joint loss to iteratively optimize and train the dual-stream feature extraction network and the volume rendering module to obtain a three-dimensional scene perception model. The inference module is used to process the camera images and LiDAR point clouds to be processed using the three-dimensional scene perception model to generate three-dimensional scene reconstruction results.

[0016] The present invention has achieved the following beneficial effects.

[0017] 1. This invention adopts a self-supervised learning mechanism based on continuous time series, using the consistency of spatiotemporal frame sequences as a supervision signal, breaking the dependence of traditional models on large-scale manually labeled 3D data, and realizing a training process with zero labeling cost.

[0018] 2. This invention introduces a multi-view geometric consistency loss function to construct a closed-loop reprojection logic between pixels and 3D points, effectively solving the drift problem of depth estimation under different observation views, and significantly enhancing the geometric robustness and logical stability of the reconstructed 3D structure.

[0019] 3. Based on the principle of neural radiation field, this invention uses a volume rendering module to predict volume density. Compared with the traditional cuboid frame representation, it can more finely depict the geometric edges and local texture details of complex irregular objects, providing more detailed obstacle avoidance representation information for autonomous vehicles.

[0020] 4. This invention achieves deep fusion of perspective semantics and bird's-eye view geometric features through a dual-stream feature extraction network, enabling the model to still have strong representation capabilities when facing occluded scenes or long-distance perception tasks, and demonstrating strong environmental adaptability.

[0021] 5. This invention achieves efficient conversion from sensor data to 3D mesh results through collaborative optimization of joint loss. The generated multi-view consistent scene can be directly applied to path planning, obstacle detection, and high-precision map construction for autonomous driving. Attached Figure Description

[0022] Figure 1 This is a flowchart of a method for constructing a consistent 3D scene from multiple perspectives, as described in this embodiment. Figure 2 This is a schematic diagram of the overall framework of a multi-view consistent 3D scene construction method in the embodiment; Figure 3 This is a schematic diagram illustrating the calculation of multi-view geometric consistency loss in the embodiment; Figure 4 This is a module diagram of a multi-view consistent 3D scene construction system in the embodiment; Figure 5 This is a 3D reconstruction effect of the scene in the embodiment.

[0023] Figure 6 This is a composite image from a new perspective in the embodiment. Detailed Implementation

[0024] To make the various technical features, advantages, or effects of the present invention more apparent and understandable, detailed descriptions are provided below through embodiments.

[0025] This invention provides a method for constructing a consistent 3D scene from multiple perspectives, the overall process architecture of which is as follows: Figure 1 and Figure 2 As shown, the specific steps include:

[0026] Step S1: Obtain the synchronized camera image sequence and LiDAR point cloud sequence.

[0027] During the initialization phase, the world coordinate system is first defined. and vehicle coordinate system Then, it receives the synchronized camera image sequence and LiDAR point cloud sequence.

[0028] Step S2: The camera image sequence and the lidar point cloud sequence are used to extract features through a dual-stream feature extraction network. The extracted features are then fused to obtain the fused sensing features.

[0029] Step S21: Extract sensor sequence data through a dual-stream feature extraction network to obtain image semantic features and point cloud geometric structure features.

[0030] The image branch of the two-stream feature extraction network uses a deep convolutional neural network to downsample camera images into feature maps. It is used to extract semantic features from a perspective view (PV), and belongs to image feature extractors. This invention calls it a perspective view encoder (PV encoder), such as SwingTransformer.

[0031] The point cloud branch uses a voxelized neural network to map the LiDAR point cloud into bird's-eye view features after voxelization. This invention aims to extract the geometric structural features of LiDAR point clouds from a bird's-eye view (BEV) perspective. It is a point cloud feature extractor and is referred to as a bird's-eye view encoder (BEV encoder) in this invention, such as SECOND.

[0032] Step S22: Multimodal feature fusion is performed based on the spatial coordinate mapping function to obtain fused sensing features.

[0033] Through spatial coordinate mapping function Any three-dimensional space point to be sampled The data are projected onto the PV plane and the BEV plane respectively. The corresponding multimodal features are obtained through bilinear interpolation and then concatenated into vectors to obtain fused sensing features that contain both global geometric information and local texture details. Through this mapping logic, the model can simultaneously utilize the high-resolution semantic information of the image and the precise depth information of the point cloud. Feature vector fusion. The acquisition follows the following mapping logic: in, Represents the fused sensing feature vector; Concat represents the vector concatenation operation; Interp represents the bilinear interpolation operation; Represents the feature map; Indicates features in a bird's-eye view; Represents the projection matrix; This represents the three-dimensional spatial point to be sampled.

[0034] Step S3: The volume rendering module performs rendering and synthesis based on the fused sensing features to obtain the synthesized image and synthesis depth.

[0035] To address the dependence of perception models on labeled data, this embodiment designs a self-supervised learning mechanism based on continuous time series. This strategy selects observation frames that are close to the current input frame in terms of both timestamp and spatial distance from the continuous data stream collected by the autonomous vehicle, based on the vehicle's motion state and pose transformation matrix, as rendering supervision frames.

[0036] Step S31: Discrete sampling is performed along the ray to obtain the volume density and color value of each sampling point, and a composite image is obtained.

[0037] This embodiment is based on the principle of neural radiation fields and uses a three-layer residual multilayer perceptron (MLP) as the volume rendering module for density estimation and color value prediction. During training, a ray along the camera focus to the pixel is used. Perform discrete sampling, where Indicates the origin of the ray. Indicates the sampling distance. Indicates the direction of the ray. For the _ , sampling points The volume rendering module predicts its volume density. and color value To reduce computational overhead, this embodiment employs a strategy of randomly sampling ray beams and prioritizes depth rendering sampling in pixel areas covered by point clouds.

[0038] Composite image pixels Predict color values ​​by sampling points The weighted integral is obtained as follows: in, Represents rays The corresponding composite color value (desired color) of the pixel; N represents the total number of sampling points along this ray direction; Indicates the first Opacity weights corresponding to each sampling point; The volume rendering module predicts the first The color value of each sampling point.

[0039] Step S32: Perform a weighted integral based on the transmittance and opacity weights to obtain the desired depth of the pixel, i.e., the composite depth.

[0040] Calculate the transmittance between sampling points and opacity weight : in, Indicates the first Transmittance corresponding to each sampling point; Indicates the first Opacity weights corresponding to each sampling point; Indicates the first Volume density at each sampling point; This indicates the distance between adjacent sampling points.

[0041] Expected depth of a pixel By sampling distance The weighted integral is obtained as follows: in, Indicates the desired depth of a pixel; Indicates the total number of sampling points; Indicates sampling point The corresponding sampling distance. In this embodiment, the consistency between the synthesized result and the original observation data of the rendered frame is used as a supervision signal to achieve spontaneous optimization of the model.

[0042] Step S4: Construct a joint loss based on the synthesized image and synthesized depth, and the original image and original depth; train a dual-stream feature extraction network and a volume rendering module based on the joint loss.

[0043] Step S41: Calculate the color deviation between pixels in the volume-rendered composite image and the original image to obtain the photometric loss. .

[0044] in, Represents the original image; Indicates the pixel values ​​of the composite image; This indicates the number of pixels sampled.

[0045] Step S42: Calculate the distance error between the synthesized depth and the lidar projection depth to obtain the depth supervision loss. .

[0046] in, Represents the original depth value; Indicates the composite depth value; This indicates the number of pixels sampled.

[0047] Step S43: Calculate the multi-view geometric consistency closed-loop constraints and obtain the geometric consistency loss. .

[0048] To address the issue of depth estimation mismatch between viewpoints and enhance the model's perception of 3D spatial logic, this embodiment introduces multi-view geometric consistency constraints. For example... Figure 3 As shown, this constraint, by constructing a closed-loop reprojection logic of pixel-to-3D point-to-pixel, forces the depth information learned by the model under different viewing perspectives to be mapped to the same 3D object surface. The constraint process includes the following sub-steps: Step S431: Perform initial mapping on the pixels under viewpoint 1 to obtain three-dimensional spatial points.

[0049] Extract pixels from viewpoint 1 Combined with its estimated depth Transform to three-dimensional space points .

[0050] Step S432: Perform cross-view projection on the three-dimensional spatial points to obtain the projected coordinates under the imaging plane of viewpoint 2.

[0051] Points in three-dimensional space Projecting onto the imaging plane of viewpoint 2, we obtain the coordinates. .

[0052] Step S433: Perform cyclic reprojection processing to obtain the reprojection coordinates under the imaging plane of viewpoint 1.

[0053] Extract coordinates from viewpoint 2 Estimated depth Restored to three-dimensional points And project it back to viewpoint 1 to obtain the reprojected coordinates. .

[0054] Step S434: Perform error measurement based on Euclidean distance to obtain geometric consistency loss. .

[0055] If the model has good geometric consistency, then the reprojected coordinates Should be consistent with the initial point Strictly coincident. Calculate the Euclidean distance error as the geometric consistency loss. This loss function not only suppresses geometric shifts across multiple viewpoints but also significantly improves the model's convergence speed in occluded and weakly textured regions. Geometric consistency loss. The calculation formula is as follows: in, This represents the geometric consistency loss; Indicates the total number of pixels; This represents the initial pixel coordinates in viewpoint 1; This represents the pixel coordinates that return to viewpoint 1 after cyclic reprojection. This embodiment ensures the consistency of geometric logic under different observation viewpoints by introducing this closed-loop constraint.

[0056] Step S44: Perform a weighted summation based on photometric loss, depth supervision loss, and geometric consistency loss to obtain the joint loss, i.e., the total loss.

[0057] This embodiment utilizes color loss, i.e., photometric loss. Deep loss, also known as deep supervision loss. and the core multi-view geometric consistency loss Provides the model with fundamental representational capabilities. Total loss function. The definition is as follows: in, Represents the total loss function; This represents the photometric loss, used to constrain the similarity between the rendered color and the original pixel. This represents the depth supervision loss, used to align the model-estimated depth with the actual depth observed by the lidar. This represents the geometric consistency loss; , , These represent the weight coefficients of each loss term. Through joint optimization training of the total loss function, high-precision modeling of the scene is achieved.

[0058] Step S45: Iteratively optimize based on the joint loss to obtain the 3D scene perception model.

[0059] During the training phase, the network parameters of the two-stream feature extraction network and the volume rendering module are iteratively optimized based on the aforementioned joint loss function. By minimizing the joint loss function, the model can learn the volume density and color distribution of the scene, and finally generate a 3D mesh of the scene by extracting zero isosurfaces.

[0060] Step S5: Use the trained 3D scene perception model to process camera images and LiDAR data to generate a consistent 3D scene reconstruction result from multiple perspectives.

[0061] Step S51: Select observation frames based on the continuous observation sequence of the scene to be reconstructed, and use them as rendering supervision references; Step S52: The trained 3D scene perception model extracts volumetric features and synthesizes scenes from real-time sensor data to generate a consistent 3D scene reconstruction result from multiple perspectives.

[0062] This invention also provides a multi-view consistent 3D scene construction system, such as... Figure 4 As shown, it includes: The data acquisition module is used to acquire synchronized raw camera images and LiDAR point clouds; A dual-stream feature extraction network is used to extract features from the original camera image and the lidar point cloud, and to fuse the extracted features to obtain fused sensing features; The volume rendering module is used to perform volume rendering processing based on the fused sensing features to obtain a synthesized image and a synthesized depth. The training module is used to construct a joint loss based on the synthesized image and the synthesized depth, as well as the original camera image and the lidar point cloud, and to use the joint loss to iteratively optimize and train the dual-stream feature extraction network and the volume rendering module to obtain a three-dimensional scene perception model. The inference module is used to process the camera images and LiDAR point clouds to be processed using the three-dimensional scene perception model to generate three-dimensional scene reconstruction results.

[0063] Method performance testing: To verify the actual technical effect of the present invention, this embodiment was validated on publicly available autonomous driving datasets (KITTI and nuScenes).

[0064] In quantitative analysis, this method achieves extremely high perceptual accuracy without relying on manual annotation. Test data shows that on the KITTI dataset, the color synthesis SSIM reaches 0.470, and the PSNR reaches 16.30; the depth synthesis absolute relative error (Abs Rel) is reduced to 0.186, and the root mean square error (RMSE) is 5.982. On the nuScenes dataset, which contains a large number of dynamic objects, the color synthesis SSIM reaches 0.450, and the depth Abs Rel is 0.242. Quantitative data directly demonstrate the high-precision representation capability of this method under various complex road conditions.

[0065] Regarding the qualitative reconstruction effect, such as Figure 5 As shown, this illustrates the color compositing and depth compositing effects of the rendered image. Figure 6 As shown, the left side is the original 2D scene reference image, and the right side is the final 3D scene reconstruction result generated by this method, namely the 3D triangular mesh model. The comparison aims to demonstrate that the perception model of this invention can successfully reconstruct the visual input of a 2D plane into a 3D geometric entity with physical collision volume with high accuracy, thereby verifying the actual technical effect of this invention in obstacle avoidance planning for unmanned vehicles. It can be seen that for moving vehicles in a driving environment, this method successfully overcomes the ghosting problem caused by single-viewpoint offset through multi-viewpoint geometric consistency constraints, accurately restoring the true physical position of dynamic objects. When the perception model reaches the convergence threshold, multiple virtual views are sampled around the vehicle's pose for depth rendering and TSDF fusion, ultimately successfully outputting a 3D triangular mesh model of the scene with complete physical collision volume. This method can synthesize depth maps under new perspectives with high accuracy, accurately restoring the fine depth changes of trees, roads, and vehicles in the scene.

[0066] The experimental results demonstrate in detail that this invention not only completely breaks the dependence of 3D perception models on labeled data, but also achieves high-precision and high-robust 3D obstacle avoidance modeling for complex scenes and irregular obstacles.

[0067] Although the present invention has been disclosed above with reference to embodiments, it is not intended to limit the present invention. Appropriate modifications or equivalent substitutions made by those skilled in the art to the technical solutions of the present invention should be covered within the protection scope of the present invention, which is defined by the claims.

Claims

1. A method for constructing a consistent 3D scene from multiple perspectives, characterized in that, Includes the following steps: Acquire synchronized raw camera images and LiDAR point clouds; A dual-stream feature extraction network is used to extract features from the original camera image and the lidar point cloud, and the extracted features are fused to obtain fused sensing features; The volume rendering module performs volume rendering processing based on the fused sensing features to obtain the synthesized image and synthesized depth; Based on the synthesized image and the synthesized depth, as well as the original camera image and the lidar point cloud, a joint loss is constructed, and the joint loss is used to iteratively optimize and train the dual-stream feature extraction network and the volume rendering module to obtain a 3D scene perception model. The 3D scene perception model is used to process the camera image and LiDAR point cloud to generate a 3D scene reconstruction result.

2. The method as described in claim 1, characterized in that, Feature extraction is performed on the original camera image and the lidar point cloud using a two-stream feature extraction network, including: The original camera image is downsampled using a deep convolutional neural network to extract semantic features of the image from a perspective view. The point cloud of the lidar is voxelized and mapped to a bird's-eye view plane to extract the geometric structure features of the point cloud from the bird's-eye view.

3. The method as described in claim 2, characterized in that, The extracted features are fused to obtain fused sensing features, including: Obtain the three-dimensional spatial points to be sampled, and use the projection matrix to project the three-dimensional spatial points to be sampled onto the perspective view plane and the bird's-eye view plane respectively; Image semantic features are obtained in the perspective plane by bilinear interpolation, and point cloud geometric features are obtained in the bird's-eye view plane. The obtained semantic features of the image and the geometric features of the point cloud are vector-concatenated to obtain fused sensing features.

4. The method as described in claim 1, characterized in that, Based on the fused sensing features, volume rendering processing is performed to obtain a synthesized image and a synthesized depth, including: Discrete sampling is performed along the ray from the camera focus to the pixel to determine the volume density and color value of each sampling point; The transmittance and opacity weights corresponding to each sampling point are calculated based on the volume density. The composite image is obtained by weighted integral of the color values ​​of the sampling points according to the opacity weight, and the composite depth is obtained by weighted integral of the sampling distance of the sampling points according to the opacity weight.

5. The method as described in claim 4, characterized in that, Discrete sampling is performed along the ray from the camera focus to the pixel to determine the volume density and color value of each sampling point, including: The ray beam to be rendered is selected within the pixel area based on a random sampling strategy, and the ray beam corresponding to the lidar point cloud coverage area is selected first. The volume density and color value are obtained by using a multilayer perceptron to analyze the sampling points on the ray beam to be rendered.

6. The method as described in claim 1, characterized in that, A joint loss is constructed based on the synthesized image and the synthesized depth, as well as the original camera image and the LiDAR point cloud, including: Calculate the color deviation between the pixels of the synthesized image and the original camera image to obtain the photometric loss; The distance error between the synthesized depth and the projection depth of the lidar point cloud is calculated to obtain the depth supervision loss; The geometric consistency loss is obtained by calculating the depth matching error between multiple observation viewpoints based on the cyclic reprojection logic. The combined loss is obtained by weighted summation of the photometric loss, the depth supervision loss, and the geometric consistency loss.

7. The method as described in claim 6, characterized in that, Based on the cyclic reprojection logic, the depth matching error between multiple observation views is calculated, resulting in geometric consistency loss, including: By combining the pixel coordinates from the first-person perspective with the estimated depth, the first three-dimensional spatial point is obtained through mapping. Project the first three-dimensional spatial point onto the imaging plane of the second viewpoint to obtain the second viewpoint projection coordinates; The estimated depth corresponding to the projection coordinates of the second viewpoint is obtained and restored to the second three-dimensional space point. The second three-dimensional space point is then projected back to the first viewpoint to obtain the reprojected pixel coordinates. The geometric consistency loss is obtained by measuring the Euclidean distance between the reprojected pixel coordinates and the pixel coordinates under the first viewpoint.

8. The method as described in claim 1, characterized in that, The dual-stream feature extraction network and the volume rendering module are iteratively optimized and trained using the joint loss, including: The network parameters of the dual-stream feature extraction network and the volume rendering module are updated using gradients based on the joint loss. After the joint loss converges, the zero isosurface is extracted to generate a three-dimensional mesh, thus completing the training of the perception model.

9. The method as described in claim 1, characterized in that, The 3D scene perception model is used to process the camera image and LiDAR point cloud to generate a 3D scene reconstruction result, including: Observation frames are selected based on the continuous observation sequence of the scene to be reconstructed, and used as rendering supervision references; The trained 3D scene perception model extracts volumetric features and synthesizes scenes from real-time sensor data to generate consistent 3D scene reconstruction results from multiple perspectives.

10. A multi-view consistent 3D scene construction system, characterized in that, include: The data acquisition module is used to acquire synchronized raw camera images and LiDAR point clouds; A dual-stream feature extraction network is used to extract features from the original camera image and the lidar point cloud, and to fuse the extracted features to obtain fused sensing features; The volume rendering module is used to perform volume rendering processing based on the fused sensing features to obtain a synthesized image and a synthesized depth. The training module is used to construct a joint loss based on the synthesized image and the synthesized depth, as well as the original camera image and the lidar point cloud, and to use the joint loss to iteratively optimize and train the dual-stream feature extraction network and the volume rendering module to obtain a three-dimensional scene perception model. The inference module is used to process the camera images and LiDAR point clouds to be processed using the three-dimensional scene perception model to generate three-dimensional scene reconstruction results.

Images