Three-dimensional scene reconstruction method and device, equipment and storage medium

Abstract

The invention belongs to the technical field of automatic driving, and discloses a three-dimensional scene reconstruction method and device, equipment and a storage medium. According to the invention, the initial point cloud data collected by the laser radar on the target vehicle is preprocessed, the point cloud depth map is generated through the preprocessed target point cloud data, and the point cloud depth map, the target image data collected by the multiple cameras and the camera pose data of each camera are combined to carry out three-dimensional scene reconstruction. The peak signal-to-noise ratio of a reconstruction index can be effectively improved through depth loss, so that three-dimensional scene reconstruction is accurately and effectively carried out.

Description

3D scene reconstruction methods, devices, equipment and storage media Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to a three-dimensional scene reconstruction method, apparatus, device and storage medium. Background Technology

[0002] In the process of autonomous driving, 3D scene reconstruction is usually performed. Currently, scene accumulation mainly relies on data collection. Algorithm engineers propose targeted data collection requirements based on the weaknesses of perception algorithms, and data collection and testing personnel plan collection routes based on these requirements. They also need to consider finding road segments that meet the collection needs as much as possible for data collection. The above-mentioned methods of scene accumulation are extremely costly in terms of time, manpower, and resources. By combining 3D scene reconstruction with dynamic and static scene asset library reconstruction and scene orchestration, new artificially synthesized scenes can be constructed from existing scenes for simulation and verification of autonomous driving algorithms. However, traditional 3D scene reconstruction methods rely on manual annotation of new scenes, which has low accuracy and high cost. Summary of the Invention

[0003] The main purpose of this application is to provide a method, apparatus, device and storage medium for three-dimensional scene reconstruction, aiming to solve the technical problem of how to accurately and effectively reconstruct three-dimensional scenes.

[0004] To achieve the above objectives, this application provides a three-dimensional scene reconstruction method, which includes the following steps: acquiring target point cloud data collected by a lidar on a target vehicle, pose data collected by an inertial sensor, and target image data collected by a camera; constructing an initial point cloud map based on the target point cloud data, the pose data, and the target image data; determining the camera pose data of each camera in the camera coordinate system based on the initial point cloud map; generating a point cloud depth map based on the target point cloud data, and performing three-dimensional scene reconstruction based on the point cloud depth map, the target image data, and the camera pose data to obtain the reconstructed target point cloud map.

[0005] Optionally, determining the camera pose data of each camera in the camera coordinate system based on the initial point cloud map includes: determining keyframe vehicle body pose data and keyframe timestamps based on the initial point cloud map, and determining the image timestamps of each camera acquiring the target image data; finding a target timestamp adjacent to the image timestamp from the keyframe timestamps; determining the camera pose relationship of each camera in the vehicle coordinate system relative to the world coordinate system based on the target pose data corresponding to the target timestamp; and determining the camera pose relationship to determine the camera pose data of each camera in the camera coordinate system.

[0006] Optionally, the step of generating a point cloud depth map based on the target point cloud data, and performing 3D scene reconstruction based on the point cloud depth map, the target image data, and the camera pose data to obtain a reconstructed target point cloud map includes: generating a point cloud depth map based on the target point cloud data; extracting features from the target image data to obtain feature information, and determining prior pose information corresponding to each camera based on the camera pose data and the feature information; performing feature matching on the target image data based on the prior pose information, and triangulating the matched feature points to obtain initial point cloud data; training a 3D Gaussian model based on the initial point cloud data and the point cloud depth map, and performing 3D scene reconstruction using the trained 3D Gaussian model to obtain a reconstructed target point cloud map.

[0007] Optionally, generating a point cloud depth map based on the target point cloud data includes: converting the target point cloud data in the lidar coordinate system to the world coordinate system to obtain point cloud data in the world coordinate system; converting the point cloud data in the world coordinate system to the camera coordinate system based on the camera parameters corresponding to each camera to obtain a point cloud image projection in the camera coordinate system; performing point cloud filtering on the point cloud image projection, and generating a point cloud depth map based on the filtered point cloud data.

[0008] Optionally, after generating a point cloud depth map based on the target point cloud data, and performing 3D scene reconstruction based on the point cloud depth map, the target image data, and the camera pose data to obtain a reconstructed target point cloud map, the method further includes: determining static vector information corresponding to static vector elements based on the initial point cloud map and the target point cloud map; detecting target traffic flow corresponding to the target vehicle; constructing a dynamic 3D scene based on the target traffic flow and the static vector information, and determining dynamic vector information corresponding to dynamic vector elements in the dynamic 3D scene.

[0009] Optionally, determining the static vector information corresponding to the static vector element based on the initial point cloud map and the target point cloud map includes: performing a planar projection on the initial point cloud map to obtain a projected map and an elevation map; annotating the projected map to obtain planar information corresponding to the vector element; determining the spatial information corresponding to the vector element based on the elevation map and the planar information; and spatially aligning the target point cloud map with the spatial information to obtain the aligned static vector information corresponding to the static vector element.

[0010] Optionally, acquiring target point cloud data collected by the lidar on the target vehicle, pose data collected by the inertial sensor, and target image data collected by the camera includes: acquiring initial point cloud data collected by the lidar on the target vehicle, pose data collected by the inertial sensor, and initial image data collected by the camera; performing motion compensation on the static point cloud data and dynamic point cloud data in the initial point cloud data to obtain target point cloud data; and performing distortion correction processing on the initial image data to obtain target image data.

[0011] Furthermore, to achieve the above objectives, this application also provides a three-dimensional scene reconstruction device, comprising: a data acquisition module for acquiring target point cloud data collected by a lidar on a target vehicle, pose data collected by an inertial sensor, and target image data collected by a camera; a point cloud map construction module for constructing an initial point cloud map based on the target point cloud data, the pose data, and the target image data; a data determination module for determining the camera pose data of each camera in the camera coordinate system based on the initial point cloud map; and a three-dimensional scene reconstruction module for generating a point cloud depth map based on the target point cloud data, and performing three-dimensional scene reconstruction based on the point cloud depth map, the target image data, and the camera pose data to obtain the reconstructed target point cloud map.

[0012] In addition, to achieve the above objectives, this application also proposes a three-dimensional scene reconstruction device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the three-dimensional scene reconstruction method described above.

[0013] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the three-dimensional scene reconstruction method described above.

[0014] This application acquires target point cloud data collected by a LiDAR on a target vehicle, pose data collected by an inertial sensor, and target image data collected by a camera. Then, it constructs an initial point cloud map based on the point cloud data, pose data, and target image data. Next, it determines the camera pose data of each camera in the camera coordinate system based on the initial point cloud map. Finally, it generates a point cloud depth map based on the target point cloud data, and performs 3D scene reconstruction based on the point cloud depth map, target image data, and camera pose data to obtain the reconstructed target point cloud map. This application first preprocesses the initial point cloud data collected by the LiDAR on the target vehicle, generates a point cloud depth map from the preprocessed target point cloud data, and combines the point cloud depth map, target image data collected by multiple cameras, and camera pose data of each camera for 3D scene reconstruction. This effectively improves the peak signal-to-noise ratio (PSNR) of the reconstruction index through depth loss, thereby accurately and effectively reconstructing the 3D scene. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 is a flowchart illustrating the first embodiment of the 3D scene reconstruction method of this application; Figure 2 is a flowchart illustrating static point cloud motion compensation of an embodiment of the 3D scene reconstruction method of this application; Figure 3 is a schematic diagram illustrating point cloud motion compensation before and after point cloud motion compensation of an embodiment of the 3D scene reconstruction method of this application; Figure 4 is a flowchart illustrating dynamic point cloud motion compensation of an embodiment of the 3D scene reconstruction method of this application; Figure 5 is a schematic diagram illustrating dynamic target projection of an embodiment of the 3D scene reconstruction method of this application; Figure 6 is a flowchart illustrating the calculation of backoff displacement distance of an embodiment of the 3D scene reconstruction method of this application; Figure 7 is a flowchart illustrating the calculation of camera pose of an embodiment of the 3D scene reconstruction method of this application. Figure 8 is a flowchart illustrating the data flow; Figure 9 is a flowchart illustrating the three-dimensional scene reconstruction of an embodiment of the three-dimensional scene reconstruction method of the present application; Figure 10 is a schematic diagram of the reconstructed target point cloud map of an embodiment of the three-dimensional scene reconstruction method of the present application; Figure 11 is a flowchart illustrating the three-dimensional scene reconstruction method of the present application; Figure 12 is a flowchart illustrating the vector information annotation of an embodiment of the three-dimensional scene reconstruction method of the present application; Figure 13 is a structural block diagram of the first embodiment of the three-dimensional scene reconstruction device of the present application; Figure 14 is a structural schematic diagram of the three-dimensional scene reconstruction device of the hardware operating environment involved in the embodiments of the present application.

[0018] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0019] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0020] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0021] It should be noted that the executing entity of this application can be a computing service device with data processing, network communication and program execution functions, such as a computer.

[0022] Based on this, the present application provides a three-dimensional scene reconstruction method. Referring to FIG1, FIG1 is a flowchart of the first embodiment of the three-dimensional scene reconstruction method of the present application.

[0023] In this embodiment, the three-dimensional scene reconstruction method includes the following steps: Step S10: Acquire target point cloud data collected by the lidar on the target vehicle, pose data collected by the inertial sensor, and target image data collected by the camera.

[0024] Understandably, the truth-based system collects data from various sensors, including LiDAR, cameras, Global Navigation Satellite System (GNSS), and Inertial Measurement Unit (IMU). Before collecting data, the truth-based system needs to ensure time synchronization between the various sensors. First, to ensure synchronized camera triggering and to maximize the synchronization between the LiDAR point cloud acquired from a complete scan and the image, we set the initial rotation angle of the LiDAR, for example, 120 degrees. This means that when the LiDAR rotates to 120 degrees, the camera exposure is triggered synchronously, and the LiDAR data packets are also parsed starting from the 120-degree mark. This method ensures that the parsed point cloud, using the joint calibration parameters of the LiDAR and camera, can be accurately projected onto the image for alignment.

[0025] In a specific implementation, this embodiment can preprocess the initial point cloud data collected by the lidar on the target vehicle to obtain target point cloud data. It can also acquire pose data collected by the inertial sensor and initial image data collected by the camera, and preprocess the initial image data to obtain target image data.

[0026] Furthermore, in order to preprocess the collected data, in this embodiment, step S10 includes: acquiring initial point cloud data collected by the lidar on the target vehicle, pose data collected by the inertial sensor, and initial image data collected by the camera; performing motion compensation on the static point cloud data and dynamic point cloud data in the initial point cloud data to obtain target point cloud data; and performing distortion correction processing on the initial image data to obtain target image data.

[0027] It should be understood that the initial point cloud data collected by the lidar on the target vehicle can be obtained first, as well as the pose data collected by the relationship sensor, which includes Euler angles and position. The initial image data collected by each camera can also be obtained. The initial image data can be processed to remove distortion to obtain the target image data.

[0028] It is understood that this embodiment can perform static point cloud motion compensation on static point cloud data in the initial point cloud data, which may include static obstacles such as road signs and roadside environmental backgrounds. In a feasible embodiment, referring to Figure 2, which is a schematic flowchart of static point cloud motion compensation in an embodiment of the three-dimensional scene reconstruction method of this application, as shown in Figure 2, the timestamp of the point cloud file of the static point cloud data can be parsed offline, and the timestamp information T_ref of each point in the file can be read. Combined with the high-frequency (100HZ) pose data output by the IMU sensor, including the Euler angle dictionary (timestamp:[roll,pitch,yaw]), i.e., IMU timestamp: roll angle, pitch angle, yaw angle, and position dictionary (timestamp:[x,y,z]), i.e., IMU timestamp: Using x-axis, y-axis, and z-axis coordinates, a binary search method is employed to find the nearest preceding and following IMU timestamps from all IMU timestamps collected by the ISU sensor for the laser point's timestamp. Spherical interpolation is then performed for the rotation parameter (Eulerian angles), and linear interpolation is performed for the translation parameter (position). This yields the approximate pose relationship for the laser point's timestamp. Specifically, the approximate pose relationships Euler_relative and Pos_relative for each laser point's timestamp T_ref are found through binary search and interpolation. If preceding and following IMU timestamps cannot be found through binary search, the static pose relationships Euler_ref and Pos_ref for the laser point can be used directly. Each laser point is then transformed from the vehicle frame of reference to the world frame using these approximate pose relationships to obtain reference pose data. This reference pose data is used as a reference to traverse all laser points, while simultaneously performing motion compensation for the static point cloud data using binary search.

[0029] In a specific implementation, referring to Figure 3, which is a schematic diagram of a three-dimensional scene reconstruction method of this application with and without point cloud motion compensation, the upper part of Figure 3 is a schematic diagram without static point cloud motion compensation, and the lower part is a schematic diagram with static point cloud motion compensation. As can be seen from these two figures, with static point cloud motion compensation, the static point cloud data corresponding to railings, trees, etc. are more consistent with the actual railings, trees, etc. compared with those without point cloud motion compensation.

[0030] In this embodiment, static motion compensation of point clouds can only compensate for static targets or dynamic targets with a small speed difference relative to the vehicle. However, for targets with a large speed difference relative to the vehicle, their own motion will cause errors in their size and position. Therefore, dynamic point cloud motion compensation can be performed on the dynamic point cloud data in the initial point cloud data. Referring to Figure 4, which is a flowchart of dynamic point cloud motion compensation in an embodiment of the three-dimensional scene reconstruction method of this application, as shown in Figure 4, dynamic point cloud motion compensation outputs the ground truth information of the dynamic target based on the point cloud target detection algorithm and post-processing. This includes the target's position, length, width, height, yaw angle, and post-processed output information such as velocity and acceleration. First, the data that has undergone static point cloud motion compensation is used to detect targets based on the 3D point cloud target detection algorithm, and the position, size, and velocity of the dynamic target are output. Based on the output target ground truth information, the ground velocity information of the dynamic target in the world coordinate system is calculated. At the same time, based on the continuous frame point cloud, the trajectory information of the dynamic target in the continuous frame segment is output using point cloud trajectory optimization. Then, the point cloud morphology is changed by using the lidar delay compensation angle scanning time ε.

[0031] Furthermore, referring to Figure 5, which is a schematic diagram of dynamic target projection according to an embodiment of the 3D scene reconstruction method of this application, the car within the red box is a dynamic target. This target is a target moving in the same direction as the car from its right front side view. After point cloud motion compensation is performed, the point cloud is projected onto the right front side view image. Compared to a static target, the projected point cloud points still cannot fit the target well. The reason for this phenomenon is that the dynamic target has a large speed difference relative to the car's movement speed, resulting in errors in the position and size of the moving target itself.

[0032] First, the motion compensation for the dynamic target targets a ground truth system equipped with a rotating lidar. The point cloud data output by this rotating lidar is used to construct a ground truth system in the cloud. Therefore, this embodiment can utilize the continuous frame point cloud after motion compensation of the static point cloud, and through target detection ground truth algorithm and post-processing, output the ground velocity information of the dynamic target in the world coordinate system. Simultaneously, based on the continuous frame point cloud, trajectory optimization can also be used to output the trajectory information of the dynamic target within the continuous frame segment. For example, in the k-th frame of the continuous frame segment, it is necessary to perform ground truth motion compensation for the dynamic car in the red box on the right front side of Figure 5 in the k-th frame. First, based on the ground truth algorithm, the 3D bounding box information of the dynamic car in the current k-th frame is output, including the x, y, z, l, w, h, yaw angles and the corresponding velocity information. The velocity information calculation of the dynamic car target in the current k-th frame requires strictly transforming the ground truth bounding box coordinates of the dynamic car target to the world coordinate system. Based on the information of multiple frames (5 frames) before and after the dynamic car target, for example, the coordinates of the dynamic target in the world coordinate system in the t-th frame are ( , , ), the coordinates of the dynamic target in the world coordinate system in frame t-5 are ( , , ), calculate the target's movement distance. and the corresponding time difference :

[0033]

[0034] Calculate the velocity information of the dynamic Car target in the current k-th frame. for: .

[0035] Based on the pre-configured real LiDAR resolution angle θ (this angle value will vary depending on the LiDAR and camera calibration scheme), the time taken for the LiDAR to scan from the initial resolution angle θ to the tangent point of the left front of the dynamic car is calculated. Combined with the vehicle's speed information in the current frame The required backtracking displacement distance D, in meters, is calculated for the point cloud points contained in the 3D bounding box of the dynamic Car. As shown in the BEV viewpoint point cloud compensation diagram below, based on the delay compensation angle α and the calculated absolute ground velocity of the current moving target (blue box), the required backtracking distance D for the point cloud points within the box is calculated. Referring to Figure 6, which is a flowchart illustrating the calculation of the backtracking displacement distance in an embodiment of the 3D scene reconstruction method of this application, as shown in Figure 6... . The frame rate of the LiDAR.

[0036] In this embodiment, the data preprocessing section, in addition to typical data parsing and image distortion correction, includes motion compensation for the point cloud, which is divided into dynamic and static motion compensation. Static compensation uses general IMU-based vehicle pose information combined with binary search for pose interpolation. The key lies in dynamic point cloud compensation. Based on prior perception results, a post-processing algorithm is used to calculate the ground velocity of the dynamic target. Combined with a pre-set compensation angle from the rotating LiDAR, the time it takes for the LiDAR point to scan from the front to the rear of the vehicle is calculated, yielding the distance traveled in the vehicle's forward direction, thus achieving spatial retraction of the point cloud. Through dynamic and static point cloud compensation, the point cloud morphology is optimized, which is crucial for subsequent SLAM mapping optimization. Optimizing the depth map obtained from the point cloud as an aid to 3DGS training is also essential.

[0037] Step S20: Construct an initial point cloud map based on the target point cloud data, the pose data, and the target image data.

[0038] Understandably, this embodiment can perform SLAM mapping based on motion-compensated target point cloud data. Specifically, it can construct a map of a continuous frame scene based on continuous frame target image data, target point cloud data, and pose data collected by IMU sensors. The continuous frame scene map construction process mainly includes preprocessing, feature extraction, IMU pre-integration (fusion of wheel speed observations), and factor map back-end optimization. Based on the point cloud data with dynamic and static target motion compensation completed in the point cloud preprocessing stage, feature extraction is performed. In the feature extraction stage, ground points are segmented, parallel points are removed, and planar and edge features are extracted. In the IMU pre-integration stage, odometry information is input for forward optimization prediction to construct the predicted score for calibration distortion. Factor map optimization uses GNSS factors, loop closure factors, and ground constraint factors to optimize the pose and construct a local map. In this way, an initial point cloud map of the entire scene for continuous frames can be generated.

[0039] Step S30: Determine the camera pose data of each camera in the camera coordinate system based on the initial point cloud map.

[0040] Understandably, while SLAM constructs the initial point cloud map, it also optimizes the pose data. Based on the optimized keyframe vehicle pose data, the camera pose data of each camera in the camera coordinate system can be obtained.

[0041] Furthermore, in order to accurately obtain camera pose data, in this embodiment, step S30 includes: determining keyframe vehicle pose data and keyframe timestamps based on the initial point cloud map, and determining the image timestamps of each camera acquiring the target image data; finding the target timestamps adjacent to the image timestamps from the keyframe timestamps; determining the camera pose relationship of each camera in the vehicle coordinate system relative to the world coordinate system based on the target pose data corresponding to the target timestamps; and determining the camera pose data of each camera in the camera coordinate system based on the camera pose relationship.

[0042] It should be understood that pose interpolation can be performed on the six cameras of the ground truth system at different times. A preliminary step in 3DGS is to use the 3D reconstruction tool colmap to extract the initial point cloud of the scene for optimization. Traditional colmap's Structure from Motion (SfM) is a process of reconstructing a static scene, with input being target image data obtained from a single camera capturing the same scene from different viewpoints. During the reconstruction process, the pose information of that camera is generated. However, in the field of autonomous driving, colmap's input is no longer a single camera, but multiple cameras mounted at fixed positions on the vehicle. The images captured are no longer of a static scene, but rather a segment of traffic flow and scene information for scene reconstruction. Based on the keyframe vehicle pose data optimized by SLAM mapping, pose interpolation can be performed on the cameras corresponding to the multi-camera images input to colmap, obtaining the camera pose data of the multiple cameras at each time step. This serves as the prior pose information for colmap to run Structure from Motion (SFM).

[0043] In a specific implementation, referring to Figure 7, which is a flowchart illustrating the calculation of camera pose data in an embodiment of the 3D scene reconstruction method of this application, as shown in Figure 7, the SLAM-optimized keyframe vehicle body pose data pose.txt is parsed to generate a pose dictionary lookup table, which can be represented as follows:

[0044]

[0045]

[0046] in, Let these represent the roll angle, pitch angle, and yaw angle in frame t, respectively. Let x, y, and z represent the x-axis coordinates, y-axis coordinates, and z-axis coordinates of the t-th frame, respectively. Indicates the keyframe timestamp.

[0047] Image timestamps based on sorted_timestamp_list and target image data captured from each viewpoint Using a binary search method, the sorted_timestamp_list timestamps of the preceding and following IMU sensors closest to the image timestamp of the target image captured from that viewpoint are determined. Specifically, the target timestamps adjacent to the image timestamps are closest_Tref_left and closest_Tref_right. Then, the Euler angle values ​​and position information corresponding to the preceding and following timestamps are obtained respectively. Based on... and The process involves converting Euler angles to quaternions, and then using spherical interpolation to interpolate the quaternion values ​​corresponding to the timestamps of the images captured from each viewpoint. Then convert back to Euler angles Calculating the camera position information corresponding to the time of the image captured from each viewpoint requires... and Perform linear interpolation to obtain ,right and Perform linear interpolation to obtain The calculation formula is:

[0048]

[0049]

[0050]

[0051]

[0052]

[0053] Using the method described above, the Euler angles and position parameters from the camera to the world frame for each frame captured from each viewpoint can be calculated, i.e., the camera pose relationship of each camera in the vehicle coordinate system relative to the world coordinate system. and Based on the camera pose relationship corresponding to each frame of image under each viewpoint obtained above, and combined with the fixed intrinsic parameter information of the camera, the rotation and translation matrix from the camera to the world frame corresponding to the timestamp of each frame image can be calculated. Thus, the camera pose data of each camera in the camera coordinate system can be determined based on the rotation and translation matrix.

[0054] This embodiment enables pose priors from multiple cameras at different times. Compared to the common method of directly interpolating vehicle pose information based on IMU, this embodiment first performs point cloud SLAM mapping based on preprocessed target point cloud data. During point cloud mapping, keyframe poses are optimized. Interpolation is then performed based on the optimized keyframe pose data to obtain camera pose data for each camera at different times. Compared to existing solutions, the camera pose data in this embodiment is optimized, especially for situations with weak GNSS signals or unstable vehicle pose information due to occlusion in tunnels or under viaducts. Interpolation based on the optimized keyframe pose data yields better multi-camera pose data at different times, providing superior prior information for subsequent feature matching, triangulation, and other processes.

[0055] Step S40: Generate a point cloud depth map based on the target point cloud data, and perform 3D scene reconstruction based on the point cloud depth map, the target image data, and the camera pose data to obtain the reconstructed target point cloud map.

[0056] Understandably, in the field of autonomous driving, the input for 3D scene reconstruction may no longer be the traditional, simple target image data. The reconstructed scene is more complex and is no longer the reconstruction of a single static object. The module's reconstruction input includes motion-compensated target point cloud data, target image data captured by multiple cameras fixed on the data acquisition vehicle, and camera pose data of each camera. The scene to be reconstructed is the 3D reconstruction of the dynamic and static targets around the data acquisition vehicle while it is moving, resulting in a reconstructed target point cloud map.

[0057] This embodiment acquires target point cloud data collected by the LiDAR on the target vehicle, pose data collected by the inertial sensor, and target image data collected by the camera. Then, an initial point cloud map is constructed based on the target point cloud data, pose data, and target image data. Next, the camera pose data of each camera in the camera coordinate system is determined based on the initial point cloud map. Finally, a point cloud depth map is generated based on the target point cloud data. Finally, a 3D scene reconstruction is performed based on the point cloud depth map, target image data, and camera pose data to obtain the reconstructed target point cloud map. This embodiment first preprocesses the initial point cloud data collected by the LiDAR on the target vehicle, generates a point cloud depth map from the preprocessed target point cloud data, and combines the point cloud depth map, target image data collected by multiple cameras, and camera pose data of each camera for 3D scene reconstruction. This effectively improves the peak signal-to-noise ratio (PSNR) of the reconstruction index through depth loss, thereby accurately and effectively reconstructing the 3D scene.

[0058] Referring to Figure 8, which is a flowchart of the second embodiment of the three-dimensional scene reconstruction method of this application.

[0059] Based on the first embodiment described above, in this embodiment, step S40 includes: step S401: generating a point cloud depth map based on the target point cloud data.

[0060] Furthermore, in order to generate a point cloud depth map, in this embodiment, step S401 includes: converting the target point cloud data in the lidar coordinate system to the world coordinate system to obtain point cloud data in the world coordinate system; converting the point cloud data in the world coordinate system to the camera coordinate system based on the camera parameters corresponding to each camera to obtain a point cloud image projection in the camera coordinate system; performing point cloud filtering on the point cloud image projection, and generating a point cloud depth map based on the filtered point cloud data.

[0061] It is understandable that, referring to Figure 9, which is a schematic diagram of the three-dimensional scene reconstruction process according to an embodiment of the three-dimensional scene reconstruction method of this application, the motion-compensated point cloud, i.e., the target point cloud data, can be converted to the world coordinate system to obtain point cloud data in the world coordinate system. Specifically, based on the motion-compensated target point cloud data and the SLAM-optimized keyframe pose data, the same method as the above-mentioned interpolation method for obtaining camera poses at different times of multiple cameras can be used to obtain the pose information of the LiDAR at different times. Through the obtained pose information, the target point cloud data is converted to the world coordinate system, and the point cloud data in the world coordinate system is converted to the camera coordinate system based on the camera parameters corresponding to each camera to obtain the point cloud image projection in the camera coordinate system. Specifically, the pose information of each camera at different times obtained by interpolation and the corresponding camera intrinsic parameters can be combined to project the point cloud data in the world coordinate system onto the image. The conversion formula is: In the formula, Represents the pixel coordinates of the projection of the point cloud image. Indicates camera intrinsic parameters. This indicates the transformation relationship from the camera coordinate system to the vehicle coordinate system. This indicates the transformation relationship from the lidar coordinate system to the vehicle coordinate system. This represents the point cloud data under the vehicle series.

[0062] In the specific implementation, the point cloud image projection is then filtered, and a point cloud depth map is generated based on the filtered point cloud data.

[0063] Step S402: Extract features from the target image data to obtain feature information, and determine the prior pose information corresponding to each camera based on the camera pose data and the feature information.

[0064] Understandably, by acquiring camera pose data from each camera at different times and combining it with the general processing flow of Colmap, the directory structure and files required for Colmap to run are constructed. Feature extraction can be performed on the target image data first, such as using the SIFT operator. After feature extraction, a database file named database.db (sqlite3) will be generated, containing the feature information.

[0065] It should be understood that, because this embodiment is a reconstruction in the field of autonomous driving, and the intrinsic parameter information of each camera is fixed, the fixed intrinsic parameter information is written into the camera table in dababase.db by editing the script. At the same time, the camera pose data of each camera at different times obtained based on pose interpolation is also written into the prior_pose table in database.db as the prior pose information of each camera.

[0066] Step S403: Perform feature matching on the target image data based on the prior pose information, and triangulate the matched feature points to obtain initialized point cloud data.

[0067] In the specific implementation, exhaustive matching can be performed on the target image data based on prior pose information, and the matched feature points can be triangulated to obtain initialized point cloud data, which can include the position information of the point cloud in the world system (which can be used as the origin of the Gaussian ellipse) and the RGB information of the point cloud points.

[0068] Step S404: Train a 3D Gaussian model based on the initial point cloud data and the point cloud depth map, and reconstruct the three-dimensional scene using the trained 3D Gaussian model to obtain the reconstructed target point cloud map.

[0069] In practical implementation, the position of each point in the point cloud depth map can be used as the position of a 3D Gaussian ellipse. A 3D Gaussian covariance matrix is ​​randomly initialized based on the initial point cloud data, and 3DGS training (i.e., 3D Gaussian model training) is performed. The trained 3D Gaussian model is then used for 3D scene reconstruction to obtain the reconstructed target point cloud map. Referring to Figure 10, which is a schematic diagram of the reconstructed target point cloud map according to an embodiment of the 3D scene reconstruction method of this application, point cloud maps from various angles can be obtained based on the target point cloud map.

[0070] This embodiment is based on 3DGS training using 3D Gaussian splashing. Traditional methods mostly do not utilize the depth information of the image, or the depth information comes from a large model. This embodiment, after preprocessing the point cloud data, uses optimized keyframe pose data to interpolate the point cloud pose. Combined with camera pose data from multiple cameras at different times and fixed intrinsic parameter information, the point cloud is projected onto a 2D image. Then, by combining projection point filtering and normalization, a point cloud depth map is generated. The ground truth depth value used for 3DGS training is used to assist training and reduce the pixel error of reprojecting the trained 3D point cloud points onto the image.

[0071] This embodiment generates a point cloud depth map based on the target point cloud data, then extracts features from the target image data to obtain feature information, and determines the prior pose information corresponding to each camera based on the camera pose data and feature information. Next, feature matching is performed on the target image data based on the prior pose information, and the matched feature points are triangulated to obtain initial point cloud data. A 3D Gaussian model is then trained based on the initial point cloud data and the point cloud depth map, and the trained 3D Gaussian model is used for 3D scene reconstruction to obtain the reconstructed target point cloud map. This embodiment uses the initial point cloud data and point cloud depth map to train a 3D Gaussian model, which serves as a depth aid in the 3D Gaussian model training stage. Through depth loss, it effectively improves reconstruction metrics, reduces pixel errors when reprojecting reconstructed point cloud points onto a 2D image, and ultimately improves the accuracy of 3D scene reconstruction through the trained 3D Gaussian model.

[0072] Referring to Figure 11, which is a flowchart of the third embodiment of the three-dimensional scene reconstruction method of this application.

[0073] Based on the above embodiments, in this embodiment, after step S40, the method further includes: step S50: determining the static vector information corresponding to the static vector element based on the initial point cloud map and the target point cloud map.

[0074] Understandably, static vector elements can be elements such as zebra crossings or traffic lights, and static vector information can include the spatial location and attributes of static vector elements.

[0075] Furthermore, in order to annotate static vector information, in this embodiment, step S50 includes: performing planar projection on the initial point cloud map to obtain a projected map and an elevation map; annotating the projected map to obtain planar information corresponding to the vector elements; determining the spatial information corresponding to the vector elements based on the elevation map and the planar information; and spatially aligning the target point cloud map with the spatial information to obtain the static vector information corresponding to the aligned static vector elements.

[0076] It is understandable that, referring to Figure 12, which is a schematic flowchart of vector information annotation in an embodiment of the 3D scene reconstruction method of this application, the initial point cloud map can be projected onto a plane and then subjected to spatial color transformation. That is, the 3D point cloud map generated based on SLAM mapping is projected onto a 2D image to generate two parts: a 2D top-view intensity projection map and a 2D elevation map. The 2D top-view intensity projection can perform ground element detection, that is, the planar information corresponding to the vector elements is obtained through pre-annotation of the model, which may include the position and attribute information of vector elements such as lane lines and road markings. The 2D elevation map can project the pixel coordinates on the image back onto the original 3D point cloud map to obtain the spatial information corresponding to the vector elements, which may include the 3D spatial position and attribute information in the world frame.

[0077] It should be understood that the element alignment attributes are supplemented by the target point cloud map and control information after the 3D scene reconstruction. The aligned vector elements can be based on the static 3D reconstructed point cloud to obtain the static vector information corresponding to the static vector elements, which may include the mean, covariance matrix, opacity, coefficients of the spherical harmonic function, and other attributes of the static vector elements.

[0078] Step S60: Detect the target traffic flow corresponding to the target vehicle.

[0079] Understandably, traffic flow can be detected through 3D target detection, and then a world coordinate transformation can be performed to obtain the target traffic flow. The target traffic flow can be the traffic flow of the target vehicle at the current moment, and may include information such as pedestrian flow and traffic lights.

[0080] Step S70: Construct a dynamic three-dimensional scene based on the target traffic flow and the static vector information, and determine the dynamic vector information corresponding to the dynamic vector elements in the dynamic three-dimensional scene.

[0081] It should be understood that, based on the 3D target detection model, the observation status of dynamic targets detected in consecutive frames is considered in both time and space. Firstly, based on the detection results, severely occluded dynamic targets in multi-view images are removed. Occluded targets are not continuously and effectively observed from multiple perspectives, resulting in a large number of extracted features failing to match effectively during feature matching, significantly reducing the number and quality of initial point clouds for 3DGS training. For the same dynamic target, the range of change in its detected azimuth angle is verified. If the target azimuth angle change exceeds a predetermined threshold (e.g., 90 degrees), the images and point clouds of the target observed from multiple perspectives are stored. The dynamic target is then reconstructed using the 3D reconstruction process described in the above embodiment, and the reconstruction result is stored in the dynamic target reconstruction asset library.

[0082] In practical implementation, the reconstructed static scene elements and dynamic target elements can be arranged in a modular form. Based on existing dynamic and static reconstruction asset libraries, new traffic flow scenes can be artificially synthesized to construct dynamic 3D scenes. For example, in a simulation scenario, the simulator sets the required simulation scenario (e.g., a ghost pedestrian scene at an intersection). An existing intersection scene is selected from the static reconstruction scene asset library, with the vehicle positioned in front of the pedestrian crossing. A reconstructed pedestrian model is selected from the dynamic reconstruction scene asset library, and combined with rotation and translation operations, the pedestrian model is precisely inserted into the selected static scene. Simultaneously, scene arrangement also supports the creation of complex traffic flows. Multiple dynamic targets can be selected from the dynamic reconstruction asset library and inserted into the same static scene to create complex traffic flows. After constructing the dynamic 3D scene, the dynamic vector information corresponding to the dynamic vector elements in the dynamic 3D scene can be determined, such as pedestrian flow information and vehicle flow information, thereby enabling effective autonomous driving based on the dynamic vector information.

[0083] In this embodiment, the 3D reconstruction of the static scene is used not only as a simulation background to add dynamic reconstruction targets and realize customized traffic flow, but also, by aligning the static reconstructed scene with the annotations of automated vector elements, it enables the automated annotation of vector elements to be achieved even when artificially synthesizing new scenes from the same static scene with different traffic participants. By aligning the existing automated annotation vector elements based on point cloud maps with the scene reconstruction, the annotations of vector elements can be directly projected onto the new scene in combination with camera pose, intrinsic parameters, etc., to quickly accumulate ground truth data for the iteration of perception algorithms.

[0084] This embodiment determines the static vector information corresponding to static vector elements based on the initial point cloud map and the target point cloud map. Then, it detects the target traffic flow corresponding to the target vehicle. Next, it constructs a dynamic 3D scene based on the target traffic flow and the static vector information, and determines the dynamic vector information corresponding to the dynamic vector elements in the dynamic 3D scene. This embodiment constructs a dynamic 3D scene based on the target traffic flow and static vector information. It can insert dynamically reconstructed targets into the same static scene based on a custom traffic flow scene arrangement. While performing simulation testing, it automatically combines ground truth vector annotation information with camera pose, intrinsic parameters, and other information to project the ground truth onto a new artificially synthesized scene, achieving rapid accumulation of ground truth scenes and supporting iterative perception algorithms.

[0085] Referring to Figure 13, which is a structural block diagram of the first embodiment of the three-dimensional scene reconstruction device of this application.

[0086] As shown in Figure 13, the three-dimensional scene reconstruction device proposed in this application includes: a data acquisition module 10, used to acquire target point cloud data collected by the lidar on the target vehicle, pose data collected by the inertial sensor, and target image data collected by the camera; a point cloud map construction module 20, used to construct an initial point cloud map based on the target point cloud data, the pose data, and the target image data; a data determination module 30, used to determine the camera pose data of each camera in the camera coordinate system based on the initial point cloud map; and a three-dimensional scene reconstruction module 40, used to generate a point cloud depth map based on the target point cloud data, and perform three-dimensional scene reconstruction based on the point cloud depth map, the target image data, and the camera pose data to obtain the reconstructed target point cloud map.

[0087] This embodiment acquires target point cloud data collected by the LiDAR on the target vehicle, pose data collected by the inertial sensor, and target image data collected by the camera. Then, an initial point cloud map is constructed based on the target point cloud data, pose data, and target image data. Next, the camera pose data of each camera in the camera coordinate system is determined based on the initial point cloud map. Finally, a point cloud depth map is generated based on the target point cloud data. Finally, a 3D scene reconstruction is performed based on the point cloud depth map, target image data, and camera pose data to obtain the reconstructed target point cloud map. This embodiment first preprocesses the initial point cloud data collected by the LiDAR on the target vehicle, generates a point cloud depth map from the preprocessed target point cloud data, and combines the point cloud depth map, target image data collected by multiple cameras, and camera pose data of each camera for 3D scene reconstruction. This effectively improves the peak signal-to-noise ratio (PSNR) of the reconstruction index through depth loss, thereby accurately and effectively reconstructing the 3D scene.

[0088] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of this application. In practical applications, those skilled in the art can select some or all of it to achieve the purpose of this embodiment according to actual needs, and no restrictions are imposed here.

[0089] In addition, for technical details not described in detail in this embodiment, please refer to the three-dimensional scene reconstruction method provided in any embodiment of this application, which will not be repeated here.

[0090] Based on the first embodiment of the three-dimensional scene reconstruction device described in this application, a second embodiment of the three-dimensional scene reconstruction device of this application is proposed.

[0091] In this embodiment, the data determination module 30 is further configured to determine keyframe vehicle body pose data and keyframe timestamps based on the initial point cloud map, and determine the image timestamps of each camera acquiring the target image data; search for target timestamps adjacent to the image timestamps from the keyframe timestamps; determine the camera pose relationship of each camera in the vehicle coordinate system relative to the world coordinate system based on the target pose data corresponding to the target timestamps; and determine the camera pose data of each camera in the camera coordinate system based on the camera pose relationship.

[0092] Furthermore, the 3D scene reconstruction module 40 is also used to generate a point cloud depth map based on the target point cloud data; extract features from the target image data to obtain feature information, and determine the prior pose information corresponding to each camera based on the camera pose data and the feature information; perform feature matching on the target image data based on the prior pose information, and triangulate the matched feature points to obtain initialized point cloud data; train a 3D Gaussian model based on the initialized point cloud data and the point cloud depth map, and perform 3D scene reconstruction using the trained 3D Gaussian model to obtain the reconstructed target point cloud map.

[0093] Furthermore, the 3D scene reconstruction module 40 is also used to convert the target point cloud data in the lidar coordinate system to the world coordinate system to obtain point cloud data in the world coordinate system; convert the point cloud data in the world coordinate system to the camera coordinate system based on the camera parameters corresponding to each camera to obtain the point cloud image projection in the camera coordinate system; perform point cloud filtering on the point cloud image projection, and generate a point cloud depth map based on the filtered point cloud data.

[0094] Furthermore, the three-dimensional scene reconstruction device also includes a vector annotation module, used to determine the static vector information corresponding to the static vector elements based on the initial point cloud map and the target point cloud map; detect the target traffic flow corresponding to the target vehicle; construct a dynamic three-dimensional scene based on the target traffic flow and the static vector information, and determine the dynamic vector information corresponding to the dynamic vector elements in the dynamic three-dimensional scene.

[0095] Furthermore, the vector annotation module is also used to perform planar projection on the initial point cloud map to obtain a projected map and an elevation map; to annotate the projected map to obtain planar information corresponding to the vector elements; to determine the spatial information corresponding to the vector elements based on the elevation map and the planar information; and to spatially align the target point cloud map with the spatial information to obtain static vector information corresponding to the aligned static vector elements.

[0096] Furthermore, the data acquisition module 10 is also used to acquire initial point cloud data collected by the lidar on the target vehicle, pose data collected by the inertial sensor, and initial image data collected by the camera; to perform motion compensation on the static point cloud data and dynamic point cloud data in the initial point cloud data to obtain target point cloud data; and to perform distortion correction processing on the initial image data to obtain target image data.

[0097] Other embodiments or specific implementations of the three-dimensional scene reconstruction device of this application can be found in the above-described method embodiments, and will not be repeated here.

[0098] This application provides a three-dimensional scene reconstruction device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the three-dimensional scene reconstruction method in the above embodiment 1.

[0099] Referring now to Figure 14, a schematic diagram of a structure suitable for implementing a three-dimensional scene reconstruction device according to embodiments of this application is shown. The three-dimensional scene reconstruction device in embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. The three-dimensional scene reconstruction device shown in Figure 14 is merely an example and should not impose any limitations on the functionality and scope of use of embodiments of this application.

[0100] As shown in Figure 14, the 3D scene reconstruction device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the 3D scene reconstruction device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. The communication device 1009 allows the 3D scene reconstruction device to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows a 3D scene reconstruction device with various systems, it should be understood that it is not required to implement or possess all of the systems shown. More or fewer systems may be implemented alternatively.

[0101] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0102] The 3D scene reconstruction device provided in this application, employing the 3D scene reconstruction method in the above embodiments, can solve the technical problem of how to accurately and effectively reconstruct 3D scenes. Compared with the prior art, the beneficial effects of the 3D scene reconstruction device provided in this application are the same as those of the 3D scene reconstruction method provided in the above embodiments, and other technical features of the 3D scene reconstruction device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0103] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0104] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0105] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the three-dimensional scene reconstruction method in the above embodiments.

[0106] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0107] The aforementioned computer-readable storage medium may be included in the 3D scene reconstruction device; or it may exist independently and not be assembled into the 3D scene reconstruction device.

[0108] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by a 3D scene reconstruction device, the 3D scene reconstruction device: acquires target point cloud data collected by a lidar on a target vehicle, pose data collected by an inertial sensor, and target image data collected by a camera; constructs an initial point cloud map based on the target point cloud data, the pose data, and the target image data; determines the camera pose data of each camera in the camera coordinate system based on the initial point cloud map; generates a point cloud depth map based on the target point cloud data; and performs 3D scene reconstruction based on the point cloud depth map, the target image data, and the camera pose data to obtain the reconstructed target point cloud map.

[0109] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages—such as Python, Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0110] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0111] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0112] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described 3D scene reconstruction method, thereby solving the technical problem of how to accurately and effectively perform 3D scene reconstruction. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the 3D scene reconstruction method provided in the above embodiments, and will not be repeated here.

[0113] The above description is only a part of the embodiments of this application and does not limit the scope of protection of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included in the scope of protection of this application.

Claims

1. A method for reconstructing a three-dimensional scene, characterized in that, The 3D scene reconstruction method includes the following steps: acquiring target point cloud data collected by the lidar on the target vehicle, pose data collected by the inertial sensor, and target image data collected by the camera; constructing an initial point cloud map based on the target point cloud data, the pose data, and the target image data; determining the camera pose data of each camera in the camera coordinate system based on the initial point cloud map; generating a point cloud depth map based on the target point cloud data, and performing 3D scene reconstruction based on the point cloud depth map, the target image data, and the camera pose data to obtain the reconstructed target point cloud map.

2. The three-dimensional scene reconstruction method as described in claim 1, characterized in that, The step of determining the camera pose data of each camera in the camera coordinate system based on the initial point cloud map includes: determining keyframe vehicle body pose data and keyframe timestamps based on the initial point cloud map, and determining the image timestamps of each camera acquiring the target image data; finding the target timestamps adjacent to the image timestamps from the keyframe timestamps; determining the camera pose relationship of each camera in the vehicle coordinate system relative to the world coordinate system based on the target pose data corresponding to the target timestamps; and determining the camera pose data of each camera in the camera coordinate system based on the camera pose relationship.

3. The three-dimensional scene reconstruction method as described in claim 1, characterized in that, The step of generating a point cloud depth map based on the target point cloud data and reconstructing a 3D scene based on the point cloud depth map, the target image data, and the camera pose data to obtain a reconstructed target point cloud map includes: generating a point cloud depth map based on the target point cloud data; extracting features from the target image data to obtain feature information, and determining prior pose information corresponding to each camera based on the camera pose data and the feature information; performing feature matching on the target image data based on the prior pose information, and triangulating the matched feature points to obtain initial point cloud data; training a 3D Gaussian model based on the initial point cloud data and the point cloud depth map, and reconstructing a 3D scene using the trained 3D Gaussian model to obtain a reconstructed target point cloud map.

4. The three-dimensional scene reconstruction method as described in claim 3, characterized in that, The step of generating a point cloud depth map based on the target point cloud data includes: converting the target point cloud data in the lidar coordinate system to the world coordinate system to obtain point cloud data in the world coordinate system; converting the point cloud data in the world coordinate system to the camera coordinate system based on the camera parameters corresponding to each camera to obtain a point cloud image projection in the camera coordinate system; performing point cloud filtering on the point cloud image projection, and generating a point cloud depth map based on the filtered point cloud data.

5. The three-dimensional scene reconstruction method according to any one of claims 1 to 4, characterized in that, After generating a point cloud depth map based on the target point cloud data, and reconstructing a 3D scene based on the point cloud depth map, the target image data, and the camera pose data to obtain a reconstructed target point cloud map, the method further includes: determining static vector information corresponding to static vector elements based on the initial point cloud map and the target point cloud map; detecting the target traffic flow corresponding to the target vehicle; constructing a dynamic 3D scene based on the target traffic flow and the static vector information, and determining the dynamic vector information corresponding to the dynamic vector elements in the dynamic 3D scene.

6. The three-dimensional scene reconstruction method as described in claim 5, characterized in that, The step of determining the static vector information corresponding to the static vector element based on the initial point cloud map and the target point cloud map includes: performing a planar projection on the initial point cloud map to obtain a projected map and an elevation map; annotating the projected map to obtain the planar information corresponding to the vector element; determining the spatial information corresponding to the vector element based on the elevation map and the planar information; and spatially aligning the target point cloud map with the spatial information to obtain the aligned static vector information corresponding to the static vector element.

7. The three-dimensional scene reconstruction method according to any one of claims 1 to 4, characterized in that, The acquisition of target point cloud data collected by the lidar on the target vehicle, pose data collected by the inertial sensor, and target image data collected by the camera includes: acquiring initial point cloud data collected by the lidar on the target vehicle, pose data collected by the inertial sensor, and initial image data collected by the camera; performing motion compensation on the static point cloud data and dynamic point cloud data in the initial point cloud data to obtain target point cloud data; and performing distortion correction processing on the initial image data to obtain target image data.

8. A three-dimensional scene reconstruction device, characterized in that, The 3D scene reconstruction device includes: a data acquisition module for acquiring target point cloud data collected by a lidar on a target vehicle, pose data collected by an inertial sensor, and target image data collected by a camera; a point cloud map construction module for constructing an initial point cloud map based on the target point cloud data, the pose data, and the target image data; a data determination module for determining the camera pose data of each camera in the camera coordinate system based on the initial point cloud map; and a 3D scene reconstruction module for generating a point cloud depth map based on the target point cloud data, and performing 3D scene reconstruction based on the point cloud depth map, the target image data, and the camera pose data to obtain the reconstructed target point cloud map.

9. A three-dimensional scene reconstruction device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the three-dimensional scene reconstruction method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the three-dimensional scene reconstruction method as described in any one of claims 1 to 7.

Images