Training method, device, equipment, medium and product of scene reconstruction model

Abstract

A training method, apparatus, device, medium, and product for a scene reconstruction model are disclosed. The method involves acquiring an image dataset collected by the vehicle under its original trajectory and a first point cloud dataset synchronized with the image dataset in time. A first reconstruction model is trained using the image dataset and the first point cloud dataset. The first reconstruction model is then used to render an image under a new trajectory, resulting in a rendered image under the new trajectory. A pre-trained video restoration model is used to restore the rendered image under the new trajectory, resulting in a restored image under the new trajectory. Finally, a second reconstruction model is trained using the restored image under the new trajectory, the Gaussian parameters of the first reconstruction model, the first pose of the vehicle under the original trajectory, and the second pose of the vehicle under the new trajectory. This disclosure helps to bridge the domain gap between the restored image under the new trajectory and the acquired image under the original trajectory, improving the rendering quality of the reconstruction model.

Description

Technical Field

[0001] This disclosure relates to the fields of artificial intelligence, scene reconstruction, and autonomous driving technology, and in particular to a training method, apparatus, device, medium, and product for a scene reconstruction model. Background Technology

[0002] In autonomous driving scene reconstruction tasks, the reconstruction of the driving scene is generally completed by inputting a driving video. Due to the sparsity of viewpoints in autonomous driving data and the dynamic complexity of driving scenes, 3D reconstruction of autonomous driving scenes faces greater challenges compared to traditional 3D scene reconstruction.

[0003] In related technologies, the datasets for autonomous driving do not contain a wealth of multi-view images. The test set is generally obtained by extracting frames from the original trajectory data, while the training set uses the remaining images after frame extraction. This makes the related technologies mainly focus on the scene reconstruction quality under the original trajectory. However, when using the reconstructed autonomous driving scene to render new trajectories such as gradual lane changes and lane shifts, the rendering effect is poor and it is easy to have phenomena such as ghosting and blurring. Summary of the Invention

[0004] To address the aforementioned technical issues, this disclosure provides a training method, apparatus, device, medium, and product for a scene reconstruction model, thereby improving image rendering quality in autonomous driving scenarios using the trained scene reconstruction model.

[0005] The first aspect of this disclosure provides a method for training a scene reconstruction model, comprising:

[0006] The system acquires an image dataset collected by the vehicle along its original trajectory and a first point cloud dataset that is time-synchronized with the image dataset. The first point cloud dataset is acquired by a lidar.

[0007] A first reconstruction model is trained using the image dataset and the first point cloud dataset;

[0008] The first reconstruction model is used to render the image under the new trajectory to obtain the rendered image under the new trajectory;

[0009] The rendered image under the new trajectory is repaired using a pre-trained video restoration model to obtain the restored image under the new trajectory;

[0010] Using the repaired image under the new trajectory, the Gaussian parameters of the first reconstruction model, the first pose of the vehicle under the original trajectory, and the second pose of the vehicle under the new trajectory, a second reconstruction model is trained, which is used to generate the rendered image under the new trajectory.

[0011] In some optional implementations, training a second reconstruction model using the repaired image under the new trajectory, the Gaussian parameters of the first reconstruction model, the first pose of the vehicle under the original trajectory, and the second pose of the vehicle under the new trajectory includes:

[0012] The Gaussian parameters and time steps of the first reconstruction model are respectively encoded to obtain Gaussian parameter position codes and time step position codes. The time step is used to indicate the frame number of the acquired image under the original trajectory in the image dataset.

[0013] Using the first multilayer perceptron, a first feature is obtained based on the Gaussian parameter position encoding and the time step position encoding;

[0014] Using a second multilayer perceptron, a second feature is obtained based on the first pose of the vehicle under the original trajectory and the second pose of the vehicle under the new trajectory;

[0015] Based on the first feature, the second feature, and the Gaussian parameters of the first reconstruction model, a first initial reconstruction model is obtained;

[0016] The first initial reconstruction model is trained using the repaired image under the new trajectory to obtain the second reconstruction model.

[0017] In some optional implementations, training the first reconstruction model using the image dataset and the first point cloud dataset includes:

[0018] The point clouds from the first point cloud dataset are fused to obtain fused point cloud data.

[0019] From the fused point cloud data, road surface point cloud data, non-road static background point cloud data, and dynamic object point cloud data are respectively segmented;

[0020] The parameters of the three-dimensional Gaussian splash model are initialized based on the road surface point cloud data to obtain the initial road surface model.

[0021] The parameters of the three-dimensional Gaussian splash model are initialized based on the non-road static background point cloud data to obtain the initial non-road background model.

[0022] The parameters of the three-dimensional Gaussian splash model are initialized based on the dynamic object point cloud data to obtain the initial dynamic object model.

[0023] The initial road surface model, the initial non-road surface background model, and the initial dynamic object model are combined to obtain a second initial reconstruction model.

[0024] The first reconstruction model is obtained by training the second initial reconstruction model using the acquired images under the original trajectory of each frame in the image dataset.

[0025] In some optional implementations, after obtaining the second initial reconstruction model, the process further includes:

[0026] Using the acquired images under the original trajectory of each frame in the image dataset, the second initial reconstruction model is under-trained to obtain an under-trained first reconstruction model;

[0027] The video restoration model is obtained by training an initial diffusion model using a poorly trained first reconstruction model and the acquired images under the original trajectory of each frame in the image dataset.

[0028] In some optional implementations, the step of training the initial diffusion model using an insufficiently trained first reconstruction model and images under the original trajectories of each frame in the image dataset to obtain the video restoration model includes:

[0029] The first reconstruction model, which is not sufficiently trained, is used to render the images under the original trajectory of each frame, thus obtaining the corresponding rendered images under the original trajectory of each frame.

[0030] A video restoration dataset is constructed using the rendered image under the original trajectory and the acquired image under the original trajectory;

[0031] The initial diffusion model is trained using the video restoration dataset to obtain the video restoration model.

[0032] In some optional implementations, the road surface point cloud data includes multiple sub-point cloud data;

[0033] The initialization process for the parameters of the 3D Gaussian splash model based on the road surface point cloud data to obtain the initial road surface model includes:

[0034] Obtain the center position of each sub-point cloud data to obtain the center position of each Gaussian sphere;

[0035] The center position of each Gaussian sphere and the number of Gaussian spheres are determined as the prior information of the three-dimensional structure.

[0036] The three-dimensional structural prior information is integrated into the three-dimensional Gaussian splash model to obtain the initial road surface model. The three-dimensional structural prior information remains unchanged during the training operation of the second initial reconstruction model.

[0037] In some alternative implementations, the initial dynamic object model is a model based on the local coordinate system of the dynamic object;

[0038] The process of combining the initial road surface model, the initial non-road surface background model, and the initial dynamic object model to obtain the second initial reconstruction model includes:

[0039] Based on the pose information of each dynamic object in the world coordinate system, obtain the rotation matrix and translation vector that transform each dynamic object from the local coordinate system to the world coordinate system;

[0040] Based on the rotation matrix and the translation vector, the initial dynamic object model is transformed to obtain a transformed dynamic object model, which is a model based on the world coordinate system.

[0041] The initial road surface model, the initial non-road surface background model, and the transformed dynamic object model are combined to obtain the second initial reconstruction model.

[0042] In some optional implementations, before rendering the image under the new trajectory using the first reconstruction model to obtain the rendered image under the new trajectory, the method further includes:

[0043] A new trajectory is sampled based on a preset sampling period, and the offset of the new trajectory sampled in each sampling period relative to the original trajectory increases sequentially.

[0044] A second aspect of this disclosure provides a training apparatus for a scene reconstruction model, comprising:

[0045] The first acquisition module is used to acquire an image dataset collected by the vehicle under the original trajectory and a first point cloud dataset that is synchronized with the image dataset in time. The first point cloud dataset is acquired by LiDAR.

[0046] The first training module is used to train a first reconstruction model using the image dataset and the first point cloud dataset.

[0047] The first rendering module is used to render the image under the new trajectory using the first reconstruction model, and obtain the rendered image under the new trajectory.

[0048] The repair module is used to repair the rendered image under the new trajectory using a pre-trained video repair model to obtain the repaired image under the new trajectory.

[0049] The second training module is used to train a second reconstruction model using the repaired image under the new trajectory, the Gaussian parameters of the first reconstruction model, the first pose of the vehicle under the original trajectory, and the second pose of the vehicle under the new trajectory. The second reconstruction model is used to generate a rendered image under the new trajectory.

[0050] In some optional implementations, the second training module includes:

[0051] The encoding submodule is used to perform position encoding on the Gaussian parameters and time steps of the first reconstruction model respectively to obtain Gaussian parameter position encoding and time step position encoding. The time step is used to indicate the frame number of the acquired image under the original trajectory in the image dataset.

[0052] The first processing submodule is used to obtain a first feature based on the Gaussian parameter position encoding and time step position encoding using the first multilayer perceptron;

[0053] The second processing submodule is used to obtain the second feature based on the first pose of the vehicle under the original trajectory and the second pose of the vehicle under the new trajectory using the second multilayer perceptron.

[0054] The parameter update submodule is used to obtain a first initial reconstruction model based on the first feature, the second feature, and the Gaussian parameters of the first reconstruction model;

[0055] The first training submodule is used to train the first initial reconstruction model using the repaired image under the new trajectory to obtain the second reconstruction model.

[0056] In some optional implementations, the first training module includes:

[0057] The point cloud fusion submodule is used to fuse multiple frames of point clouds in the first point cloud dataset to obtain fused point cloud data.

[0058] The point cloud segmentation submodule is used to segment road surface point cloud data, non-road static background point cloud data, and dynamic object point cloud data from the fused point cloud data.

[0059] The first initialization submodule is used to initialize the parameters of the three-dimensional Gaussian splash model based on the road surface point cloud data to obtain the initial road surface model.

[0060] The second initialization submodule is used to initialize the parameters of the three-dimensional Gaussian splash model based on the non-road static background point cloud data to obtain the initial non-road background model.

[0061] The third initialization submodule is used to initialize the parameters of the three-dimensional Gaussian splash model based on the dynamic object point cloud data to obtain the initial dynamic object model.

[0062] The combination submodule is used to combine the initial road surface model, the initial non-road surface background model and the initial dynamic object model to obtain a second initial reconstruction model.

[0063] The second training submodule is used to train the second initial reconstruction model using the acquired images under the original trajectory of each frame in the image dataset, so as to obtain the first reconstruction model.

[0064] In some alternative embodiments, the apparatus further includes:

[0065] The third training module is used to use the acquired images under the original trajectory of each frame in the image dataset to perform insufficient training on the second initial reconstruction model, so as to obtain an insufficiently trained first reconstruction model.

[0066] The fourth training module is used to train the initial diffusion model using the insufficiently trained first reconstruction model and the acquired images under the original trajectory of each frame in the image dataset, so as to obtain the video restoration model.

[0067] In some optional implementations, the fourth training module includes:

[0068] The first rendering submodule is used to render the images under the original trajectory of each frame using the insufficiently trained first reconstruction model, and obtain the corresponding rendered images under the original trajectory of each frame.

[0069] The dataset construction submodule is used to construct a video restoration dataset using the rendered image under the original trajectory and the acquired image under the original trajectory;

[0070] The third training submodule is used to train the initial diffusion model using the video restoration dataset to obtain the video restoration model.

[0071] In some optional implementations, the road surface point cloud data includes multiple sub-point cloud data;

[0072] The first initialization submodule includes:

[0073] The first acquisition unit is used to acquire the center position of each sub-point cloud data and obtain the center position of each Gaussian sphere.

[0074] The prior information determination unit is used to determine the center position of each Gaussian sphere and the number of Gaussian spheres as the prior information of the three-dimensional structure.

[0075] An integration unit is used to integrate the three-dimensional structural prior information into the three-dimensional Gaussian splash model to obtain the initial road surface model. The three-dimensional structural prior information remains unchanged during the training operation of the second initial reconstruction model.

[0076] In some alternative implementations, the initial dynamic object model is a model based on the local coordinate system of the dynamic object;

[0077] The combined submodule includes:

[0078] The second acquisition unit is used to acquire the rotation matrix and translation vector that transforms each dynamic object from the local coordinate system to the world coordinate system based on the pose information of each dynamic object in the world coordinate system.

[0079] A transformation unit is used to transform the initial dynamic object model based on the rotation matrix and the translation vector to obtain a transformed dynamic object model, wherein the transformed dynamic object model is a model based on the world coordinate system.

[0080] The combination unit is used to combine the initial road surface model, the initial non-road surface background model, and the transformed dynamic object model to obtain the second initial reconstruction model.

[0081] In some alternative embodiments, the apparatus further includes:

[0082] The sampling module is used to sample new trajectories based on a preset sampling period, and the offset of the new trajectory sampled in each sampling period relative to the original trajectory increases sequentially.

[0083] A third aspect of this disclosure provides a computer-readable storage medium storing computer program instructions that, when executed, implement the training method for the scene reconstruction model proposed in the first aspect of this disclosure.

[0084] A fourth aspect of this disclosure provides an electronic device, the electronic device comprising:

[0085] Memory, used to store computer program products;

[0086] A processor is configured to execute a computer program product stored in the memory, and when the computer program product is executed, to implement the training method for the scene reconstruction model proposed in the first aspect of this disclosure.

[0087] A fifth aspect of this disclosure provides a computer program product including computer program instructions that, when executed by a processor, implement the training method for the scene reconstruction model proposed in the first aspect embodiment above.

[0088] Based on the embodiments of this disclosure, for the autonomous driving scene reconstruction task, an image dataset collected by the vehicle under the original trajectory and a first point cloud dataset synchronized with the image dataset in time can be obtained. Then, a first reconstruction model is trained using the image dataset and the first point cloud dataset. The first reconstruction model is then used to render the image under the new trajectory to obtain a rendered image under the new trajectory. A pre-trained video restoration model is then used to restore the rendered image under the new trajectory to obtain a restored image under the new trajectory. Finally, a second reconstruction model is trained using the restored image under the new trajectory, the Gaussian parameters of the first reconstruction model, the first pose of the vehicle under the original trajectory, and the second pose of the vehicle under the new trajectory. The second reconstruction model is used to generate the rendered image under the new trajectory. Therefore, by using the restored image under the new trajectory obtained by the video restoration model as training samples to further train the scene reconstruction model, it helps to bridge the domain gap between the restored image under the new trajectory and the collected image under the original trajectory, improving the rendering quality and robustness of the reconstruction model. Attached Figure Description

[0089] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0090] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein:

[0091] Figure 1 The image shows the scene reconstruction effect achieved using existing methods and the scene reconstruction model disclosed herein.

[0092] Figure 2 A flowchart illustrating one embodiment of the training method for the scene reconstruction model disclosed herein;

[0093] Figure 3 This is a schematic diagram of the training process of a training method for a scene reconstruction model provided in an exemplary embodiment of this disclosure;

[0094] Figure 4 This is a flowchart illustrating step 205 provided in an exemplary embodiment of this disclosure;

[0095] Figure 5 This is a flowchart illustrating step 202 provided in an exemplary embodiment of this disclosure;

[0096] Figure 6 This is a schematic diagram of the training process of a video reconstruction model provided in an exemplary embodiment of this disclosure;

[0097] Figure 7 This is a schematic diagram of the structure of a training device for a scene reconstruction model provided in an exemplary embodiment of this disclosure;

[0098] Figure 8 This is a schematic diagram of the structure of a training device for a scene reconstruction model provided in another exemplary embodiment of this disclosure;

[0099] Figure 9 This is a schematic diagram of the structure of a training device for a scene reconstruction model provided in another exemplary embodiment of this disclosure;

[0100] Figure 10 This is a structural diagram of an electronic device provided in an exemplary embodiment of this disclosure. Detailed Implementation

[0101] To explain this disclosure, exemplary embodiments of the disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the disclosure, and not all of them. It should be understood that the disclosure is not limited to exemplary embodiments.

[0102] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0103] This disclosure outlines

[0104] In realizing this disclosure, the inventors discovered through research that in the field of autonomous driving scenarios, autonomous driving data acquisition generally refers to data collected during a segment of the vehicle's driving process, and the trajectory of this driving process is called the original trajectory. If the vehicle only travels in one lane during this driving process, the collected data may only contain data from one lane, resulting in the autonomous driving dataset typically containing only the original trajectory data and lacking rich multi-view images, leading to sparse viewpoints in the autonomous driving data. If it is desired to render videos from new perspectives, such as rendering new trajectories like gradual lane changes or lane shifts, the generated videos under the new trajectories will exhibit obvious ghosting phenomena, such as motion blur, fragmentation, and blurring.

[0105] like Figure 1 The image on the left shows the rendered image under the original trajectory (indicated by marker 11) and the rendered image under the new trajectory with lane shift (3 meters) generated using Street Gaussians (a method for modeling dynamic city street scenes) (indicated by marker 12). According to the rendering effect indicated by the square box in marker 12, it can be seen that the image quality of the rendered image under the new trajectory generated using Street Gaussians is poor.

[0106] like Figure 1The middle image shows the rendered image under the original trajectory (indicated by marker 13) and the rendered image under the new trajectory with lane shift (3 meters) generated using ReconDreamer (a scene reconstruction model built through online restoration technology) (indicated by marker 14). It can be seen that the rendered image under the new trajectory generated by online restoration technology and reconstruction model is of high quality overall. However, according to the rendering effect indicated by the square box in marker 14, it can be seen that the rendered image under the new trajectory generated by ReconDreamer has quality degradation problems in some local locations.

[0107] Based on the technical problems existing in the prior art, this application provides a training method for a scene reconstruction model. First, it acquires an image dataset collected by the vehicle under its original trajectory and a first point cloud dataset synchronized with the image dataset in time. Then, it uses the image dataset and the first point cloud dataset to train a first reconstruction model. The first reconstruction model is then used to render an image under a new trajectory, resulting in a rendered image under the new trajectory. Next, a pre-trained video restoration model is used to restore the rendered image under the new trajectory, resulting in a restored image under the new trajectory. Finally, using the restored image under the new trajectory, the Gaussian parameters of the first reconstruction model, the first pose of the vehicle under its original trajectory, and the second pose of the vehicle under the new trajectory, a second reconstruction model is trained. This second reconstruction model is used to generate the rendered image under the new trajectory. Therefore, by using the restored image under the new trajectory obtained from the video restoration model as training samples to further train the scene reconstruction model, it helps to bridge the domain gap between the restored image under the new trajectory and the acquired image under the original trajectory, improving the rendering quality and robustness of the reconstruction model.

[0108] See Figure 1 The right-hand image shows the rendered image under the original trajectory (indicated by marker 15) and the rendered image under the new trajectory with lane shift (3 meters) generated by the scene reconstruction model generated using the technical solution of this disclosure (indicated by marker 16). It can be seen that the rendered images under the original trajectory and the new trajectory generated by the scene reconstruction model trained based on the technical solution of this disclosure are of high quality.

[0109] Furthermore, the rendering quality of the image under the original trajectory can be measured using a deep learning-based image similarity metric (Learned Perceptual Image Patch Similarity, or LPIPS). This metric measures the similarity between the rendered image and the captured image under the original trajectory; a smaller value indicates greater similarity and better rendering quality. The rendering quality of the image under the new trajectory can be measured using the FID (Fréchet Inception Distance) metric. This metric measures the similarity of image features between the rendered image under the new trajectory and the captured image under the original trajectory; a smaller value indicates greater similarity and better rendering quality. Figure 1 As can be seen from the rightmost indicator, the scene reconstruction model trained based on the method provided in this disclosure has the best rendering effect (indicators indicated by markers 17 and 18).

[0110] Exemplary methods

[0111] Figure 2 This is a flowchart illustrating a training method for a scene reconstruction model provided in an exemplary embodiment of this disclosure. This embodiment can be applied to electronic devices, such as... Figure 2 As shown, it includes the following steps:

[0112] Step 201: Obtain the image dataset collected by the vehicle under the original trajectory and the first point cloud dataset synchronized with the image dataset in time. The first point cloud dataset is obtained by LiDAR.

[0113] In this embodiment, the image dataset collected by the vehicle under the original trajectory can be multiple frames of images of the external environment captured by image acquisition devices such as cameras on the vehicle during the vehicle's driving process. It can be obtained by extracting frames from video data collected during a driving process, or by using each frame of the video data as an image in the aforementioned image dataset. The multiple frames in the image dataset represent driving data in a realistic driving environment, also referred to as ground truth (GT) video under the original trajectory. The scene reconstruction model training tool, which is the main body executing the training method of the first reconstruction model of this disclosure, first acquires the GT video of the original trajectory, and then uses this video to train the first reconstruction model in subsequent steps.

[0114] The original trajectory is the collected trajectory, for example, obtained by positioning through the vehicle's Global Positioning System (GPS).

[0115] In this embodiment, the first point cloud dataset synchronized with the image dataset in time refers to the three-dimensional point cloud acquired by the LiDAR on the vehicle. The three-dimensional point cloud can be a set of three-dimensional points with color information, obtained by storing the set of three-dimensional points with color information in a general three-dimensional point cloud format.

[0116] After acquiring 3D point cloud data, preprocessing operations such as point cloud thinning, homogenization, and noise reduction can be performed on the point cloud collected by LiDAR. These preprocessing operations can reduce the amount of 3D point cloud data and improve task processing efficiency.

[0117] In practice, the camera and the LiDAR are synchronized in time and calibrated to achieve a geometric mapping between them.

[0118] Step 202: Use the image dataset and the first point cloud dataset to train the first reconstruction model.

[0119] In this embodiment, the training process of the first reconstruction model can be divided into initializing the model parameters using the first point cloud dataset and training the model using the image dataset to adjust the model parameters.

[0120] In practice, the driving scene representation can first be decoupled into an initial road surface model, an initial non-road surface background model, and an initial dynamic object model; then, the initial road surface model, the initial non-road surface background model, and the initial dynamic object model can be combined into a second initial reconstruction model; then, the second initial reconstruction model can be trained using the acquired images under the original trajectory of each frame in the image dataset to obtain the first reconstruction model.

[0121] For details on training the first reconstruction model, please refer to [link / reference]. Figure 5 The embodiments shown are not described in detail here.

[0122] Step 203: Render the image under the new trajectory using the first reconstruction model to obtain the rendered image under the new trajectory.

[0123] In this embodiment of the disclosure, a new trajectory may be sampled first. For example, based on the original trajectory, the viewpoint is switched to one that shifts the lane 1.5 meters to the left or 2 meters to the right, and a new trajectory is sampled from that viewpoint.

[0124] In some implementations, before rendering the image under the new trajectory using the first reconstruction model, the new trajectory can be sampled based on a preset sampling period, and the offset of the new trajectory sampled in each sampling period relative to the original trajectory increases sequentially.

[0125] In practical implementation, a progressive strategy can be adopted for sampling new trajectories. This means that during model training, the offset of the new trajectory relative to the original trajectory gradually increases. For example, first, based on the original trajectory, switch to a viewpoint that shifts the lane 1.5 meters to the left, sample a new trajectory from this viewpoint, and then render the image under the new trajectory. Next, execute step 204 to use a video restoration model to restore the rendered image under the new trajectory, obtaining a restored image under the new trajectory. Then, based on a preset sampling period, switch to a viewpoint that shifts the lane 3.0 meters to the left from the original trajectory, and sample a new trajectory with an even larger offset from this viewpoint. In this sampling strategy, the offset of the new trajectory sampled in each sampling period relative to the original trajectory increases sequentially. Training first on trajectories with smaller offsets and then gradually switching to trajectories with larger offsets allows the model to progressively increase the learning difficulty, achieving better training results.

[0126] Understandably, since the first reconstruction model was trained using the original trajectory images from each frame of the image dataset, it was not sufficiently trained for the new trajectory. Therefore, the quality of the rendered image under the new trajectory generated by the first reconstruction model is poor, and there may be problems such as ghosting and blurring. Therefore, step 204 can be used to repair the rendered image under the new trajectory.

[0127] Step 204: Use the pre-trained video restoration model to restore the rendered image under the new trajectory to obtain the restored image under the new trajectory.

[0128] Among them, the pre-trained video restoration model is used to restore the rendered image under the new trajectory, so that the restored image under the new trajectory can have a better overall rendering effect.

[0129] Specifically, autonomous driving datasets typically only contain image datasets of the original trajectory, lacking video restoration datasets for training video restoration models. Therefore, a video restoration dataset can be constructed by using an under-trained reconstruction model to render images under the original trajectory. Specifically, an under-trained scene reconstruction model renders images under the original trajectory, and this image is combined with frames acquired from the original trajectory to construct a video restoration dataset. This dataset is then used to train a video restoration model.

[0130] For details on how to train the video restoration model, please refer to [link / reference]. Figure 6 The embodiments shown are not described in detail here.

[0131] Step 205: Using the repaired image under the new trajectory, the Gaussian parameters of the first reconstruction model, the first pose of the vehicle under the original trajectory, and the second pose of the vehicle under the new trajectory, a second reconstruction model is trained. The second reconstruction model is used to generate the rendered image under the new trajectory.

[0132] In this model, the Gaussian parameters of the first reconstruction model are used to define a 3D Gaussian ellipsoid, representing the scene. The Gaussian parameters specifically include position, covariance, color, and transparency. The first pose of the vehicle on the original trajectory indicates the position and orientation information collected by the vehicle's positioning module. The second pose on the new trajectory indicates a new trajectory sampled from a viewpoint that shifts the lane 1.5 meters to the left or 2 meters to the right, based on the original trajectory. The third pose on the new trajectory indicates the vehicle's position and orientation information within the new trajectory.

[0133] For details on how to train the second reconstruction model, please refer to [link / reference]. Figure 4 The embodiments shown are not described in detail here.

[0134] Through steps 201-205 above, for the autonomous driving scene reconstruction task, an image dataset collected by the vehicle under the original trajectory and a first point cloud dataset synchronized with the image dataset in time can be obtained. Then, a first reconstruction model is trained using the image dataset and the first point cloud dataset. The first reconstruction model is then used to render the image under the new trajectory, resulting in a rendered image under the new trajectory. A pre-trained video restoration model is then used to restore the rendered image under the new trajectory, resulting in a restored image under the new trajectory. Finally, a second reconstruction model is trained using the restored image under the new trajectory, the Gaussian parameters of the first reconstruction model, the first pose of the vehicle under the original trajectory, and the second pose of the vehicle under the new trajectory. The second reconstruction model is used to generate the rendered image under the new trajectory. Therefore, by using the restored image under the new trajectory obtained by the video restoration model as training samples to further train the scene reconstruction model, it helps to bridge the domain gap between the restored image under the new trajectory and the collected image under the original trajectory, improving the rendering quality and robustness of the reconstruction model.

[0135] In some alternative implementations, the training process for the first and second reconstruction models can be found in [reference needed]. Figure 3 .

[0136] Figure 3First, the Dynamic Driving Scene Representation is decoupled into a Ground Model, a Background Model, and an Object Model. These are then combined to obtain a second initial reconstruction model (Gaussians). The camera's intrinsic and extrinsic parameters, obtained from capturing images of each original trajectory, are input into the second initial reconstruction model (Gaussians) to render the original trajectory images. Multiple frames of these original trajectory rendering images are stitched together to obtain the original trajectory rendered video. Finally, the loss function of the scene reconstruction model is used to constrain the model. Specifically, based on the output of the second initial reconstruction model and the captured images of the original trajectory, the loss value of the second initial reconstruction model (Gaussians) is calculated. The model parameters of the second initial reconstruction model are adjusted based on the loss value to obtain the first reconstruction model.

[0137] The second reconstruction model (Novel Trajectory Deformable Network) consists of a Pose Feature Module and a Temporal Field Module. The Temporal Field Module takes as input the Gaussian parameter position codes corresponding to the Gaussian parameters of the first reconstruction model and the temporal step position codes corresponding to the time step (which indicates the frame number of the acquired image under the original trajectory in the image dataset). The Temporal Field Module implicitly mixes the Gaussian parameter position codes and the temporal step position codes to obtain the first feature. The Pose Feature Module takes as input the pose change (Delta Pose) determined based on the vehicle's first pose under the original trajectory and its second pose under the new trajectory. The Pose Feature Module performs feature extraction and transformation on this position change to obtain the second feature. The Gaussian parameter change is obtained by element-wise summing the first and second features through the output linear layer. Adding the Gaussian parameters of the first reconstruction model element-wise to the Gaussian parameter change yields the first initial reconstruction model (Updated Gaussians). By inputting the corresponding camera intrinsic and extrinsic parameters for each new trajectory into the first initial reconstruction model (UpdatedGaussians), the rendered image under the new trajectory can be obtained using the first initial reconstruction model. Stitching together multiple frames of the rendered image under the original trajectory yields the rendered video under the new trajectory (Novel Trajectory Rendered Video). Then, the loss function of the scene reconstruction model is used to constrain the model. Specifically, based on the output of the first initial reconstruction model and the restored image under the new trajectory (image extracted from the Novel Trajectory Restored Video), the loss value of the first initial reconstruction model (UpdatedGaussians) is calculated. The model parameters of the first initial reconstruction model are adjusted according to the loss value to obtain the second reconstruction model.

[0138] The restored images under the new trajectory (images extracted from the Novel Trajectory Restored Video) are generated by the video restoration model (Drive Restorer) based on the degraded video (Novel Trajectory Degraded Video) under the new trajectory. Each frame in the Novel Trajectory Degraded Video is a rendered image under the new trajectory obtained using the first reconstruction model.

[0139] based on Figure 3As illustrated in the diagram, the first reconstruction model is obtained by first decoupling the dynamic driving scene representation into a road surface model, a non-road surface background model, and a dynamic object model, and then initializing the model parameters before training. By modeling the road surface separately, the rendering effect of the road surface under different viewpoints can be significantly improved, enhancing its spatial consistency, and thus improving the rendering effect and quality of the image under the original trajectory. The training of the second reconstruction model involves updating the Gaussian parameters based on the first reconstruction model. This allows the trained second reconstruction model to bridge the domain gap between the generated data and the original data, improving the rendering effect and quality of the image under the new trajectory.

[0140] Figure 4 This is a flowchart illustrating step 205 provided in an exemplary embodiment of this disclosure. Figure 2 Based on the illustrated embodiment, the specific implementation of step 205, which trains the second reconstruction model, includes the following steps:

[0141] Step 251: The Gaussian parameters and time steps of the first reconstruction model are respectively encoded to obtain the Gaussian parameter position code and the time step position code. The time step is used to indicate the frame number of the acquired image under the original trajectory in the image dataset.

[0142] Specifically, by performing position encoding on the Gaussian parameters (g) of the first reconstruction model, the Gaussian parameter position encoding can be obtained; by performing position encoding on the time step, the time step position encoding can be obtained.

[0143] Since rendering images using the reconstruction model is performed on a per-frame basis, in order to generate video based on the rendered images, the frame number of each frame can be position-encoded during rendering. Thus, after generating each frame of rendered images, the frame number of different rendered images can be distinguished according to the time step position encoding parameters.

[0144] Step 252: Using the first multilayer perceptron, the first feature is obtained based on Gaussian parameter position encoding and time step position encoding.

[0145] Among them, the first multilayer perceptron corresponds to Figure 3 The temporal field module implicitly mixes the Gaussian parameter position encoding and the time step position encoding to output the first feature. Specifically, this mixing method can perform various processing on the Gaussian parameter position encoding and the time step position encoding, such as forward propagation, loss calculation, backpropagation, and weight update.

[0146] Step 253: Using the second multilayer perceptron, based on the first pose of the vehicle under the original trajectory and the second pose of the vehicle under the new trajectory, the second feature is obtained.

[0147] In this embodiment of the disclosure, the pose change can be determined first based on the first pose and the second pose, and can be calculated by formula (1).

[0148]

[0149] In equation (1), L is a hyperparameter used to normalize the changing posture. This represents the position and orientation at each time step under the new trajectory. This represents the position and orientation at each time step of the original trajectory.

[0150] The second multilayer perceptron corresponds to Figure 3 The pose feature module in the middle is used to analyze Δp t Various processes, such as forward propagation, loss calculation, backpropagation, and weight update, are performed to obtain the second feature.

[0151] Step 254: Based on the first feature, the second feature, and the Gaussian parameters of the first reconstruction model, the first initial reconstruction model is obtained.

[0152] In this embodiment, the first and second features can be added together by the output linear layer to obtain the change in the Gaussian parameters. Specifically, see equation (2).

[0153]

[0154] In equation (1), It is the output of the pose feature module (second feature), F θ (PE(g),PE(t)) is the output of the time field module (first feature), F out It is an output linear layer.

[0155] After obtaining the change in the Gaussian parameters through equation (2), the updated Gaussian parameters g′=g+Δg can be obtained based on the change in the Gaussian parameters Δg and the Gaussian parameters (g) of the first reconstruction model.

[0156] Step 255: Using the repaired image under the new trajectory, train the first initial reconstruction model to obtain the second reconstruction model.

[0157] In some implementations, after obtaining the updated Gaussian parameters, the corresponding camera intrinsic and extrinsic parameters of each new trajectory can be input into the first initial reconstruction model to render the image under the new trajectory using the first initial reconstruction model. Then, the loss function of the scene reconstruction model is used to constrain the model. Specifically, based on the output of the first initial reconstruction model and the repaired image under the new trajectory, the loss value of the first initial reconstruction model is calculated, and the model parameters of the first initial reconstruction model are adjusted according to the loss value to obtain the second reconstruction model.

[0158] Through steps 251-255 above, this technical solution discloses the implementation method of training to obtain the second reconstruction model. By updating the Gaussian parameters of the second reconstruction model, it helps to bridge the domain gap between the rendered image under the new trajectory generated by the second reconstruction model and the acquired image under the original trajectory, thereby improving the rendering quality of the image under the new trajectory.

[0159] Figure 5 This is a flowchart illustrating step 202 provided in an exemplary embodiment of this disclosure; Figure 2 Based on the illustrated embodiment, the specific implementation of step 202, which trains the first reconstruction model, includes the following steps:

[0160] Step 221: Fuse the multiple frames of point clouds in the first point cloud dataset to obtain the fused point cloud data.

[0161] In this embodiment of the disclosure, point cloud data of different frames can first be registered to align them to the world coordinate system. Then, based on spatial positional relationships, motion models of objects, etc., the points corresponding to the same object in different frames are determined, thereby establishing the association relationship of point cloud data of different frames. Then, based on the association relationship of point cloud data of different frames, the point cloud data of different frames are fused.

[0162] Step 222: Separate the road surface point cloud data, non-road surface static background point cloud data, and dynamic object point cloud data from the fused point cloud data.

[0163] In this embodiment of the disclosure, a clustering algorithm can be used to segment road surface point cloud data, non-road surface static background point cloud data, and dynamic object point cloud data from the fused point cloud data.

[0164] Specifically, points in point cloud data can be clustered based on their location and characteristics. More specifically, features can be extracted or transformed from each point's spatial or local space to obtain various attributes, such as normal vector, density, distance, elevation, and intensity, thereby enabling the segmentation of point cloud data with different attributes.

[0165] Furthermore, after segmenting the road surface point cloud data, the non-road surface static background point cloud data, and the dynamic object point cloud data, the initial road surface model, the initial non-road surface background model, and the initial dynamic object model can be generated through steps 223, 224, and 225.

[0166] Step 223: Initialize the parameters of the 3D Gaussian splash model based on the road surface point cloud data to obtain the initial road surface model.

[0167] The implementation of step 223 specifically includes the following steps:

[0168] Step 1: Obtain the center position of each sub-point cloud data to obtain the center position of each Gaussian sphere;

[0169] Step 2: Determine the center position of each Gaussian sphere and the number of Gaussian spheres as prior information for the three-dimensional structure;

[0170] Step 3: Integrate the prior information of the three-dimensional structure into the three-dimensional Gaussian splash model to obtain the initial road surface model. The prior information of the three-dimensional structure remains unchanged during the training of the second initial reconstruction model.

[0171] The initial road surface model is represented by a set of Gaussian distributions in the world coordinate system. Each Gaussian distribution consists of Gaussian parameters (center position x, opacity γ, covariance matrix ∑, and RGB color parameters), where the RGB color parameters are controlled by spherical harmonic functions.

[0172] In this embodiment of the disclosure, to ensure stability, each covariance matrix is ​​decomposed into ∑=RSS T R T , where the scaling matrix S and the rotation matrix R are learnable parameters, represented by the scaling factor and the quaternion, respectively.

[0173] Specifically, the center position of each sub-point cloud data can be determined as the center position of each Gaussian sphere, and the center position of each Gaussian sphere and the number of Gaussian spheres can be determined as the three-dimensional structure prior information. This three-dimensional structure prior information is integrated into the three-dimensional Gaussian splash model to obtain the initial road surface model.

[0174] It should be noted that, since the point cloud data obtained by fusing multi-frame point cloud data contains abundant road surface point cloud data, the technical solution disclosed in this paper directly fixes the number and location parameters (center position) of Gaussian distributions in the road surface model. In the subsequent process of training the second initial reconstruction model with the acquired images under the original trajectory to obtain the first reconstruction model, the prior information of the three-dimensional structure remains unchanged, and no modification is made to the number and center position of the Gaussian spheres. This significantly reduces the search space of the second initial reconstruction model and enhances its generalization ability.

[0175] Step 224: Initialize the parameters of the 3D Gaussian splash model based on the non-road static background point cloud data to obtain the initial non-road background model.

[0176] In some implementations, the parameters of the three-dimensional Gaussian splash model can be initialized in the world coordinate system using non-road static background point cloud data, that is, the parameters such as the number and center position of Gaussian spheres are initialized to obtain an initial non-road background model.

[0177] It should be noted that in the subsequent process of training the second initial reconstruction model using the acquired images under the original trajectory to obtain the first reconstruction model, the Gaussian parameters of the initial non-road background model can be optimized.

[0178] Step 225: Initialize the parameters of the 3D Gaussian splash model based on the dynamic object point cloud data to obtain the initial dynamic object model.

[0179] The number of dynamic object models is the same as the number of dynamic objects segmented from the point cloud data. For each dynamic object, a separate dynamic object model can be defined in the local coordinate system of each dynamic object.

[0180] In this embodiment, the point cloud data of each dynamic object can be used to initialize the parameters of the three-dimensional Gaussian splash model, that is, to initialize the parameters such as the number and center position of the Gaussian spheres, and obtain the initial dynamic object model in the local coordinate system of each dynamic object.

[0181] Step 226: Combine the initial road surface model, the initial non-road surface background model, and the initial dynamic object model to obtain the second initial reconstruction model.

[0182] The implementation of step 226 specifically includes the following steps:

[0183] Step 1: Based on the pose information of each dynamic object in the world coordinate system, obtain the rotation matrix and translation vector that transform each dynamic object from the local coordinate system to the world coordinate system.

[0184] Step 2: Based on the rotation matrix and translation vector, the initial dynamic object model is transformed to obtain the transformed dynamic object model, which is a model based on the world coordinate system.

[0185] Step 3: Combine the initial road surface model, the initial non-road surface background model, and the transformed dynamic object model to obtain the second initial reconstruction model.

[0186] Since each dynamic object model is based on a local coordinate system, it is necessary to convert each initial dynamic model to the world coordinate system before combining them.

[0187] Specifically, since the dataset annotations include the position and orientation of each dynamic object in the world coordinate system, the rotation matrix and translation vector for transforming the dynamic objects to the world coordinate system can be determined based on the local coordinate system of each dynamic object and its position and orientation in the world coordinate system. This allows for the transformation of the initial dynamic model to a Gaussian distribution position x in the world coordinate system. w and rotation quaternion qw For details, please refer to equations (3) and (4).

[0188] x w =R t x o +T t Equation (3)

[0189] q w =ROT(R t ,q o Equation (4)

[0190] In the above formula (3), x o T represents the location of the Gaussian distribution in the local coordinate system. t R represents the translation vector. t This represents the rotation matrix.

[0191] In equation (4) above, R t Let q represent the rotation matrix. o ROT(·) represents a Gaussian-distributed rotation quaternion in the local object coordinate system, and ROT(·) represents the operation of rotating the quaternion by a rotation matrix.

[0192] After converting the dynamic object model to the world coordinate system, the converted dynamic object model, the initial road surface model, and the initial non-road surface background model can be combined to obtain the second initial reconstruction model.

[0193] Step 227: Use the acquired images under the original trajectory of each frame in the image dataset to train the second initial reconstruction model to obtain the first reconstruction model.

[0194] In this embodiment, by inputting the camera's intrinsic and extrinsic parameters when acquiring images of each original trajectory into the second initial reconstruction model, a rendered image under the original trajectory can be obtained. Multiple frames of rendered images under the original trajectory are then stitched together to obtain a rendered video under the original trajectory. The loss function of the scene reconstruction model is then used to constrain the model. Specifically, based on the output of the second initial reconstruction model and the acquired images under the original trajectory, the loss value of the second initial reconstruction model is calculated. The model parameters of the second initial reconstruction model are adjusted according to the loss value to obtain the first reconstruction model.

[0195] Through steps 221-227 above, this disclosure provides a specific implementation method for obtaining a first reconstruction model by decoupling the driving scene representation into a road surface model, a non-road surface background model, and a dynamic object model, and then combining and training them. Since this disclosure implements separate modeling of the road surface model, it helps improve the rendering results of the road surface from different perspectives and enhances the spatial consistency of the road surface. Furthermore, during training, by fixing the number of Gaussian distributions and the center position of the road surface model, the search space of the reconstruction model can be significantly reduced, and the generalization ability of the model can be enhanced.

[0196] Figure 6 This is a schematic diagram of the training process of a video reconstruction model provided in an exemplary embodiment of this disclosure; as shown below. Figure 6 As shown, the steps for training the video restoration model include steps 601 and 602. The details of each step are explained below.

[0197] Step 601: Using the acquired images under the original trajectory of each frame in the image dataset, the second initial reconstruction model is insufficiently trained to obtain the first reconstruction model that is insufficiently trained.

[0198] In some implementations, insufficient training is used to indicate that the number of iterations is not high enough, the loss value between the acquired image and the rendered image under the original trajectory is still relatively large, and the model has not yet fully converged.

[0199] Step 602: Using the insufficiently trained first reconstruction model and the acquired images under the original trajectory of each frame in the image dataset, the initial diffusion model is trained to obtain the video restoration model.

[0200] In some implementations, an under-trained first reconstruction model can be used to render images under the original trajectory for each frame to obtain corresponding rendered images under the original trajectory for each frame; then, the rendered images under the original trajectory and the acquired images under the original trajectory are used to construct a video restoration dataset; finally, the initial diffusion model is trained using the video restoration dataset to obtain a video restoration model.

[0201] Specifically, during the process of training the second initial reconstruction model using the images acquired under the original trajectory to obtain the first reconstruction model, before the model is fully trained, the video under the original trajectory is directly rendered to obtain a video with degraded original trajectory, i.e., a poorly rendered image. This image, along with the images acquired under the original trajectory, forms a video restoration pair, completing the construction of the video restoration dataset. The images acquired under the original trajectory can be used as supervisory data during model training to monitor and guide the learning process of the video restoration model.

[0202] In this embodiment, an insufficiently trained scene reconstruction model is used to render the acquired images under the original trajectory. Based on the rendered images under the original trajectory, a video restoration dataset is constructed, providing a method for building a dataset to train a video restoration model. Using this method to construct the dataset solves the problem of insufficient training data in existing autonomous driving datasets. This dataset allows for sufficient training of the video restoration model, improving its restoration performance. Simultaneously, the well-trained video restoration model can assist the scene reconstruction model in image rendering, thereby improving rendering quality.

[0203] Exemplary device

[0204] Figure 7 This is a schematic diagram of the structure of a training device for a scene reconstruction model provided in an exemplary embodiment of this disclosure, as shown below. Figure 7 As shown, the device may include:

[0205] The first acquisition module 71 is used to acquire the image dataset collected by the vehicle under the original trajectory and the first point cloud dataset synchronized with the image dataset in time. The first point cloud dataset is acquired by LiDAR.

[0206] The first training module 72 is used to train the first reconstruction model using the image dataset and the first point cloud dataset.

[0207] The first rendering module 73 is used to render the image under the new trajectory using the first reconstruction model, and obtain the rendered image under the new trajectory.

[0208] Repair module 74 is used to repair the rendered image under the new trajectory using a pre-trained video repair model to obtain the repaired image under the new trajectory.

[0209] The second training module 75 is used to train a second reconstruction model using the repaired image under the new trajectory, the Gaussian parameters of the first reconstruction model, the first pose of the vehicle under the original trajectory, and the second pose of the vehicle under the new trajectory. The second reconstruction model is used to generate the rendered image under the new trajectory.

[0210] Figure 8 This is a schematic diagram of the structure of a training device for a scene reconstruction model provided in another exemplary embodiment of this disclosure, such as... Figure 8 As shown, in Figure 7 Based on the illustrated embodiment, in some implementations, the second training module 75 includes:

[0211] The encoding submodule 751 is used to perform position encoding on the Gaussian parameters and time steps of the first reconstruction model respectively, to obtain Gaussian parameter position encoding and time step position encoding. The time step is used to indicate the frame number of the acquired image under the original trajectory in the image dataset.

[0212] The first processing submodule 752 is used to obtain the first feature by utilizing the first multilayer perceptron based on Gaussian parameter position encoding and time step position encoding;

[0213] The second processing submodule 753 is used to obtain the second feature based on the first pose of the vehicle under the original trajectory and the second pose of the vehicle under the new trajectory using the second multilayer perceptron.

[0214] The parameter update submodule 754 is used to obtain the first initial reconstruction model based on the first feature, the second feature, and the Gaussian parameters of the first reconstruction model.

[0215] The first training submodule 755 is used to train the first initial reconstruction model using the repaired image under the new trajectory to obtain the second reconstruction model.

[0216] In some embodiments, the apparatus further includes:

[0217] The third training module 76 is used to use the acquired images under the original trajectory of each frame in the image dataset to perform insufficient training on the second initial reconstruction model, so as to obtain an insufficiently trained first reconstruction model.

[0218] The fourth training module 77 is used to train the initial diffusion model using the poorly trained first reconstruction model and the acquired images under the original trajectory of each frame in the image dataset, so as to obtain the video restoration model.

[0219] In some implementations, the fourth training module 77 includes:

[0220] The first rendering submodule 771 is used to render the images under the original trajectory of each frame using the insufficiently trained first reconstruction model, so as to obtain the corresponding rendered images under the original trajectory of each frame.

[0221] The dataset construction submodule 772 is used to construct a video restoration dataset using rendered images and acquired images under the original trajectory.

[0222] The third training submodule 773 is used to train the initial diffusion model using the video restoration dataset to obtain the video restoration model.

[0223] Figure 9 This is a schematic diagram of the structure of a training device for a scene reconstruction model provided in another exemplary embodiment of this disclosure, such as... Figure 9 As shown, in Figure 7 and / or Figure 8 Based on the illustrated embodiment, in some implementations, the first training module 72 includes:

[0224] The point cloud fusion submodule 721 is used to fuse multiple frames of point clouds in the first point cloud dataset to obtain fused point cloud data;

[0225] The point cloud segmentation submodule 722 is used to segment road surface point cloud data, non-road surface static background point cloud data, and dynamic object point cloud data from the fused point cloud data.

[0226] The first initialization submodule 723 is used to initialize the parameters of the three-dimensional Gaussian splash model based on the road surface point cloud data to obtain the initial road surface model.

[0227] The second initialization submodule 724 is used to initialize the parameters of the three-dimensional Gaussian splash model based on the non-road static background point cloud data to obtain the initial non-road background model.

[0228] The third initialization submodule 725 is used to initialize the parameters of the three-dimensional Gaussian splash model based on the dynamic object point cloud data to obtain the initial dynamic object model.

[0229] Combination submodule 726 is used to combine the initial road surface model, the initial non-road surface background model and the initial dynamic object model to obtain the second initial reconstruction model.

[0230] The second training submodule 727 is used to train the second initial reconstruction model using the acquired images under the original trajectory of each frame in the image dataset, so as to obtain the first reconstruction model.

[0231] In some implementations, the road surface point cloud data includes multiple sub-point cloud data;

[0232] The first initialization submodule 723 includes:

[0233] The first acquisition unit 7231 is used to acquire the center position of each sub-point cloud data and obtain the center position of each Gaussian sphere.

[0234] The prior information determination unit 7232 is used to determine the center position of each Gaussian sphere and the number of Gaussian spheres as the prior information of the three-dimensional structure.

[0235] Integration unit 7233 is used to integrate the three-dimensional structural prior information into the three-dimensional Gaussian splash model to obtain the initial road surface model. The three-dimensional structural prior information remains unchanged during the operation of training the second initial reconstruction model.

[0236] In some implementations, the initial dynamic object model is a model based on the local coordinate system of the dynamic object;

[0237] Combination submodule 726 includes:

[0238] The second acquisition unit 7261 is used to acquire the rotation matrix and translation vector that transforms each dynamic object from the local coordinate system to the world coordinate system based on the pose information of each dynamic object in the world coordinate system.

[0239] The transformation unit 7262 is used to transform the initial dynamic object model based on the rotation matrix and translation vector to obtain the transformed dynamic object model, which is a model based on the world coordinate system.

[0240] Combination unit 7263 is used to combine the initial road surface model, the initial non-road surface background model, and the transformed dynamic object model to obtain the second initial reconstruction model.

[0241] In some embodiments, the apparatus further includes:

[0242] The sampling module 78 is used to sample new trajectories based on a preset sampling period, and the offset of the new trajectory sampled in each sampling period relative to the original trajectory increases sequentially.

[0243] Exemplary electronic devices

[0244] Figure 10 A structural diagram of an electronic device provided in an embodiment of this disclosure includes at least one processor 101 and a memory 102.

[0245] The processor 101 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0246] The memory 102 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 101 may execute one or more computer program instructions to implement the training methods and / or other desired functions of the scene reconstruction models of the various embodiments of this disclosure described above.

[0247] In one example, the electronic device 10 may also include an input device 103 and an output device 104, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0248] The input device 103 may also include, for example, a keyboard, a mouse, etc.

[0249] The output device 104 can output various information to the outside, including, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0250] Of course, for the sake of simplicity, Figure 10 Only some of the components of the electronic device 10 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 10 may include any other suitable components depending on the specific application.

[0251] Exemplary computer program products and computer-readable storage media

[0252] In addition to the methods and apparatus described above, embodiments of this disclosure may also provide a computer program product, including computer program instructions, which, when executed by a processor, cause the processor to perform the steps in the training methods for scene reconstruction models of various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0253] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0254] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the training methods of the scene reconstruction models of the various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0255] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, but is not limited to, systems, apparatuses, or devices that are electrical, magnetic, optical, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0256] The basic principles of this disclosure have been described above with reference to specific embodiments. However, the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0257] Various modifications and variations can be made to this disclosure without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, this disclosure is also intended to include such modifications and variations.

Claims

1. A method for training a scene reconstruction model, comprising: The system acquires an image dataset collected by the vehicle along its original trajectory and a first point cloud dataset that is time-synchronized with the image dataset. The first point cloud dataset is acquired by a lidar. A first reconstruction model is trained using the image dataset and the first point cloud dataset; The first reconstruction model is used to render the image under the new trajectory to obtain the rendered image under the new trajectory; The rendered image under the new trajectory is repaired using a pre-trained video restoration model to obtain the restored image under the new trajectory; Using the repaired image under the new trajectory, the Gaussian parameters of the first reconstruction model, the first pose of the vehicle under the original trajectory, and the second pose of the vehicle under the new trajectory, a second reconstruction model is trained, and the second reconstruction model is used to generate the rendered image under the new trajectory. The second reconstruction model is trained using the repaired image under the new trajectory, the Gaussian parameters of the first reconstruction model, the first pose of the vehicle under the original trajectory, and the second pose of the vehicle under the new trajectory, including: The Gaussian parameters and time steps of the first reconstruction model are respectively encoded to obtain Gaussian parameter position codes and time step position codes. The time step is used to indicate the frame number of the acquired image under the original trajectory in the image dataset. Using a first multilayer perceptron, the first feature is obtained based on the Gaussian parameter position encoding and the time step position encoding; Using a second multilayer perceptron, a second feature is obtained based on the first pose of the vehicle under the original trajectory and the second pose of the vehicle under the new trajectory; The first feature and the second feature are added together to obtain the change in Gaussian parameters. Based on the change in Gaussian parameters and the Gaussian parameters of the first reconstruction model, a first initial reconstruction model is obtained. The first initial reconstruction model is trained using the repaired image under the new trajectory to obtain the second reconstruction model.

2. The method of claim 1, wherein, The process of training a first reconstruction model using the image dataset and the first point cloud dataset includes: The point clouds from the first point cloud dataset are fused to obtain fused point cloud data. From the fused point cloud data, road surface point cloud data, non-road static background point cloud data, and dynamic object point cloud data are respectively segmented; The parameters of the three-dimensional Gaussian splash model are initialized based on the road surface point cloud data to obtain the initial road surface model. The parameters of the three-dimensional Gaussian splash model are initialized based on the non-road static background point cloud data to obtain the initial non-road background model. The parameters of the three-dimensional Gaussian splash model are initialized based on the dynamic object point cloud data to obtain the initial dynamic object model. The initial road surface model, the initial non-road surface background model, and the initial dynamic object model are combined to obtain a second initial reconstruction model. The first reconstruction model is obtained by training the second initial reconstruction model using the acquired images under the original trajectory of each frame in the image dataset.

3. The method of claim 2, wherein, After obtaining the second initial reconstruction model, the process also includes: Using the acquired images under the original trajectory of each frame in the image dataset, the second initial reconstruction model is under-trained to obtain an under-trained first reconstruction model; The video restoration model is obtained by training an initial diffusion model using a poorly trained first reconstruction model and the acquired images under the original trajectory of each frame in the image dataset.

4. The method of claim 3, wherein, The process of training an initial diffusion model using an insufficiently trained first reconstruction model and images under the original trajectories of each frame in the image dataset to obtain the video restoration model includes: The first reconstruction model, which is not sufficiently trained, is used to render the images under the original trajectory of each frame, thus obtaining the corresponding rendered images under the original trajectory of each frame. A video restoration dataset is constructed using the rendered image under the original trajectory and the acquired image under the original trajectory; The initial diffusion model is trained using the video restoration dataset to obtain the video restoration model.

5. The method according to claim 2, characterized in that, The road surface point cloud data includes multiple sub-point cloud data; The initialization process for the parameters of the 3D Gaussian splash model based on the road surface point cloud data to obtain the initial road surface model includes: Obtain the center position of each sub-point cloud data to obtain the center position of each Gaussian sphere; The center position of each Gaussian sphere and the number of Gaussian spheres are determined as the prior information of the three-dimensional structure. The three-dimensional structural prior information is integrated into the three-dimensional Gaussian splash model to obtain the initial road surface model. The three-dimensional structural prior information remains unchanged during the training operation of the second initial reconstruction model.

6. The method according to claim 2, characterized in that, The initial dynamic object model is a model based on the local coordinate system of the dynamic object; The process of combining the initial road surface model, the initial non-road surface background model, and the initial dynamic object model to obtain the second initial reconstruction model includes: Based on the pose information of each dynamic object in the world coordinate system, obtain the rotation matrix and translation vector that transform each dynamic object from the local coordinate system to the world coordinate system; Based on the rotation matrix and the translation vector, the initial dynamic object model is transformed to obtain a transformed dynamic object model, which is a model based on the world coordinate system. The initial road surface model, the initial non-road surface background model, and the transformed dynamic object model are combined to obtain the second initial reconstruction model.

7. The method according to any one of claims 1-6, characterized in that, Before using the first reconstruction model to render the image under the new trajectory and obtain the rendered image under the new trajectory, the method further includes: A new trajectory is sampled based on a preset sampling period, and the offset of the new trajectory sampled in each sampling period relative to the original trajectory increases sequentially.

8. A training device for a scene reconstruction model, comprising: The first acquisition module is used to acquire an image dataset collected by the vehicle under the original trajectory and a first point cloud dataset that is synchronized with the image dataset in time. The first point cloud dataset is acquired by LiDAR. The first training module is used to train a first reconstruction model using the image dataset and the first point cloud dataset. The first rendering module is used to render the image under the new trajectory using the first reconstruction model, and obtain the rendered image under the new trajectory. The repair module is used to repair the rendered image under the new trajectory using a pre-trained video repair model to obtain the repaired image under the new trajectory. The second training module is used to train a second reconstruction model using the repaired image under the new trajectory, the Gaussian parameters of the first reconstruction model, the first pose of the vehicle under the original trajectory, and the second pose of the vehicle under the new trajectory. The second reconstruction model is used to generate a rendered image under the new trajectory. The second training module includes: The encoding submodule is used to perform position encoding on the Gaussian parameters and time steps of the first reconstruction model respectively to obtain Gaussian parameter position encoding and time step position encoding. The time step is used to indicate the frame number of the acquired image under the original trajectory in the image dataset. The first processing submodule is used to obtain the first feature based on the Gaussian parameter position encoding and the time step position encoding using the first multilayer perceptron; The second processing submodule is used to obtain the second feature based on the first pose of the vehicle under the original trajectory and the second pose of the vehicle under the new trajectory using the second multilayer perceptron. The parameter update submodule is used to add the first feature and the second feature to obtain the change in the Gaussian parameter, and to obtain the first initial reconstruction model based on the change in the Gaussian parameter and the Gaussian parameter of the first reconstruction model. The first training submodule is used to train the first initial reconstruction model using the repaired image under the new trajectory to obtain the second reconstruction model.

9. An electronic device, characterized in that, include: Memory, used to store computer program products; A processor for executing a computer program product stored in the memory, wherein when the computer program product is executed, it implements the method described in any one of claims 1-7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-7.

11. A computer program product comprising computer program instructions, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-7.

Images