Training method, device, equipment, medium and product of scene reconstruction model

Abstract

The embodiments of the present disclosure disclose a scene reconstruction model training method and device, equipment, medium and product. The method comprises: acquiring collected images under original trajectories in a driving scene; training a video repair model by using a scene reconstruction model with insufficient training and the collected images under the original trajectories; rendering images under new trajectories by using the scene reconstruction model to obtain rendered images under the new trajectories; repairing the rendered images under the new trajectories by using the trained video repair model to obtain repaired images under the new trajectories; and training the scene reconstruction model by using the repaired images under the new trajectories and the collected images under the original trajectories. Thus, the trained scene reconstruction model can improve the rendering quality of the images under the new trajectories, and further improve the performance of the scene reconstruction model.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, scene reconstruction, and automatic driving, and particularly relates to a scene reconstruction model training method, device, equipment, medium and product. BACKGROUND

[0002] In the automatic driving scene reconstruction task, the reconstruction of the scene is generally completed by giving an input driving video. Compared with traditional 3D (three-dimensional) reconstruction, the 3D reconstruction in the automatic driving scene faces greater challenges. This is mainly due to the sparsity of the view angle of the automatic driving data. Generally, compared with traditional 3D reconstruction, there are no rich multi-view images in the data set of automatic driving, and the test set is generally obtained by frame extraction on the original trajectory data, and the training set uses the images left after frame extraction. This makes the existing method mainly focus on the reconstruction quality under the original trajectory, and the rendering effect of the existing method under the new trajectory is poor when rendering the new trajectory such as gradual lane changing and lane translation. SUMMARY

[0003] In view of the above problems of the prior art, the embodiments of the present disclosure provide a scene reconstruction model training method, device, equipment, medium and product.

[0004] The first aspect of the embodiments of the present disclosure provides a scene reconstruction model training method, comprising:

[0005] acquiring collected images under an original trajectory in a driving scene;

[0006] training a video repair model by using a scene reconstruction model that is not fully trained and the collected images under the original trajectory;

[0007] rendering images under a new trajectory by using the scene reconstruction model to obtain rendered images under the new trajectory;

[0008] repairing the rendered images under the new trajectory by using the trained video repair model to obtain repaired images under the new trajectory;

[0009] training the scene reconstruction model by using the repaired images under the new trajectory and the collected images under the original trajectory.

[0010] As a possible implementation manner of the first aspect, the training of the video repair model by using the scene reconstruction model that is not fully trained and the collected images under the original trajectory comprises:

[0011] rendering the collected images under the original trajectory by using the scene reconstruction model that is not fully trained to obtain rendered images under the original trajectory;

[0012] constructing a video repair data set by using the rendered images under the original trajectory and the collected images under the original trajectory;

[0013] Training the video inpainting model by using the video inpainting dataset.

[0014] As a possible implementation manner of the first aspect, training the video inpainting model by using the video inpainting dataset comprises:

[0015] obtaining a rendered image under an original trajectory and a captured image under the original trajectory from the video inpainting dataset;

[0016] inputting the rendered image under the original trajectory into the video inpainting model to obtain an output result by processing of the video inpainting model;

[0017] calculating a loss value of the video inpainting model according to the output result and the captured image under the original trajectory;

[0018] adjusting a model parameter of the video inpainting model according to the loss value.

[0019] As a possible implementation manner of the first aspect, inputting the rendered image under the original trajectory into the video inpainting model to obtain an output result by processing of the video inpainting model comprises:

[0020] adding a mask in the rendered image under the original trajectory, and using the mask to indicate an attention area in a processing process of the video inpainting model.

[0021] As a possible implementation manner of the first aspect, before rendering an image under a new trajectory by using the scene reconstruction model, the method further comprises:

[0022] sampling the new trajectory based on a preset sampling period, and an offset of each sampled new trajectory relative to the original trajectory is sequentially increased.

[0023] As a possible implementation manner of the first aspect, the preset sampling period comprises: sampling the new trajectory after a preset number of iterations in a training process of the scene reconstruction model.

[0024] As a possible implementation manner of the first aspect, the offset of each sampled new trajectory relative to the original trajectory is sequentially increased, comprising:

[0025] setting the offset as a product of a preset offset step and a sampling number.

[0026] As a possible implementation manner of the first aspect, training the scene reconstruction model by using the inpainted image under the new trajectory and the captured image under the original trajectory comprises:

[0027] constructing a reconstruction dataset by using the inpainted image under the new trajectory and the captured image under the original trajectory;

[0028] The sampling probability is set for the repaired image under the new track and the collected image under the original track respectively.

[0029] The image is extracted from the reconstruction dataset based on the sampling probability.

[0030] The extracted image is used to train the scene reconstruction model.

[0031] As a possible implementation of the first aspect, the sampling probability is set for the repaired image under the new track and the collected image under the original track respectively, including:

[0032] The sampling probability corresponding to the new track with a large offset relative to the original track is large.

[0033] As a possible implementation of the first aspect, the extracted image is used to train the scene reconstruction model, including:

[0034] The extracted image is rendered by using the scene reconstruction model to obtain a rendered image of the extracted image.

[0035] The rendered image of the extracted image is repaired by using the trained video repair model to obtain a repaired image of the extracted image.

[0036] The repaired image of the extracted image is added to the reconstruction dataset to obtain an updated reconstruction dataset.

[0037] The scene reconstruction model is trained by using the updated reconstruction dataset.

[0038] The second aspect of the embodiment of the present disclosure provides a training device of a scene reconstruction model, including:

[0039] An acquisition unit is configured to acquire collected images under an original track in a driving scene.

[0040] A first training unit is configured to train a video repair model by using a scene reconstruction model trained insufficiently and the collected images under the original track.

[0041] A rendering unit is configured to render images under a new track by using the scene reconstruction model to obtain rendered images under the new track.

[0042] A repairing unit is configured to repair the rendered images under the new track by using a trained video repair model to obtain repaired images under the new track.

[0043] A second training unit is configured to train the scene reconstruction model by using the repaired images under the new track and the collected images under the original track.

[0044] As a possible implementation of the second aspect, the first training unit includes:

[0045] a rendering subunit configured to render the captured image under the original trajectory using the scene reconstruction model trained insufficiently to obtain a rendered image under the original trajectory;

[0046] a first construction subunit configured to construct a video inpainting dataset using the rendered image under the original trajectory and the captured image under the original trajectory;

[0047] a first training subunit configured to train a video inpainting model using the video inpainting dataset.

[0048] As a possible implementation manner of the second aspect, the first training subunit is configured to:

[0049] obtain the rendered image under the original trajectory and the captured image under the original trajectory from the video inpainting dataset;

[0050] input the rendered image under the original trajectory into the video inpainting model to obtain an output result through processing of the video inpainting model;

[0051] calculate a loss value of the video inpainting model according to the output result and the captured image under the original trajectory;

[0052] adjust a model parameter of the video inpainting model according to the loss value.

[0053] As a possible implementation manner of the second aspect, the first training subunit is configured to:

[0054] add a mask to the rendered image under the original trajectory, and use the mask to indicate an attention area in a processing process of the video inpainting model.

[0055] As a possible implementation manner of the second aspect, the apparatus further includes a sampling unit configured to:

[0056] sample a new trajectory based on a preset sampling period before rendering an image under the new trajectory using the scene reconstruction model, and an offset amount of the new trajectory relative to the original trajectory is sequentially increased in each sampling period.

[0057] As a possible implementation manner of the second aspect, the preset sampling period includes: sampling the new trajectory after a preset number of iterations in a training process of the scene reconstruction model.

[0058] As a possible implementation manner of the second aspect, the sampling unit is configured to:

[0059] set the offset amount as a product of a preset offset step and a sampling number.

[0060] As a possible implementation manner of the second aspect, the second training unit includes:

[0061] The second construction sub-unit is configured to construct a reconstruction dataset by using the repaired image under the new track and the collected image under the original track.

[0062] The setting sub-unit is configured to set a corresponding sampling probability for the repaired image under the new track and the collected image under the original track respectively.

[0063] The extraction sub-unit is configured to extract an image from the reconstruction dataset based on the sampling probability.

[0064] The second training sub-unit is configured to train the scene reconstruction model by using the extracted image.

[0065] As a possible implementation manner of the second aspect, the setting sub-unit is configured to:

[0066] The sampling probability corresponding to the new track with a large offset relative to the original track is large.

[0067] As a possible implementation manner of the second aspect, the second training sub-unit is configured to:

[0068] Render the extracted image by using the scene reconstruction model to obtain a rendered image of the extracted image.

[0069] Repair the rendered image of the extracted image by using the trained video repair model to obtain a repaired image of the extracted image.

[0070] Add the repaired image of the extracted image to the reconstruction dataset to obtain an updated reconstruction dataset.

[0071] Train the scene reconstruction model by using the updated reconstruction dataset.

[0072] The third aspect of the embodiments of the present disclosure provides an electronic device, comprising:

[0073] A memory configured to store a computer program product.

[0074] A processor configured to execute the computer program product stored in the memory, and when the computer program product is executed, the method of any one of the first aspect is implemented.

[0075] The fourth aspect of the embodiments of the present disclosure provides a computer readable storage medium having computer program instructions stored thereon, and when the computer program instructions are executed by a processor, the method of any one of the first aspect is implemented.

[0076] The fifth aspect of the present disclosure provides a computer program product comprising computer program instructions, which, when executed by a processor, implement the method of any one of the first aspect.

[0077] Based on the embodiments of the present disclosure, first, the video inpainting model is trained by using the insufficiently trained scene reconstruction model and the collected images under the original trajectory, then the trained video inpainting model is used to assist the reconstruction model, to obtain the inpainting images under the new trajectory, and the reconstruction dataset is constructed by using the inpainting images under the new trajectory and the collected images under the original trajectory, and the scene reconstruction model is trained by using the reconstruction dataset. The inpainting images under the new trajectory are included in the above reconstruction dataset, and thus the trained scene reconstruction model can improve the image rendering quality under the new trajectory, and further improve the performance of the scene reconstruction model.

[0078] The technical solutions of the present disclosure are described in further detail below by means of the accompanying drawings and embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0079] The accompanying drawings, which form a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0080] The present disclosure can be more clearly understood and appreciated from the following detailed description, taken in conjunction with the accompanying drawings, in which:

[0081] Figure 1 A scene reconstruction effect diagram of an existing method;

[0082] Figure 2 A flowchart of an embodiment of a training method of a scene reconstruction model of the present disclosure;

[0083] Figure 3 A schematic diagram of an overall framework of an embodiment of a training method of a scene reconstruction model of the present disclosure;

[0084] Figure 4 A flowchart of an embodiment of a training method of a scene reconstruction model of the present disclosure;

[0085] Figure 5 A training schematic diagram of an embodiment of a training method of a scene reconstruction model of the present disclosure;

[0086] Figure 6 A training framework diagram of an embodiment of a training method of a scene reconstruction model of the present disclosure;

[0087] Figure 7 A structural schematic diagram of an embodiment of a training device of a scene reconstruction model of the present disclosure;

[0088] Figure 8 A structural schematic diagram of an embodiment of a training device of a scene reconstruction model of the present disclosure;

[0089] Figure 9 A structural schematic diagram of an embodiment of a training device of a scene reconstruction model of the present disclosure;

[0090] Figure 10 A structural schematic diagram of an embodiment of a training device of a scene reconstruction model of the present disclosure;

[0091] Figure 11 A block diagram of an electronic device of an embodiment of the present disclosure. DETAILED DESCRIPTION

[0092] The existing method will be introduced first, and then the technical solution of the present application will be introduced in detail.

[0093] The data collection of automatic driving is generally the data collected for a driving process of a vehicle. The trajectory of the driving process is called an original trajectory. If the vehicle only drives on one lane during the driving process, the collected data may only have data of one lane, so that the data set of automatic driving usually only has data of the original trajectory, and lacks rich multi-view images, resulting in sparsity of the view of the automatic driving data. If a video under a new view is to be rendered, for example, when a new trajectory such as gradual lane changing and lane translation is rendered, the video under the new trajectory generated will have obvious ghosting phenomena, such as trailing, fragmentation, and blurring. Figure 1 As shown in FIG. 1, the left side is the Original Video (video under the original trajectory) of the vehicle collection process. The vehicle in the video drives straight on the original lane, and a lane changing driving scene is to be rendered. 4DGS (4D Gaussian Splatting) is a dynamic scene representation method. Based on this representation, several existing modeling methods on the right side are used to reconstruct (Reconstruction) the Lane Change Novel Trajectory (new trajectory of lane changing), and the rendering (Rendering) results in the rendered images. The right side is the reconstruction results of several existing methods under the trajectory of lane changing. It can be found that the rendering effects of the existing several methods PVG (Periodic Vibration Gaussian), S 3 Gaussian (self-supervised street Gaussian) and Deformable-GS (deformable 3D Gaussian) under the new view of lane changing are very poor. In summary, the rendering effect of the video under the new trajectory generated by the existing method is poor.

[0094] Based on the technical problems existing in the prior art, the embodiment of the present application provides a training method of a scene reconstruction model. First, a video repair model is trained by using an insufficiently trained scene reconstruction model and collected images under an original trajectory. Then, the trained video repair model is used to assist the reconstruction model to obtain a repaired image under a new trajectory. The repaired image under the new trajectory and the collected images under the original trajectory can be used to construct a reconstruction dataset, and the scene reconstruction model is trained by using the reconstruction dataset. The reconstruction dataset includes the repaired image under the new trajectory, thereby solving the problem of sparse view angle of the autonomous driving data. Thus, the trained scene reconstruction model can improve the image rendering quality under the new trajectory, and further improve the performance of the scene reconstruction model, thereby solving the technical problem of poor rendering effect of the generated video under the new trajectory in the prior art.

[0095] Figure 2 A flowchart of an embodiment of the training method of the scene reconstruction model of the present disclosure is shown in FIG. 1. As shown in FIG. 1, the method can specifically include the following steps. Figure 2

[0096] In step S110, collected images under an original trajectory in a driving scene are obtained.

[0097] The collected images under the original trajectory can be data collected in the process of driving a vehicle for a period of time, which is real-world driving data, also referred to as GT (ground truth) video under the original trajectory. The scene reconstruction model training tool, which is the execution subject of the training method of the scene reconstruction model of the present disclosure, first obtains the GT video under the original trajectory, and then uses the video to train a video repair model and a scene reconstruction model in subsequent steps.

[0098] In step S120, a video repair model is trained by using an insufficiently trained scene reconstruction model and collected images under an original trajectory.

[0099] First, the GT video under the original trajectory is used to train the scene reconstruction model. During the training process, an image under an original view is rendered by using the scene reconstruction model by sampling under the original trajectory. The rendering can include a process of extracting an image from a scene given a view angle. When the scene reconstruction model is insufficiently trained, that is, when the model does not completely converge, the result data output by the model is usually a ghosting image with poor rendering effect. The above image with poor rendering effect and the GT video under the original trajectory are used as a video pair, and the video repair model is trained by using the video pair. In subsequent steps, the trained video repair model is used to assist the scene reconstruction model, so that the scene reconstruction model can well render a video under a new view to improve the performance of the scene reconstruction model.

[0100] ​Step S130, render the image under the new trajectory using the scene reconstruction model to obtain a rendered image under the new trajectory.

[0101] In this step, a new trajectory can be sampled first. For example, based on the original trajectory, switch to a perspective that shifts the lane to the left by 1.5 meters, and sample a new trajectory under this perspective. Render the image under the new trajectory using the scene reconstruction model. Since the scene reconstruction model is not fully trained for the new trajectory, the rendering quality of the image under the new trajectory generated by the model is poor, and needs to be repaired using the video repair model.

[0102] Step S140, repair the rendered image under the new trajectory using the trained video repair model to obtain a repaired image under the new trajectory.

[0103] In step S120, the video repair model is trained using the poor-quality rendered image under the original perspective generated by the insufficiently trained scene reconstruction model and the GT video of the original trajectory as a video pair. After the video repair model is trained, the model parameters are locked, and the rendered image under the new trajectory generated in step S130 is repaired using the model to obtain a repaired image under the new trajectory.

[0104] Step S150, train the scene reconstruction model using the repaired image under the new trajectory and the collected image under the original trajectory.

[0105] In this step, the scene reconstruction model is trained using the repaired image under the new trajectory obtained in step S140 and the collected image under the original trajectory obtained in step S110.

[0106] Figure 3 The overall process of training the scene reconstruction model using the training tool of the scene reconstruction model is shown. As shown in Figure 3 “Dynamic Scene Reconstruction” represents the scene reconstruction model. At the beginning of training, the dataset (Dataset) only has the GT video of the original trajectory, as shown in Figure 3The "Original Trajectory GT Video" is used as the initial reference. First, the reconstruction model is trained using the original trajectory's GT video for a period of time. After the scene reconstruction model achieves good processing results with the original trajectory's GT video, a new trajectory can be sampled, for example, shifting the original trajectory slightly to the left by 1.5m. This instructs the scene reconstruction model to render the image under the new trajectory. A progressive trajectory sampling strategy can be employed, gradually increasing the lane offset and the offset viewpoint of the new trajectory relative to the original trajectory during multiple sampling processes, obtaining progressive trajectory samples—a series of new trajectories with different offsets. Then, the scene reconstruction model renders the new trajectory, obtaining the rendered image under the new trajectory. Figure 3 The Novel Trajectory RenderedVideo is generated at this stage because the scene reconstruction model is only trained on the original trajectory. Since the rendered image on the new trajectory generated at this stage may be of poor quality.

[0107] See Figure 3 "Online Restoration" refers to the online restoration process performed by the video restoration model. A projection module is pre-set in the scene reconstruction model training tool. This module is used to reproject other vehicles and lane lines (excluding the vehicle itself) onto the rendered image based on the new trajectory. This projection yields a "3D Box Sequence" and a "HDMap Sequence." Then, the pre-trained, parameter-locked video restoration model, DriveRestorer, is used to restore the rendered image, resulting in a restored image under the new trajectory. Specifically, Figure 3 In this context, "Enc" represents encoding and "Video Decoder" represents decoding. The "Noisy Images," control condition c, and the encoded "3D Box Sequence" and "HDMap Sequence" are input into the video restoration model DriveRestorer. After processing by this model, the result is decoded (Video Decoder) to obtain the restored image under the new trajectory. Figure 3 The "Novel Trajectory Restored Video" is mentioned. The restored image is then added to the reconstructed dataset, and the reconstructed dataset is updated. See [link to documentation]. Figure 3DatasetUpdate, so that the repaired image under the new trajectory is included in the updated dataset. The above completes one iteration process of the process scene reconstruction model training.

[0108] In the next iteration process, the trajectory is resampled from the updated dataset, so that the image under the new trajectory occupies a certain proportion, and the above process is repeatedly executed. The image under the new trajectory is rendered by using the scene reconstruction model, the rendered image under the new trajectory is repaired by using the trained video repair model, and the repaired image is added to the reconstructed dataset for updating the reconstructed dataset. Since the image under the new trajectory occupies a certain proportion when the trajectory is sampled from the reconstructed dataset, the rendering quality of the image under the new trajectory can be improved by multiple iterations.

[0109] In summary, based on the embodiments of the present disclosure, first, the video repair model is trained by using the insufficiently trained scene reconstruction model and the collected image under the original trajectory, and then the trained video repair model is used to assist the reconstruction model to obtain the repaired image under the new trajectory. The reconstructed dataset can be constructed by using the repaired image under the new trajectory and the collected image under the original trajectory, and the scene reconstruction model is trained by using the reconstructed dataset. The above reconstructed dataset includes the repaired image under the new trajectory, and the image under the new trajectory occupies a certain proportion when sampling. Therefore, the rendering quality of the image under the new trajectory can be improved by using the trained scene reconstruction model, and the performance of the scene reconstruction model can be improved.

[0110] As shown in FIG. 1, Figure 4 in an embodiment, Figure 1 the step S120 in the method 100, the video repair model is trained by using the insufficiently trained scene reconstruction model and the collected image under the original trajectory, comprising:

[0111] the step S210, the collected image under the original trajectory is rendered by using the insufficiently trained scene reconstruction model to obtain the rendered image under the original trajectory;

[0112] the step S220, the video repair dataset is constructed by using the rendered image under the original trajectory and the collected image under the original trajectory;

[0113] the step S230, the video repair model is trained by using the video repair dataset.

[0114] The data set of automatic driving usually only has the GT video of the original trajectory, and lacks the video restoration data set for training the video restoration model. The video restoration data set is constructed by rendering the image under the original trajectory by using the under-trained reconstruction model. Specifically, the image under the original trajectory is rendered by using the under-trained scene reconstruction model, and the video restoration data set is constructed with the GT video of the original trajectory, so as to solve the problem of lacking the video restoration data set, and then the video restoration model is trained by using the data set.

[0115] Figure 5 “Restoration Dataset Construction” in FIG. 1 indicates the process of constructing the video restoration data set, and “Training for DriveRestorer” indicates the training process of the video restoration model. See Figure 4 and Figure 5 , Figure 5 “Under-Trained Gaussian Splatting” in FIG. 1 is used as a scene reconstruction model, and the scene reconstruction model is first trained by using the “Original Trajectory GT Video”. The “Original Trajectory GT Video” is the collected image under the original trajectory. In the training process, when the scene reconstruction model is not fully trained, the video under the original trajectory is directly rendered to obtain the “Original Trajectory Degraded Video”, that is, the poor effect of the rendered image, and the image and the collected image under the original trajectory form a video restoration pair to complete the construction of the video restoration data set. The collected image under the original trajectory can be used as the supervision data for model training in the training process to supervise the learning process of the video restoration model DriveRestorer.

[0116] In the embodiments of the present disclosure, the collected image under the original trajectory is rendered by using the under-trained scene reconstruction model, and the video restoration data set is constructed based on the obtained rendered image under the original trajectory, thereby providing a way of constructing the data set for training the video restoration model. By using the way to construct the data set, the problem of insufficient training data of the automatic driving data set in the prior art can be solved, so that the video restoration model can be fully trained by using the data set, and the restoration effect of the model is improved. Meanwhile, the trained video restoration model can assist the scene reconstruction model to render the image, thereby improving the rendering quality.

[0117] In one embodiment, Figure 4In step S230, the video restoration model is trained using the video restoration dataset, including:

[0118] The rendered image under the original trajectory and the captured image under the original trajectory are obtained from the video restoration dataset.

[0119] The rendered image under the original trajectory is input into the video restoration model, and the output result is obtained by processing the video restoration model.

[0120] According to the output result and the captured image under the original trajectory, the loss value of the video restoration model is calculated.

[0121] According to the loss value, the model parameters of the video restoration model are adjusted.

[0122] Referring to Figure 4 and Figure 6 , the video restoration model needs to use the captured image under the original trajectory as supervision data. Figure 6 In the “GT Video (real captured video)” in , the captured image under the original trajectory is represented. The captured image under the original trajectory and the rendered image under the original trajectory are input into the video restoration model as a video pair, and the video restoration model is trained. Figure 6 In the “Degraded Video (original trajectory degraded video)” in , the rendered image under the original trajectory is a poor effect, and the “U-Net” network model is used as the video restoration model. “Enc” represents encoding. Specifically, the rendered image under the original trajectory rendered by the insufficiently trained scene reconstruction model is encoded and input into the video restoration model, and the video restoration model is processed to output the restored image under the original trajectory. Then the loss function of the restoration model is used to constrain the model. Specifically, according to the output result of the above model and the captured image under the original trajectory, the loss value of the video restoration model is calculated, and the model parameters of the video restoration model are adjusted according to the loss value. Figure 6 In the “Diffusion loss (diffusion loss function)” in , the loss function related to the diffusion model (Diffusion Models) is used to guide the model to learn how to gradually remove noise and restore the original structure of the data.

[0123] In one example, the formula of the loss function of the video restoration model is as follows:

[0124]

[0125] wherein, represents the mean value; “‖·‖” represents the norm; ε t represents the noise added in the tth iteration of the diffusion process; x t represents the result after adding noise in the tth iteration; c represents the first frame, structured information, text, and other control conditions; εθ (x t t, c) model prediction of the t-th iteration plus noise.

[0126] Figure 6 “Noise” in the above formula represents noise. The GT Video is input into the encoder, and the encoding process maps the input data into the latent space, and encodes the features in the latent space. The features in the above-mentioned latent space are added with noise (Noise), and are input into the video inpainting model together with the control condition c. The loss (loss) is calculated according to the result output by the model and the added noise (Noise), that is, the predicted random noise and the real random noise are compared, and the loss value is calculated. Using the video inpainting dataset and the above loss function, the inpainting effect of the model can be improved.

[0127] In an embodiment, the rendered image under the original trajectory is input into the video inpainting model, and the output result is obtained after processing by the video inpainting model, which includes:

[0128] A mask is added to the rendered image under the original trajectory, and the mask is used to indicate the area of interest in the processing process of the video inpainting model.

[0129] Specifically, since the rendering result of the scene reconstruction model is usually poor at the junction of the sky and the background in the image, and the quality is poor at the center of the image (specifically, far from the center of the camera), a mask strategy can be used for the above-mentioned areas, that is, a number of random masks (Masks) are added to the above-mentioned areas during training, aiming to guide the model to learn the task of video inpainting, and specifically guide the model to focus on the area where the mask is located. The position and size of the mask can be random, and the boundary shape of the mask can be rectangular or other graphics. See Figure 5 and Figure 6 “Mask Sequence” in the above formula represents a series of masks respectively set for multiple frames of images in the “Degraded Video”, and the mask can improve the inpainting effect of the model.

[0130] On the other hand, see Figure 6 and Figure 3 In order to improve the control of the model on the foreground vehicle and the background lane line, the 3D bounding box of the vehicle and the lane line are introduced as the control condition, and the corresponding vehicle and lane line are generated based on the corresponding position of the control condition in the image. Figure 3 and Figure 6“3D Box Sequence” in the “3D Box Sequence” represents the 3D bounding box of the vehicle, and the relevant information of the lane line can be obtained from the “HDMap Sequence”. Specifically, the camera pose under the new trajectory can be calculated according to the distance of the lane offset, and then the data sampled by the laser radar in the driving scene and the GT video of the original trajectory, as well as the camera pose under the new trajectory, are input into the projection module in the scene reconstruction model training tool. After the processing of the projection module, the 3D Box Sequence and the HDMap Sequence are obtained.

[0131] The video repair model is a diffusion model, and its input is a random noise image (Noise). According to the noise, a video is generated. The rendered image under the new trajectory, the 3D Box Sequence and the HDMap Sequence are used as control conditions. The above three control conditions jointly guide the video repair model to generate a repaired video according to the noise image, thereby improving the quality of the generated image.

[0132] In an embodiment, before rendering the image under the new trajectory by the scene reconstruction model in step S130, the above method further comprises:

[0133] The new trajectory is sampled based on a preset sampling period, and the offset amount of the new trajectory sampled in each sampling period relative to the original trajectory is sequentially increased.

[0134] Referring back to Figure 3 The scene reconstruction model can be trained using a progressive data update strategy (Progressive Data Update Strategy). The progressive data update strategy can include using a progressive strategy to sample the new trajectory. That is, during the model training process, the offset amount of the new trajectory relative to the original trajectory gradually increases. For example, first based on the original trajectory, switch to a perspective of offsetting the lane to the left by 1.5 meters, and sample a new trajectory under the perspective. Then perform step S130 to render the image under the new trajectory by the scene reconstruction model to obtain the rendered image under the new trajectory. Then perform step S140 to repair the rendered image under the new trajectory by the video repair model to obtain the repaired image under the new trajectory. Then add the repaired image to the reconstruction dataset to update the reconstruction dataset. Referring back to Figure 3"lane shift @ 1.5m Restored Video" in FIG. 6 indicates the restored image under the new trajectory with a lane shift of 1.5 m. The above iteration process is completed once. Based on the preset sampling period, in the next iteration process, the view angle is switched to a view angle with a lane shift of 3.0 m to the left based on the original trajectory, and a new trajectory with a larger shift is resampled at the view angle. Then steps S130 and S140 are repeatedly executed, and the new restored image is updated to the reconstruction data set. Referring to FIG. 6, "lane shift @ 3.0m Restored Video" indicates the restored image under the new trajectory with a lane shift of 3.0 m. The above iteration process is repeated, and the shift of the new trajectory sampled at each sampling period relative to the original trajectory is sequentially increased. Figure 3

[0135] In the above sampling strategy, the shift of the new trajectory sampled at each sampling period relative to the original trajectory is sequentially increased, the model is gradually trained under the trajectory with a smaller shift first, and then gradually trained under the trajectory with a larger shift, which can gradually increase the difficulty of learning of the model and achieve a better training effect.

[0136] In an embodiment, the preset sampling period includes: sampling the new trajectory after a preset number of iteration processes in the training process of the scene reconstruction model.

[0137] In an example, when training the scene reconstruction model, a new trajectory is sampled once every S steps. For example, after the model is trained under the original trajectory for S times, a new trajectory is sampled, and the view angle is switched to a view angle with a lane shift of 1.5 m to the left for further training. After the view angle with a lane shift of 1.5 m to the left is iterated for S times, a new trajectory with a larger shift is further sampled. Similarly, a new trajectory with a shift to the right can also be gradually sampled. The preset sampling period includes sampling a new trajectory after S iterations. In the above example, the iteration number S of each sampling period can be set to 100. By reasonably setting the sampling period, the training efficiency of the model can be improved.

[0138] In an embodiment, the shift of the new trajectory sampled at each period relative to the original trajectory is sequentially increased, including:

[0139] The shift is set to the product of the preset shift step and the sampling number.

[0140] ​In one example, for the k-th sampling of the novel trajectory, the offset of the novel trajectory is set as y=kΔy, where Δy is the step length of the translation. For example, the step length of the translation is set as 1.5 meters, the offset of the novel trajectory for the 1st sampling is 1.5 meters, and the offset of the novel trajectory for the 2nd sampling is 3.0 meters. According to the actual situation of the scene, for example, the width of the road, the number of iterations is determined, and the iteration is stopped until the translation is far away. In each iteration process, the model parameters are adjusted according to the loss function. In this way, the offset of the view angle is gradually increased, so that the scene view angle in the data set is more and more, and the scene view angle is more and more far away. And by reasonably setting the offset step length, a good model training effect can be achieved.

[0141] In one embodiment, Figure 2 In step S150 of the method 100, the scene reconstruction model is trained using the restored images under the novel trajectory and the collected images under the original trajectory, including:

[0142] The restored images under the novel trajectory and the collected images under the original trajectory are used to construct a reconstruction data set.

[0143] The sampling probabilities corresponding to the restored images under the novel trajectory and the collected images under the original trajectory are set respectively.

[0144] Based on the sampling probabilities, images are extracted from the reconstruction data set.

[0145] The extracted images are used to train the scene reconstruction model.

[0146] Referring back to FIG. 1, Figure 3 “Original Trajectory GT Video (Original Trajectory GT Video)” represents the collected images under the original trajectory, and “lane shift@1.5m Restored Video (lane shift@1.5m Restored Video)” and “lane shift@3.0m Restored Video (lane shift@3.0m Restored Video)” represent the restored images under the novel trajectory. First, the restored images under the novel trajectory and the collected images under the original trajectory are used to construct a reconstruction data set in the process of iterative updating. Referring to FIG. 2, Figure 3 “Data Sample Probability (Data Sample Probability)” in FIG. 2, and the icons of “Original Trajectory Video (Original Trajectory Video)” and “Novel Trajectory Video (Novel Trajectory Video)”, when updating the data for training the scene reconstruction model, different sampling probabilities can be set for different trajectories. Referring to FIG. 3, Figure 3In the "Weighted Data Sample" in the specification, at the beginning of each iteration, an image is extracted from the reconstruction dataset based on the sampling probability, and the extracted image is used to train the scene reconstruction model. Based on the reasonable setting of the sampling probability, the training effect of the scene reconstruction model can be improved.

[0147] In an embodiment, the corresponding sampling probability is set for the repaired image under the new trajectory and the collected image under the original trajectory, respectively, including:

[0148] For the new trajectory with a large offset relative to the original trajectory, the corresponding sampling probability is large.

[0149] For example, the sampling probability of the collected image under the original trajectory is set to 0.5, and the sum of the sampling probabilities of the repaired images under the other new trajectories is also set to 0.5. The sampling probability of the new trajectory data follows the following rules:

[0150] D novel =(1-ω)·D novel ∪ω·V novel

[0151] Where V novel is the latest trajectory repaired image, that is, the trajectory repaired image corresponding to the kth sampling; D novel is the new trajectory dataset (containing new trajectories with different offsets); ω represents the sampling probability of the latest trajectory, that is, the new trajectory with a large offset relative to the original trajectory. Based on this formula, the data sampling probability of the new trajectory with a large offset can be guaranteed to be higher.

[0152] In an example, in the reconstruction dataset, for each trajectory with an offset of one step, there is a corresponding video. During the training of the scene reconstruction model, an image is extracted from the video in the reconstruction dataset and input into the model for each model inference, and the loss value of the model is calculated after the model processing. For example, the scene reconstruction model is trained for 1000 times, that is, iterated for 1000 times, and an image is extracted from the reconstruction dataset each time. The sampling probability of the collected image under the original trajectory is set to 0.5, so that 500 times of iteration extract the collected image under the original trajectory. A total of 500 images are extracted in the remaining 500 iterations. Among them, the probability of extracting images from the trajectory video with a larger offset relative to the original trajectory is the largest.

[0153] In contrast, the trajectory video with a smaller offset has been trained more sufficiently in the previous iteration process, and the processing difficulty of the trajectory with a smaller offset is also smaller. Therefore, the probability of extracting images in the trajectory video with a larger offset is larger, so that the model can achieve better processing effect for the difficult processing task. Based on the above progressive strategy, the sampling probability of the trajectory with a larger offset is larger, and the more new perspective training data is, which can improve the image generation effect of the scene reconstruction model.

[0154] In summary, the progressive data generation strategy in the embodiment of the present disclosure includes:

[0155] 1) When sampling a new trajectory, the new perspective gradually expands from the original trajectory to both sides of the lane, for example, the offset is 1.5 m each time, and the offset is 3.0 m the second time.

[0156] 2) When training the scene reconstruction model, set a progressive data sampling probability. The collected images under the original trajectory are set to a certain fixed sampling probability, and the remaining sampling probability is allocated by the videos under all new trajectories. The sampling probability of the video corresponding to the trajectory with the largest offset is larger.

[0157] In one embodiment, the scene reconstruction model is trained using the extracted images, including:

[0158] rendering the extracted images using the scene reconstruction model to obtain rendered images of the extracted images;

[0159] repairing the rendered images of the extracted images using the trained video repair model to obtain repaired images of the extracted images;

[0160] adding the repaired images of the extracted images to the reconstruction dataset to obtain an updated reconstruction dataset;

[0161] training the scene reconstruction model using the updated reconstruction dataset.

[0162] Referring to Figure 3 extracting images from the reconstruction dataset Dataset, rendering them using the scene reconstruction model, repairing them using the video repair model, and adding the repaired images to the reconstruction dataset to update the dataset. After completing one iteration in this way, a trajectory with a larger offset is sampled in the next iteration, and the updated reconstruction dataset is used to adopt a progressive data update strategy, so that the trajectory sampling probability of the newly added trajectory is larger, and the scene reconstruction model is trained again.

[0163] In one example, when training the reconstruction model, the training loss (loss function) of the data under the original trajectory is as follows:

[0164]

[0165] wherein, represents a learnable parameter of a Gaussian sphere, is an RGB loss, is a depth map loss, is an SSIM (Structural Similarity Index Measure) loss.

[0166] wherein, the Gaussian sphere is a digital representation of a 3D scene. The RGB loss is an L1 norm loss. The GT data originally collected includes radar data, and the radar data has depth information. The Gaussian sphere is a whole scene three-dimensional representation, and a depth map can also be obtained based on the Gaussian sphere. The depth map of the above GT data, the depth map rendered by the Gaussian sphere, and the loss (loss value) obtained therebetween are the depth map loss. The SSIM is an index for measuring the structural similarity between two images, and is mainly used to evaluate the effect of image processing algorithms, such as image compression, image enhancement, super-resolution, etc. The SSIM loss is the SSIM loss value obtained between the image rendered by the Gaussian sphere and the GT data image.

[0167] When training the reconstruction model, the training loss (loss function) of the data under the new track is as follows:

[0168]

[0169] wherein, there is no corresponding depth map in the data under the new track, so there is no depth information for supervision, and only the RGB and SSIM losses in the above formula.

[0170] When training the reconstruction model, the overall training loss is as follows:

[0171]

[0172] As shown in Figure 7 The present disclosure also provides an embodiment of a training device of a scene reconstruction model. The beneficial effects or technical problems solved by the device can be referred to the description in the method corresponding to the device, or the summary, which will not be repeated here.

[0173] In the embodiment of the training device of the scene reconstruction model, the device comprises:

[0174] The acquisition unit 100 is configured to acquire the collected images under the original track in the driving scene.

[0175] The first training unit 200 is configured to train the video inpainting model by using the scene reconstruction model with insufficient training and the collected images under the original track.

[0176] The rendering unit 300 is configured to render an image under a new trajectory by using a scene reconstruction model to obtain a rendered image under the new trajectory.

[0177] The repairing unit 400 is configured to repair the rendered image under the new trajectory by using the trained video repairing model to obtain a repaired image under the new trajectory.

[0178] The second training unit 500 is configured to train the scene reconstruction model by using the repaired image under the new trajectory and the collected image under the original trajectory.

[0179] As shown in FIG. 1, in an embodiment, the first training unit 200 includes: Figure 8 The rendering sub-unit 210 is configured to render the collected image under the original trajectory by using the scene reconstruction model that is not fully trained to obtain a rendered image under the original trajectory.

[0180] The first constructing sub-unit 220 is configured to construct a video repairing data set by using the rendered image under the original trajectory and the collected image under the original trajectory.

[0181] The first training sub-unit 230 is configured to train the video repairing model by using the video repairing data set.

[0182] In an embodiment, the first training sub-unit 230 is configured to:

[0183] obtain the rendered image under the original trajectory and the collected image under the original trajectory from the video repairing data set;

[0184] input the rendered image under the original trajectory into the video repairing model to obtain an output result by processing of the video repairing model;

[0185] calculate a loss value of the video repairing model according to the output result and the collected image under the original trajectory;

[0186] adjust a model parameter of the video repairing model according to the loss value.

[0187] In an embodiment, the first training sub-unit 230 is configured to:

[0188] add a mask in the rendered image under the original trajectory, and use the mask to indicate an attention area in a processing process of the video repairing model.

[0189] As shown in FIG. 1, in an embodiment, the apparatus further includes a sampling unit 600, which is configured to:

[0190] Figure 9

[0191] ​​Before rendering images under the new trajectory by using the scene reconstruction model, the new trajectory is sampled based on a preset sampling period, and offset amounts of the new trajectory sampled in each sampling period relative to the original trajectory are sequentially increased.

[0192] In an implementation, the preset sampling period comprises: sampling the new trajectory after a preset number of iterations in a training process of the scene reconstruction model.

[0193] In an implementation, the sampling unit 600 is configured to:

[0194] The offset amount is set as a product of a preset offset step and the sampling number.

[0195] As shown in FIG. 6, in an implementation, the second training unit 500 comprises: Figure 10 The second construction sub-unit 510 is configured to construct a reconstruction dataset by using the repaired images under the new trajectory and the collected images under the original trajectory.

[0196] The setting sub-unit 520 is configured to set a corresponding sampling probability for the repaired images under the new trajectory and the collected images under the original trajectory respectively.

[0197] The extraction sub-unit 530 is configured to extract images from the reconstruction dataset based on the sampling probability.

[0198] The second training sub-unit 540 is configured to train the scene reconstruction model by using the extracted images.

[0199] In an implementation, the setting sub-unit 520 is configured to:

[0200] The sampling probability corresponding to the new trajectory with a larger offset amount relative to the original trajectory is set to be larger.

[0201] In an implementation, the second training sub-unit 540 is configured to:

[0202] Render the extracted images by using the scene reconstruction model to obtain rendered images of the extracted images.

[0203] Repair the rendered images of the extracted images by using the trained video repair model to obtain repaired images of the extracted images.

[0204] Add the repaired images of the extracted images to the reconstruction dataset to obtain an updated reconstruction dataset.

[0205] Train the scene reconstruction model by using the updated reconstruction dataset.

[0206]

[0207] ​The training apparatus of the scene reconstruction model of the embodiments of the present disclosure and the training method embodiments of the scene reconstruction model of the present disclosure correspond to each other in specific implementation and beneficial technical effects, and related content can be mutually referred to, which will not be repeated here.

[0208] The electronic device according to the embodiments of the present disclosure will be described below with reference to Figure 11 The electronic device can be either or both of the first device and the second device, or a single device independent of them, which can communicate with the first device and the second device to receive the collected input signals therefrom.

[0209] Figure 11 FIG. 1 illustrates a block diagram of an electronic device according to an embodiment of the present disclosure.

[0210] As shown in Figure 11 The electronic device includes one or more processors and a memory.

[0211] The processor can be a central processing unit (CPU) or other form of processing unit having data processing and / or instruction execution capabilities, and can control other components in the electronic device to perform desired functions.

[0212] The memory can store one or more computer program products, and the memory can include various forms of computer readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM), cache memory, and / or the like. The non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, and / or the like. One or more computer program products can be stored on the computer readable storage medium, and the processor can run the computer program product to implement the training method of the scene reconstruction model of the embodiments of the present disclosure and / or other desired functions.

[0213] In one example, the electronic device can further include an input device and an output device, which are interconnected through a bus system and / or other forms of connection mechanism (not shown).

[0214] In addition, the input device can further include, for example, a keyboard, a mouse, and / or the like.

[0215] The output device can output various information, including determined distance information, direction information, and / or the like, to the outside. The output device can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and / or the like.

[0216] Of course, in order to simplify, Figure 11Only some of the components of the electronic device related to the present disclosure are shown in the figure, and components such as a bus, an input / output interface, and the like are omitted. In addition to this, the electronic device can include any other appropriate components according to a specific application.

[0217] In addition to the above method and device, an embodiment of the present disclosure can also be a computer program product, which includes computer program instructions, and the computer program instructions make the processor execute the steps in the training method of the scene reconstruction model according to various embodiments of the present disclosure described in the above part of the specification when the processor runs.

[0218] The computer program product can be written in any combination of one or more programming languages, including an object-oriented programming language, such as Java, C++, and the like, and conventional procedural programming languages, such as the "C" programming language, or the like. The program code can execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server.

[0219] In addition, an embodiment of the present disclosure can also be a computer readable storage medium, which stores computer program instructions, and the computer program instructions make the processor execute the steps in the training method of the scene reconstruction model according to various embodiments of the present disclosure described in the above part of the specification when the processor runs.

[0220] The computer readable storage medium can take any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium may, for example, include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any combination of the above. More specific examples (a non-exhaustive list) of the readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any appropriate combination of the above.

[0221] The above describes the basic principles of the present disclosure in conjunction with specific embodiments, but it should be noted that the advantages, benefits, effects and the like mentioned in the present disclosure are merely examples and are not limiting, and these advantages, benefits, effects and the like cannot be considered as necessary for each embodiment of the present disclosure. In addition, the above specific details are merely for the purpose of example and understanding, and are not limiting, and the above details do not limit the present disclosure to be necessarily implemented with the above specific details.

[0222] Each embodiment in the specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between each embodiment can be mutually referred to. For system embodiments, since they basically correspond to method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0223] The block diagrams of the devices, apparatuses, equipment, systems involved in the present disclosure are merely exemplary examples and are not intended to require or imply the connection, arrangement, configuration shown in the block diagram. As those skilled in the art will recognize, these devices, apparatuses, equipment, systems can be connected, arranged, configured in any manner. Words such as "include", "contain", "have" and the like are open-ended words, which mean "including but not limited to", and can be used interchangeably. The words "or" and "and" used herein mean the word "and / or", and can be used interchangeably unless the context clearly indicates otherwise. The word "such as" used herein means the phrase "such as but not limited to", and can be used interchangeably.

[0224] The methods and devices of the present disclosure can be implemented in many ways. For example, the methods and devices of the present disclosure can be implemented by software, hardware, firmware, or any combination of software, hardware, firmware. The above order of steps for the method is merely for illustration, and the steps of the method of the present disclosure are not limited to the above specific description, unless otherwise specifically described. In addition, in some embodiments, the present disclosure can also be implemented as programs recorded in recording media, which include machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers the recording media storing the programs for executing the method according to the present disclosure.

[0225] It should also be noted that in the devices, equipment and methods of the present disclosure, each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions of the present disclosure.

[0226] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other aspects without departing from the scope of the disclosure. Thus, the present disclosure is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0227] The above description has been presented to enable any person skilled in the art to make or use the disclosure. Furthermore, the purpose of the above description is not intended to limit the embodiments of the present disclosure to the form disclosed herein. Although various example aspects and embodiments have been discussed above, those of ordinary skill in the art will appreciate a variety of modifications, alternatives, permutations, additions, and sub-combinations of the described aspects and embodiments.

Claims

1. A method for training a scene reconstruction model, characterized in that, The method comprises: acquiring collected images under an original trajectory in a driving scene; training a video inpainting model by using an insufficiently trained scene reconstruction model and the collected images under the original trajectory; rendering images under a new trajectory by using the scene reconstruction model to obtain rendered images under the new trajectory; inpainting the rendered images under the new trajectory by using the trained video inpainting model to obtain inpainted images under the new trajectory; training the scene reconstruction model by using the inpainted images under the new trajectory and the collected images under the original trajectory; wherein the training of the video inpainting model by using the insufficiently trained scene reconstruction model and the collected images under the original trajectory comprises: rendering the collected images under the original trajectory by using the insufficiently trained scene reconstruction model to obtain rendered images under the original trajectory; constructing a video inpainting dataset by using the rendered images under the original trajectory and the collected images under the original trajectory; training the video inpainting model by using the video inpainting dataset; wherein the inpainting of the rendered images under the new trajectory by using the trained video inpainting model to obtain the inpainted images under the new trajectory comprises: inputting the rendered images under the new trajectory, a 3D bounding box sequence of a vehicle, and a high-definition map sequence into the trained video inpainting model for inpainting to obtain the inpainted images under the new trajectory.

2. The method of claim 1, wherein, The training of the video inpainting model by using the video inpainting dataset comprises: obtaining the rendered images under the original trajectory and the collected images under the original trajectory from the video inpainting dataset; inputting the rendered images under the original trajectory into the video inpainting model to obtain an output result by processing of the video inpainting model; calculating a loss value of the video inpainting model according to the output result and the collected images under the original trajectory; adjusting model parameters of the video inpainting model according to the loss value.

3. The method of claim 2, wherein, The inputting of the rendered images under the original trajectory into the video inpainting model to obtain the output result by processing of the video inpainting model comprises: adding a mask to the rendered images under the original trajectory, and using the mask to indicate an attention area in a processing process of the video inpainting model.

4. The method according to any one of claims 1 to 3, characterized in that, Before the rendering of the images under the new trajectory by using the scene reconstruction model, the method further comprises: sampling the new trajectory based on a preset sampling period, and an offset of each sampled new trajectory relative to the original trajectory is sequentially increased.

5. The method of claim 4, wherein, The preset sampling period comprises: sampling the new trajectory after a preset number of iteration processes in a training process of the scene reconstruction model.

6. The method of claim 4, wherein, The sequentially increased offset of each sampled new trajectory relative to the original trajectory comprises: setting the offset as a product of a preset offset step and a sampling number.

7. The method according to any one of claims 1 to 3, characterized in that, The training of the scene reconstruction model by using the inpainted images under the new trajectory and the collected images under the original trajectory comprises: constructing a reconstruction dataset by using the inpainted images under the new trajectory and the collected images under the original trajectory; setting a corresponding sampling probability for the inpainted images under the new trajectory and the collected images under the original trajectory, respectively; extract an image from the reconstructed dataset based on the sampling probability; train the scene reconstruction model using the extracted image.

8. The method of claim 7, wherein, The setting of the corresponding sampling probability for the repair image under the new trajectory and the collection image under the original trajectory respectively includes: For a new trajectory with a large offset relative to the original trajectory, the corresponding set sampling probability is large.

9. The method of claim 7, wherein, The training of the scene reconstruction model using the extracted image includes: rendering the extracted image using the scene reconstruction model to obtain a rendered image of the extracted image; repairing the rendered image of the extracted image using the trained video repair model to obtain a repair image of the extracted image; adding the repair image of the extracted image to the reconstructed dataset to obtain an updated reconstructed dataset; training the scene reconstruction model using the updated reconstructed dataset.

10. A training device for a scene reconstruction model, characterized in that, It includes: An acquisition unit is configured to acquire a collection image under an original trajectory in a driving scene. A first training unit is configured to train a video repair model using an insufficiently trained scene reconstruction model and the collection image under the original trajectory. A rendering unit is configured to render an image under a new trajectory using the scene reconstruction model to obtain a rendered image under the new trajectory. A repair unit is configured to repair the rendered image under the new trajectory using the trained video repair model to obtain a repair image under the new trajectory. A second training unit is configured to train the scene reconstruction model using the repair image under the new trajectory and the collection image under the original trajectory. The first training unit includes: A rendering subunit is configured to render the collection image under the original trajectory using the insufficiently trained scene reconstruction model to obtain a rendered image under the original trajectory. A first construction subunit is configured to construct a video repair dataset using the rendered image under the original trajectory and the collection image under the original trajectory. A first training subunit is configured to train the video repair model using the video repair dataset. The repair unit is specifically configured to input the rendered image under the new trajectory, a 3D bounding box sequence of a vehicle, and a high-definition map sequence into the trained video repair model for repair to obtain the repair image under the new trajectory.

11. An electronic device, comprising: It includes: A memory is configured to store a computer program product. A processor is configured to execute the computer program product stored in the memory, and when the computer program product is executed, the method of any one of claims 1-9 is implemented.

12. A computer-readable storage medium having stored thereon computer program instructions, wherein, The computer program instructions are executed by the processor to implement the method of any one of claims 1-9.

13. A computer program product comprising computer program instructions, characterised in that, The computer program instructions are executed by the processor to implement the method of any one of claims 1-9. The computer program instructions are executed by the processor to implement the method of any one of claims 1-9.

Images