Enhancement of scene reconstruction model, video generation method and device, equipment and medium

Abstract

This invention provides an enhancement method for a scene reconstruction model, a video generation method, an apparatus, a device, and a medium. The method includes: generating a video with a new trajectory that differs from the camera trajectory in the original video using an original scene reconstruction model trained on the original video. Next, an image inpainting model is used to inpaint the images in the generated video with the new trajectory, resulting in an inpainted video with the new trajectory. The original scene reconstruction model is then trained using the inpainted video with the new trajectory to obtain an enhanced scene reconstruction model. This process uses the inpainted video as extended training data, enabling the model to learn not only from the original video but also from the new trajectory data, thereby enhancing scene reconstruction capabilities. The enhanced model effectively reduces the difference between the generated video and the original video, significantly reduces artifacts (such as ghosting and speckle), and improves reconstruction quality. Furthermore, this model performs better when rendering complex scenes (such as multi-lane transitions or large viewpoint shifts).

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method for enhancing a scene reconstruction model, a video generation method, a device for enhancing a scene reconstruction model, a video generation device, an electronic device, and a readable storage medium. Background Technology

[0002] End-to-end planning (directly translating sensor data into control commands) is one of the most critical tasks in autonomous driving. However, current open-loop evaluation methods fall short in providing accurate assessments of end-to-end planning algorithms, highlighting the need for a more robust evaluation framework. A promising approach to address this issue is to perform closed-loop evaluation in real-world scenarios, which requires retrieving sensor data from new trajectory views. This necessitates that the driving scenario representation be able to reconstruct the complexity and dynamics of the driving environment.

[0003] Closed-loop simulation primarily relies on scene reconstruction methods such as Neural Radiance Field (NeRF) and 3D Gaussian Splatting (3DGS). While these techniques contribute, they are fundamentally limited by the density and diversity of the training data, typically restricting their rendering capabilities to scenes closely related to the original training data. Consequently, they perform poorly in complex and varied driving operations. Current advancements in autonomous driving world models have introduced the ability to generate diverse videos based on specific driving commands, reigniting the potential for more robust closed-loop simulation. The recent DriveDreamer4D further demonstrates that utilizing pre-trained world models as data machines can significantly improve the quality of dynamic driving scene reconstruction. However, while this approach is highly efficient without requiring additional training to integrate the knowledge of the world model, its current design can introduce artifacts when handling complex scenes such as multi-lane changes or large viewpoint shifts, impacting the quality of the reconstruction results. Summary of the Invention

[0004] The technical problem to be solved by the embodiments of the present invention is to provide a method for enhancing a scene reconstruction model, a video generation method, an apparatus, an electronic device, and a readable storage medium, so as to solve the problem of poor scene reconstruction quality, especially when dealing with complex scenes (such as multi-lane changes or large viewpoint shifts) which may produce artifacts and affect the quality of reconstruction results.

[0005] To address the above problems, this invention provides an enhancement method for a scene reconstruction model, the method comprising:

[0006] A new trajectory is generated by using the original scene reconstruction model; wherein the original scene reconstruction model is trained using the original video, and the new trajectory is a trajectory different from the original trajectory, and the original trajectory is the camera trajectory of the original video;

[0007] An image restoration model is used to restore the images in the generated video of the new trajectory, resulting in a restored video of the new trajectory.

[0008] The original scene reconstruction model is trained using the repaired video with the new trajectory to obtain an enhanced scene reconstruction model.

[0009] Optionally, before generating a new trajectory video by reconstructing the model using the original scene, the method further includes:

[0010] Using the original scene reconstruction model, a generated video of the original trajectory is generated;

[0011] Based on the generated video of the original trajectory and the original video, image pair training data is determined, wherein the image in the generated video of the original trajectory is used as the input image in the image pair, and the image in the original video is used as the ground truth image in the image pair.

[0012] The image restoration model is trained using the image pairs as training data.

[0013] Optionally, before training the image restoration model using the image pairs as training data, the method further includes:

[0014] Obtain the true video frame;

[0015] Based on the true video frame, a masked video frame is randomly generated;

[0016] The pedestal image restoration model is trained using the masked video frames and the ground truth video frames.

[0017] The step of training the image restoration model using the image pairs as training data includes:

[0018] The image pairs are used as training data to train the base image restoration model, thus obtaining the image restoration model.

[0019] Optionally, the process of using the repaired video with the new trajectory to train the original scene reconstruction model to obtain an enhanced scene reconstruction model includes:

[0020] The original video and the repaired video with the new trajectory are used to train the original scene reconstruction model to obtain an enhanced scene reconstruction model.

[0021] Optionally, the new trajectory is the first sub-trajectory in the new complete trajectory, and the new complete trajectory includes m consecutive adjacent sub-trajectories. The process of training the original scene reconstruction model using the repaired video with the new trajectory to obtain an enhanced scene reconstruction model includes:

[0022] The repaired video of the first sub-trajectory is used to train the original scene reconstruction model to obtain the scene reconstruction model of the first training stage;

[0023] The method further includes:

[0024] Using the scene reconstruction model in the nth training stage, generate a video of the (n+1)th sub-trajectory in the new complete trajectory, where n is a positive integer from 1 to (m-1);

[0025] Using the image restoration model, the image in the generated video of the (n+1)th sub-trajectory is restored to obtain the restored video of the (n+1)th sub-trajectory;

[0026] The scene reconstruction model of the nth training stage is trained using the repaired video of the (n+1)th sub-trajectory to obtain the scene reconstruction model of the (n+1)th training stage.

[0027] Until n increases from 1 to m-1, the scene reconstruction model in the m-th training stage is used as the final enhanced scene reconstruction model.

[0028] Optionally, the step of using the repaired video of the (n+1)th sub-trajectory to train the scene reconstruction model of the nth training stage to obtain the scene reconstruction model of the (n+1)th training stage includes:

[0029] The scene reconstruction model in the nth training stage is trained using the original video and the repaired video of the (n+1)th sub-trajectory to obtain the scene reconstruction model in the (n+1)th training stage.

[0030] Optionally, the step of using the original scene reconstruction model to generate a new trajectory video includes:

[0031] Obtain the structured information of the images in the original video;

[0032] Reconstruct the model using the original scene to generate 3D scene information;

[0033] Based on the required new trajectory and the structured information, the 3D scene information is rendered through projection transformation to obtain the generated video of the new trajectory.

[0034] The present invention also provides a video generation method, the method comprising:

[0035] Input the target trajectory into the trained enhanced scene reconstruction model as described above;

[0036] Based on the target trajectory, the enhanced scene reconstruction model generates a video of the target trajectory.

[0037] The present invention also provides an enhancement device for a scene reconstruction model, the device comprising:

[0038] The video generation module is used to generate a new trajectory video using the original scene reconstruction model; wherein the original scene reconstruction model is trained using the original video, and the new trajectory is a trajectory different from the original trajectory, and the original trajectory is the camera trajectory of the original video;

[0039] The video restoration module is used to restore the images in the generated video of the new trajectory using an image restoration model, so as to obtain the restored video of the new trajectory.

[0040] The model enhancement module is used to train the original scene reconstruction model using the repaired video with the new trajectory to obtain an enhanced scene reconstruction model.

[0041] Optionally, the device further includes:

[0042] The original trajectory video generation module is used to generate the original trajectory video using the original scene reconstruction model before generating the new trajectory video using the original scene reconstruction model.

[0043] The image pair determination module is used to determine image pair training data based on the generated video of the original trajectory and the original video, wherein the images in the generated video of the original trajectory are used as input images in the image pair, and the images in the original video are used as ground truth images in the image pair.

[0044] The image restoration model training module is used to train the image restoration model using the image pairs as training data.

[0045] Optionally, the device further includes:

[0046] The truth value acquisition module is used to acquire truth video frames before the image restoration model is trained using the image pair training data;

[0047] The mask generation module is used to randomly generate masked video frames based on the true video frames;

[0048] The base training module is used to train a base image restoration model using the masked video frame and the ground truth video frame.

[0049] The image restoration model training module includes:

[0050] The image restoration model training submodule is used to train the base image restoration model using the image as training data to obtain the image restoration model.

[0051] Optionally, the model enhancement module includes:

[0052] The model enhancement submodule is used to train the original scene reconstruction model using the original video and the repaired video of the new trajectory to obtain an enhanced scene reconstruction model.

[0053] Optionally, the new trajectory is the first sub-trajectory in the new complete trajectory, and the new complete trajectory includes m consecutively adjacent sub-trajectories. The model enhancement module includes:

[0054] The first model enhancement submodule is used to train the original scene reconstruction model using the repaired video of the first sub-trajectory to obtain the scene reconstruction model in the first training stage.

[0055] The device further includes:

[0056] The nth video generation module is used to generate a video of the (n+1)th sub-trajectory in the new complete trajectory using the scene reconstruction model of the nth training stage, where n is a positive integer from 1 to (m-1).

[0057] The repair module is used to repair the image in the generated video of the (n+1)th sub-trajectory using the image repair model, so as to obtain the repaired video of the (n+1)th sub-trajectory.

[0058] The training module is used to train the scene reconstruction model of the nth training stage using the repaired video of the (n+1)th sub-trajectory, so as to obtain the scene reconstruction model of the (n+1)th training stage.

[0059] The model determination module is used to take the scene reconstruction model of the m-th training stage as the final enhanced scene reconstruction model until n reaches m-1 from 1.

[0060] Optionally, the training module includes:

[0061] The training submodule is used to train the scene reconstruction model of the nth training stage using the original video and the repaired video of the (n+1)th sub-trajectory, so as to obtain the scene reconstruction model of the (n+1)th training stage.

[0062] Optionally, the video generation module includes:

[0063] The information acquisition submodule is used to acquire the structured information of the images in the original video;

[0064] The 3D information generation submodule is used to reconstruct the model from the original scene and generate 3D scene information.

[0065] The rendering submodule is used to render the 3D scene information through projection transformation based on the new trajectory and the structured information as needed, so as to obtain the generated video of the new trajectory.

[0066] The present invention also provides a video generation apparatus, the apparatus comprising:

[0067] The trajectory input module is used to input the target trajectory into the trained enhanced scene reconstruction model as described above;

[0068] The video generation module is used to generate a video of the target trajectory based on the enhanced scene reconstruction model.

[0069] This invention also discloses an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0070] Memory, used to store computer programs;

[0071] The processor, when executing a program stored in memory, implements the method described above.

[0072] This invention also discloses a readable storage medium that, when the instructions in the storage medium are executed by the processor of an electronic device, enables the electronic device to perform the method described above.

[0073] According to an embodiment of the present invention, a generated video of a new trajectory is first generated using an original scene reconstruction model. This original scene reconstruction model is trained based on the original video, while the new trajectory refers to a trajectory different from the camera trajectory in the original video. Next, an image inpainting model is used to inpaint the images in the generated video of the new trajectory, resulting in a repaired video of the new trajectory. Then, the original scene reconstruction model is trained using the repaired video of the new trajectory, ultimately yielding an enhanced scene reconstruction model. Through this process, the repaired video of the new trajectory is used as expanded training data to retrain the scene reconstruction model. This allows the model to learn not only the original video corresponding to the original trajectory but also training data outside the original trajectory, thereby enhancing the scene reconstruction capability of the scene reconstruction model. Specifically, the enhanced scene reconstruction model can effectively reduce the gap between the generated video and the original video and significantly reduce artifacts and ghosting phenomena in the generated video, thus improving the overall quality of scene reconstruction. Furthermore, the model also exhibits better performance when rendering larger operations. Attached Figure Description

[0074] Figure 1 A flowchart illustrating the steps of an enhancement method for a scene reconstruction model provided in an embodiment of the present invention is shown.

[0075] Figure 2 A schematic diagram of the overall framework of ReconDreamer is shown;

[0076] Figure 3 A schematic diagram of the training process for DriveRestorer is shown;

[0077] Figure 4 A schematic diagram of repaired data pairs used for training DriveRestorer is shown;

[0078] Figure 5 A schematic diagram is shown of a degraded video frame rendered under the new trajectory and an example of the frame repaired by DriveRestorer;

[0079] Figure 6 A flowchart illustrating the steps of a video generation method according to an embodiment of the present invention is shown.

[0080] Figure 7 The diagram shows a structural block diagram of an embodiment of a scene reconstruction model enhancement device provided by an embodiment of the present invention;

[0081] Figure 8 This diagram illustrates a structural block diagram of a video generation apparatus embodiment provided by one embodiment of the present invention;

[0082] Figure 9 A structural block diagram of an enhanced electronic device for scene reconstruction model is shown according to an exemplary embodiment. Detailed Implementation

[0083] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0084] The scene reconstruction model enhancement method in this embodiment of the invention can be executed by electronic devices such as smart vehicle systems and mobile terminals. For example, the scene reconstruction model enhancement method can be applied to smart vehicle systems or any other applicable hardware and software. This embodiment of the invention does not limit this.

[0085] Reference Figure 1 The diagram illustrates a flowchart of a method for enhancing a scene reconstruction model according to an embodiment of the present invention, which may specifically include the following steps:

[0086] Step 101: Use the original scene reconstruction model to generate a video of the new trajectory; wherein the original scene reconstruction model is trained using the original video, the new trajectory is a trajectory different from the original trajectory, and the original trajectory is the camera trajectory of the original video.

[0087] In this embodiment of the invention, the scene reconstruction model is a computational model used to reconstruct a three-dimensional scene from two-dimensional images, videos, or other sensor data. Its goal is to generate a three-dimensional representation of the scene by analyzing and processing input data, thereby enabling the re-rendering of high-quality images or videos at different viewpoints or time points. Scene reconstruction models are widely used in fields such as autonomous driving, virtual reality, augmented reality, robot navigation, and computer vision.

[0088] In this embodiment of the invention, the original scene reconstruction model is trained using the original video, and its main purpose is to generate the corresponding scene view based on the input trajectory information (such as camera path or vehicle trajectory). However, due to factors such as data sparsity, model complexity, or viewpoint changes, the original scene reconstruction model may produce artifacts (such as ghosting, speckles, etc.) when generating a new trajectory view, affecting the reconstruction quality.

[0089] For example, Neural Radiance Field (NeRF) and 3D Gaussian Splatting (3DGS) have become prominent techniques in scene reconstruction. The NeRF model uses a multilayer perceptron (MLP) network to represent continuous volumetric scenes, achieving superior rendering quality. 3DGS introduces a novel approach by defining anisotropic Gaussians in 3D space and employing adaptive density control, enabling high-quality rendering even from sparse point cloud data. Many studies have adapted NeRF and 3DGS for driving scene reconstruction. To accommodate the dynamics of the driving environment, some methods incorporate time as an additional parameter to capture temporal variations in dynamic scenes, while others treat the scene as a combination of moving objects superimposed on a static background. Despite these advances, NeRF and 3DGS-based methods still face challenges related to data density. Their effectiveness in rendering heavily relies on the close alignment of sensor trajectories with the training distribution. To address these challenges, methods such as SGD (Stochastic Gradient Descent), GGS (Generalizable Gaussian Splatting), and MagicDrive3D (a 3D-based method for autonomous driving scene generation) leverage generative models to expand the diversity of training perspectives. However, these methods primarily focus on sparse image data or static background components, making them insufficient to fully capture the complexity of dynamic driving environments.

[0090] For example, world models predict possible future world states based on hypothetical action sequences proposed by actors. Based on world models, recent methods have advanced environmental simulation techniques by generating videos guided by free-text actions. A leading technique in this field is Sora (a generative model), which uses cutting-edge generative methods to create complex visual sequences that conform to the physical laws of the environment. This capability not only improves the realism of generated video content but also has significant application potential in real-world driving scenarios. In autonomous driving, world models utilize predictive techniques to interpret the driving environment. These methods generate realistic driving scenarios while extracting driving strategies from video data, reigniting the potential for more robust closed-loop simulations. The recent DriveDreamer4D (a four-dimensional-based method for autonomous driving scene generation) further demonstrates that utilizing pre-trained world models as data machines can improve dynamic driving scene reconstruction. However, it still faces challenges when performing larger operations (such as multi-lane changes).

[0091] In this embodiment of the invention, the original video refers to the foundational video data used to train and evaluate the scene reconstruction model. It records the true appearance of a specific scene under the original trajectory and is key input data in the scene reconstruction task. The original video is a sequence of videos taken from a specific perspective (usually the vehicle's camera perspective), recording the dynamic changes in the driving scene. It contains all visual information in the scene, such as roads, vehicles, pedestrians, buildings, etc.

[0092] In this embodiment of the invention, the original trajectory is the camera trajectory of the original video, which is the recorded actual movement path of the camera or vehicle. For example, in an autonomous driving scenario, the original trajectory can be the actual path of the vehicle traveling on the road, including the vehicle's movement trajectory and the camera's perspective changes; essentially, it is the camera's trajectory.

[0093] In this embodiment of the invention, a new trajectory refers to a trajectory different from the original trajectory, used to generate a new perspective or a new motion path. The new trajectory can be a path obtained through simulation or design, or it can be a path actually recorded but different from the original trajectory. Specifically, it can include any applicable new trajectory, and this embodiment of the invention does not impose any limitations on this.

[0094] In this embodiment of the invention, the original scene reconstruction model is a trained model capable of generating a corresponding scene view sequence, i.e., generating a video, based on the input new trajectory. Specifically:

[0095] First, the new trajectory is passed as input to the original scene reconstruction model. For example, the model needs the following information: Trajectory parameters: including the trajectory's start point, end point, midpoint, and shape (e.g., straight line, curve). Temporal information: if the trajectory is dynamic, temporal parameters are also required so the model can generate consecutive video frames. Viewpoint information: including camera orientation, focal length, and other parameters, which determine the viewpoint and viewpoint changes in the generated video.

[0096] Secondly, the original scene reconstruction model generates corresponding image or video frames based on the input new trajectory. For example, the model first generates a 3D representation of the scene based on the new trajectory. Then, based on the viewpoint information of the new trajectory, the model projects the 3D representation onto a 2D plane to generate the corresponding image or video frame.

[0097] Finally, the images or video frames generated by the model are combined into a complete video sequence, i.e., the video of the generated new trajectory.

[0098] For example, first use the original data V ori (i.e., original video) Training scene reconstruction method (i.e., the scene reconstruction model). Then, the trained method renders the generated video of the new trajectory.

[0099]

[0100] in It's a new trajectory.

[0101] It is worth noting that, due to data sparsity, the rendering... Artifacts (such as ghosting) will appear.

[0102] In one optional embodiment of the present invention, a specific implementation of generating a video of a new trajectory using an original scene reconstruction model may include: obtaining structured information of images in the original video; generating three-dimensional scene information using the original scene reconstruction model; and rendering the three-dimensional scene information through projection transformation according to the required new trajectory and the structured information to obtain the generated video of the new trajectory.

[0103] Structured information refers to the geometric and semantic information about a scene that corresponds to the images in the original video. For example, 3D bounding boxes represent the position, size, and orientation of objects (such as vehicles, pedestrians, etc.) in a scene, while high-definition maps (HDMaps) provide detailed geographic information and dynamic data. Specifically, it can include any applicable structured information, and this embodiment of the invention does not impose any limitations on it.

[0104] A native scene reconstruction model is a trained model capable of generating a 3D representation of a scene from an input native video. Images from the native video are input into the native scene reconstruction model. The model then generates a 3D representation of the scene based on the input images. For example, a NeRF model can be used to generate a volumetric representation of the scene, or 3D Gaussian rendering (3DGS) can be used to generate a point cloud.

[0105] Projection transformation is the process of converting 3D scene information from the world coordinate system to the image coordinate system. This step ensures that the generated image can be correctly rendered from the perspective of the new trajectory. The new trajectory and structured information are input into the rendering module. Based on the perspective parameters of the new trajectory, the 3D scene information is projected onto the 2D image plane. For example... Figure 2 As shown, the viewpoint parameters of the new trajectory are obtained using progressive trajectory sampling, and 3D scene information such as 3D bounding boxes and HD maps are projected onto the 2D image plane. It is important to note that this structured information is transformed and integrated with the new trajectory. Alignment. An image is generated based on the projected 3D points. Structured information is used to ensure the spatiotemporal consistency of traffic elements, thereby improving the quality of scene reconstruction.

[0106] Step 102: Using an image restoration model, the images in the generated video of the new trajectory are restored to obtain the restored video of the new trajectory.

[0107] In this embodiment of the invention, the image restoration model is a computational model for improving image quality. It restores the original content of an image by removing noise, artifacts, blur, or other distortions.

[0108] In this embodiment of the invention, the image inpainting model can repair various problems (such as artifacts) in the images of the generated video of the new trajectory. By inputting the images from the generated video of the new trajectory into the image inpainting model, repaired images can be obtained. The repaired images are then combined into a video, which is denoted as the repaired video of the new trajectory.

[0109] In this embodiment of the invention, the image restoration model can be a pre-trained model, and any applicable training method can be used. This embodiment of the invention does not limit this.

[0110] For example, traditional scene reconstruction methods face challenges due to the sparsity of training data. Recent methods alleviate this problem by leveraging generative priors to increase data density. However, a gap still exists between generated and real data. In contrast, this invention proposes ReconDreamer (the proposed scene reconstruction model) to expand the training data through an online inpainting process. DriveRestorer (the proposed image inpainting model) is introduced to repair these artifacts. The inpainting process is similar to a diffusion denoising process. Therefore, the video V is repaired.novel With the original video V ori The gap is smaller, making it easier to repair video V novel More suitable as training samples for scene reconstruction.

[0111] In an optional embodiment of the present invention, before generating a new trajectory video using the original scene reconstruction model, the method may further include: generating a new trajectory video using the original scene reconstruction model; determining image pair training data based on the generated trajectory video and the original video, wherein the images in the generated trajectory video are used as input images in the image pair, and the images in the original video are used as ground truth images in the image pair; and training the image restoration model using the image pair training data.

[0112] The original trajectory is passed as input to the original scene reconstruction model. The original scene reconstruction model generates corresponding image or video frames based on the input original trajectory. The images or video frames generated by the model are combined into a complete video sequence, i.e., the generated video of the original trajectory.

[0113] Due to data sparsity, images in the generated video of the original trajectory from the original scene reconstruction model may exhibit artifacts and other defects. Any image from the generated video of the original trajectory and its corresponding image from the original video are treated as an image pair. The image from the generated video serves as the input image for the image pair, and the image from the original video serves as the ground truth image. Many such image pairs constitute the image pair training data.

[0114] An image restoration model is trained using image pairs as training data. During training, images from the generated video of the original trajectory are used as input images, and images from the original video are used as ground truth images.

[0115] In one specific implementation, the image inpainting model is a diffusion denoising model. Images from the original trajectory generation video are encoded and then input into a reference network to generate reference feature data, which is then input into a denoising network. Images from the original video are encoded and then input into the denoising network. The denoising network fuses the reference feature data and the feature data from the original video images to generate denoised feature data. This denoised feature data is then decoded to obtain the restored image. Finally, the difference between the restored image and the image in the original video is forward-propagated to train the image inpainting model. This process is iterated continuously until a well-trained image inpainting model is obtained.

[0116] For example, traditional scene reconstruction methods often exhibit artifacts when rendering new trajectory views (i.e., images from the generated video). To address this issue, this invention introduces an image inpainting model (DriveRestorer) to repair these degraded renderings. Figure 2 As shown, during the training process of dynamic driving scene reconstruction, the viewpoint of the new trajectory is first rendered. These rendered videos are then processed by an image inpainting model (DriveRestorer) to restore their quality. These restored videos (i.e., the inpainted videos) are then used together with the original videos to optimize the original scene reconstruction model. This iterative process continues until the original scene reconstruction model converges (the training process of the image inpainting model (DriveRestorer) is omitted in this figure).

[0117] Specifically Figure 2 The online restoration process includes: encoding the video of the generated new trajectory, encoding the sequence of 3D bounding boxes, and encoding the sequence of HD Maps. The encoded sequence data and the sequence data of the noisy image are combined and then input into the image restoration model. After passing through the image restoration model, denoised feature data is obtained. The denoised feature data is then used for video decoding to obtain the restored video of the new trajectory.

[0118] Next, we will go into detail about the training and inference of the image restoration model (DriveRestorer).

[0119] Training. A major challenge in training the image inpainting model (DriveRestorer) is the lack of rendered inpainting datasets. Therefore, a novel method for constructing inpainting pairs is proposed. For example... Figure 3 As shown, the principle of constructing the repair dataset is as follows: using insufficiently trained Gaussian points (i.e., an undertrained original scene reconstruction model) to render the video along the original trajectory. The generated video of the original trajectory is obtained, but due to insufficient model fitting, ghosting artifacts will naturally occur. Then, these degraded frames are compared with their corresponding ground truth video V. ori (i.e., the original video) are paired to form a rendering and restoration dataset (i.e., image pairs training data). Figure 4 Visualizations of images from the generated video and the original video (i.e., image pairs) are provided. Based on the constructed dataset, a supervised training model (DriveRestorer) is performed to repair artifacts in the generated video. Along... Figure 3Using mid-Gaussian points to draw the arrow directions during the training phase, and observing the generated video images, it can be seen that as the original scene reconstruction model is trained, artifacts in the generated video gradually decrease. The image inpainting model (DriveRestorer) is fine-tuned based on the world model. Specifically, it re-degrades video frames... Introduced as a control condition to provide appearance prior.

[0120] Inference. After training the image inpainting model (DriveRestorer), its parameters are frozen to inpaint the new trajectory rendering (i.e., the video generated by the new trajectory):

[0121]

[0122] in, It is an image restoration model (DriveRestorer). This is a video showing the generation of the new trajectory. These are structural conditions (3D frame and HDMap). It's a new trajectory. This represents a projection transformation, used to compare structural conditions with... Alignment.

[0123] By using generated videos of the original trajectories and images composed of the original videos as training data, the problem of insufficient training data for image inpainting models is solved. An image inpainting model that is more suitable for inpainting generated videos output by scene reconstruction models is obtained, resulting in better image inpainting effects and improved quality of inpainted videos with new trajectories. This, in turn, provides better assistance for training the original scene reconstruction model and improves the quality of scene reconstruction.

[0124] In an optional embodiment of the present invention, before training the image restoration model using the image pair training data, the method may further include: acquiring ground truth video frames; randomly generating masked video frames based on the ground truth video frames; and training the pedestal image restoration model using the masked video frames and the ground truth video frames. Correspondingly, a specific implementation of training the image restoration model using the image pair training data may include: training the pedestal image restoration model using the image pair training data to obtain the image restoration model.

[0125] To train the image inpainting model, an additional step was introduced to enhance its robustness and inpainting capabilities. This step involved acquiring ground truth video frames, randomly generating masked video frames, and using this data to train a pedestal image inpainting model.

[0126] The ground truth video frames can be high-quality, artifact-free image sequences directly obtained from the original scene. Masked video frames are generated by randomly applying a mask to the ground truth video frames. The mask simulates artifacts or noise in the image, thus providing the model with degraded image samples. By randomly generating masks, the model can learn various degradation patterns, thereby improving its generalization ability.

[0127] The pedestal image inpainting model is a fundamental image inpainting model used to learn the mapping from masked video frames to ground truth video frames. Masked video frames are paired with their corresponding ground truth video frames to form a training dataset. An appropriate loss function is chosen to evaluate the inpainting performance. An optimization algorithm is used to train the pedestal image inpainting model to minimize the loss function. The training objective is to enable the model to recover images from masked video frames that closely approximate the ground truth video frames.

[0128] For example, such as Figure 3 As shown, the mask sequence is a sequence of masked video frames. The pedestal image restoration model is trained using the sequence of masked video frames and the original video.

[0129] After training the pedestal image restoration model, the image restoration model is further trained using images as training data. The training method is basically the same as that used for training the pedestal image restoration model. The image restoration model is further optimized based on the pedestal model, enabling it to handle more complex degradation patterns and improve restoration results.

[0130] By using image pairs as training data, the model can learn a mapping from degraded images to high-quality images, thus performing well in real-world applications.

[0131] For example, to further emphasize the restoration of challenging regions, degraded video frames are processed during training. Apply a mask. Since video quality degrades in distant areas far from the camera center and at the boundaries of the sky and scene, the mask M is primarily focused on these problematic areas. The mask is then multiplied pixel-by-pixel by the ground truth video frame to generate the masked video frame. The formula is as follows:

[0132]

[0133] in, These are true video frames, and M is the mask. It is a masked video frame.

[0134] Then, the masked video frames are input into encoder ε to obtain low-dimensional latent features. The fine-tuning process of the world model is optimized using diffusion loss:

[0135]

[0136] Where, ∈ t Denotes random noise at time step t, ∈ θ It is a parameterized denoising network, z t This refers to the latent noise variable at time step t, where c represents the control conditions, including degraded video. 3D frame and high-definition map. The integration of video, 3D frame and high-definition map can be carried out in any applicable manner, and the embodiments of the present invention do not impose any restrictions on this.

[0137] like Figure 5 As shown, by comparing the images in the generated video of the new trajectory and the images in the restored video of the new trajectory, it can be found that the trained image restoration model (DriveRestorer) can reduce ghosting artifacts in the rendering of the new trajectory.

[0138] Step 103: Using the repaired video of the new trajectory, train the original scene reconstruction model to obtain an enhanced scene reconstruction model.

[0139] In this embodiment of the invention, the enhanced scene reconstruction model is obtained by training the original scene reconstruction model using repaired videos with new trajectories. This process not only expands the training dataset but also optimizes the model's performance using the repaired, high-quality videos.

[0140] In this embodiment of the invention, the repaired video of the new trajectory is input into the original scene reconstruction model to generate a reconstructed scene representation (such as an image, video, or point cloud). A predefined loss function is used to calculate the difference between the reconstruction result and the real data. The gradient of the loss function with respect to the model parameters is calculated through backpropagation. The model parameters are updated using an optimization algorithm (such as gradient descent) to minimize the loss function. When the model converges or reaches a preset number of training iterations, training is stopped, resulting in an enhanced scene reconstruction model.

[0141] In one optional embodiment of the present invention, a specific implementation of training the original scene reconstruction model using the repaired video of the new trajectory to obtain an enhanced scene reconstruction model may include: training the original scene reconstruction model using the original video and the repaired video of the new trajectory to obtain an enhanced scene reconstruction model.

[0142] The original video and the repaired video with the new trajectory are mixed to construct a hybrid training dataset. This hybrid training dataset is then used to train the original scene reconstruction model, resulting in an enhanced scene reconstruction model. Specific training methods can be found in the training methods described above; this embodiment of the invention does not limit these methods.

[0143] By mixing the original video with the restored video of the new trajectory, the model can learn from more diverse data, taking into account both the original and restored videos to reconstruct the scene and further improve the quality of scene reconstruction.

[0144] According to an embodiment of the present invention, a generated video of a new trajectory is first generated using an original scene reconstruction model. This original scene reconstruction model is trained based on the original video, while the new trajectory refers to a trajectory different from the camera trajectory in the original video. Next, an image inpainting model is used to inpaint the images in the generated video of the new trajectory, resulting in a repaired video of the new trajectory. Then, the original scene reconstruction model is trained using the repaired video of the new trajectory, ultimately yielding an enhanced scene reconstruction model. Through this process, the repaired video of the new trajectory is used as expanded training data to retrain the scene reconstruction model. This allows the model to learn not only the original video corresponding to the original trajectory but also training data beyond the original trajectory, thereby enhancing the scene reconstruction capability of the scene reconstruction model. Specifically, the enhanced scene reconstruction model can effectively reduce the gap between the generated video and the original video and significantly reduce artifacts (such as ghosting, speckles, etc.) appearing in the generated video, thus improving the overall quality of scene reconstruction. Furthermore, this model also exhibits better performance when rendering complex scenes (such as multi-lane transitions or large viewpoint shifts).

[0145] In an optional embodiment of the present invention, the new trajectory is the first sub-trajectory in a new complete trajectory, and the new complete trajectory includes m consecutively adjacent sub-trajectories. A specific implementation of training the original scene reconstruction model using the repaired video of the new trajectory to obtain an enhanced scene reconstruction model may include: training the original scene reconstruction model using the repaired video of the first sub-trajectory to obtain a scene reconstruction model for a first training stage. Then, it may further include: using the scene reconstruction model for the nth training stage to generate a video of the (n+1)th sub-trajectory in the new complete trajectory, where n is a positive integer from 1 to (m-1); using the image repair model to repair the image in the video of the (n+1)th sub-trajectory to obtain a repaired video of the (n+1)th sub-trajectory; using the repaired video of the (n+1)th sub-trajectory to train the scene reconstruction model for the nth training stage to obtain a scene reconstruction model for the (n+1)th training stage; until n reaches m-1, the scene reconstruction model for the mth training stage is used as the final enhanced scene reconstruction model.

[0146] A new complete trajectory can be the complete motion path of a camera or vehicle from a starting point to an end point in a dynamic driving scenario. It is a continuous trajectory that covers the entire motion process of the scene. New complete trajectories are often used to evaluate the performance of scene reconstruction models under unseen viewpoints or paths.

[0147] A sub-trajectory is a portion of a new complete trajectory, representing a segment of the trajectory. Sub-trajectories can be used for progressive training and optimization of scene reconstruction models to improve model performance.

[0148] To obtain an enhanced scene reconstruction model, this invention employs a progressive data update strategy. First, the original scene reconstruction model is used to generate a video of the first sub-trajectory in the new complete trajectory. This generated video may contain artifacts or noise, so it needs to be repaired. Specifically, an image inpainting model is used to repair the images in the generated video of the first sub-trajectory, resulting in a repaired video, i.e., the repaired video of the first sub-trajectory. Then, this repaired video is used to train the original scene reconstruction model, resulting in the scene reconstruction model after the first training iteration, which is the scene reconstruction model in the first training stage.

[0149] Next, we proceed with the iterative training process. Using the scene reconstruction model obtained from the first training stage, we generate a video of the second sub-trajectory in the new complete trajectory. Similarly, we use an image inpainting model to inpaint the images in the generated video of the second sub-trajectory, obtaining the inpainted video of the second sub-trajectory. We then use the inpainted video of the second sub-trajectory to train the scene reconstruction model from the first training stage, obtaining the scene reconstruction model from the second training stage.

[0150] Following the method described above, the scene reconstruction model from the nth training stage is used sequentially to generate videos of the (n+1)th sub-trajectory in the new complete trajectory, where n is a positive integer from 1 to m-1. Then, an image inpainting model is used to inpaint the images in the (n+1)th sub-trajectory's generated video, resulting in the (n+1)th sub-trajectory's inpainted video. Next, the scene reconstruction model from the nth training stage is trained using the (n+1)th sub-trajectory's inpainted video, resulting in the (n+1)th training stage's scene reconstruction model. This process continues until n gradually increases from 1 to m-1.

[0151] Finally, when n reaches m-1, the scene reconstruction model of the m-th training stage is obtained, which is the final enhanced scene reconstruction model. Through this progressive training method, the model can be optimized using the repaired video of the sub-trajectory at each training stage, thereby gradually improving the reconstruction quality of the new complete trajectory. The final enhanced scene reconstruction model can better handle complex scenes, reduce artifacts and noise, improve the accuracy and reliability of reconstruction results, and ensure high-quality rendering of complex scenes (such as multi-lane changes or large viewpoint shifts).

[0152] For example, based on DriveRestorer's ability to reconstruct generated videos of new trajectories, a Progressive Data Update Strategy (PDUS) is proposed to enhance driving scene reconstruction under high-operational-conditions. The Progressive Data Update Strategy first constructs a hybrid dataset D = 0.5D. ori ∪0.5D novel , where D ori It is the dataset of the original videos, D novel This refers to a dataset of repaired videos with restored trajectories that can be updated during training. The update strategy uses an update distance of y = kΔy meters to progressively update the new trajectory in the k-th update step. Then, the scene reconstruction model renders the generated video of the new trajectory. These videos are then processed by DriveRestorer to obtain a restored video of the new track. novel To ensure that the newly generated repaired videos provide additional prior knowledge for the scene reconstruction model, the updated dataset D... novel It can be obtained in the following ways:

[0153] D novel = (1-w)·D novel ∪w·V novel ,

[0154] in, It is V novel The sampling probability. For the driving scene reconstruction method based on Gaussian sputtering, the Gaussian parameters are trained using the original video dataset:

[0155]

[0156] in, This is a reconstruction loss commonly used in Gaussian sputtering optimization, where λ1, λ2, and λ3 are the loss weights. Furthermore, for the generated video of the new trajectory, a depthless training strategy is used to train ReconDreamer.

[0157]

[0158] The overall loss function for hybrid training is:

[0159]

[0160] In one optional embodiment of the present invention, a specific implementation of training the scene reconstruction model of the nth training stage using the repaired video of the (n+1)th sub-trajectory to obtain the scene reconstruction model of the (n+1)th training stage may include: training the scene reconstruction model of the nth training stage using the original video and the repaired video of the (n+1)th sub-trajectory to obtain the scene reconstruction model of the (n+1)th training stage.

[0161] The original video and the repaired video of the (n+1)th sub-trajectory are mixed to construct a hybrid training dataset. This hybrid training dataset is then used to train the original scene reconstruction model, resulting in an enhanced scene reconstruction model. Specific training methods can be found in the training methods described above; this embodiment of the invention does not limit these methods.

[0162] For example, such as Figure 2 The dataset update for the online repair process shown can employ a progressive data update strategy. Through progressive trajectory sampling, each time a repair video of a sub-trajectory is rendered using the online repair tool, a dataset is constructed consisting of the original video and a repair video with a lane offset of 1.5 meters. This dataset is then used to train the scene reconstruction model. Next, a dataset consisting of the original video and a repair video with a lane offset of 3 meters is constructed, and the scene reconstruction model is trained again. Weighted data samples can also be added during training. This process is repeated until the scene reconstruction model for dynamic scene reconstruction is trained.

[0163] By mixing the original video and the repaired video of the (n+1)th sub-trajectory, the model can learn from more diverse data, taking into account both the original and repaired videos to reconstruct the scene and further improve the quality of scene reconstruction.

[0164] Reference Figure 6 The diagram illustrates a flowchart of a video generation method provided by an embodiment of the present invention, which may specifically include the following steps:

[0165] Step 201: Input the target trajectory into the trained enhanced scene reconstruction model as described above.

[0166] In this embodiment of the invention, after obtaining the enhanced scene reconstruction model using the above training method, the target trajectory is passed as input to the enhanced scene reconstruction model when using it.

[0167] Step 202: Based on the target trajectory, generate a video of the target trajectory using the enhanced scene reconstruction model.

[0168] In this embodiment of the invention, the enhanced scene reconstruction model generates corresponding image or video frames based on the input target trajectory. For example, the enhanced scene reconstruction model first generates a three-dimensional representation of the scene based on the target trajectory. Then, based on the viewpoint information of the target trajectory, the enhanced scene reconstruction model projects the three-dimensional representation onto a two-dimensional plane to generate corresponding image or video frames. Finally, the image or video frames generated by the enhanced scene reconstruction model are combined into a complete video sequence, i.e., the generated video of the target trajectory.

[0169] According to an embodiment of the present invention, the target trajectory is input into the pre-trained enhanced scene reconstruction model as described above. Based on the target trajectory, the enhanced scene reconstruction model generates a video of the target trajectory. Through this process, the repaired video of the new trajectory is used as extended training data to retrain the scene reconstruction model. This allows the model to learn not only the original video corresponding to the original trajectory, but also training data outside the original trajectory, thereby enhancing the scene reconstruction capability of the scene reconstruction model. Specifically, the enhanced scene reconstruction model can effectively reduce the gap between the generated video and the original video, and significantly reduce artifacts and ghosting phenomena in the generated video, thereby improving the overall quality of scene reconstruction. In addition, the model also shows better performance when rendering larger operations.

[0170] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0171] Reference Figure 7 The diagram illustrates a structural block diagram of an embodiment of a scene reconstruction model enhancement device provided by another embodiment of the present invention, which may specifically include the following modules:

[0172] The video generation module 301 is used to generate a new trajectory video using the original scene reconstruction model; wherein the original scene reconstruction model is trained using the original video, the new trajectory is a trajectory different from the original trajectory, and the original trajectory is the camera trajectory of the original video;

[0173] The video restoration module 302 is used to restore the images in the generated video of the new trajectory using an image restoration model, so as to obtain the restored video of the new trajectory.

[0174] The model enhancement module 303 is used to train the original scene reconstruction model using the repaired video of the new trajectory to obtain an enhanced scene reconstruction model.

[0175] Optionally, the device further includes:

[0176] The original trajectory video generation module is used to generate the original trajectory video using the original scene reconstruction model before generating the new trajectory video using the original scene reconstruction model.

[0177] The image pair determination module is used to determine image pair training data based on the generated video of the original trajectory and the original video, wherein the images in the generated video of the original trajectory are used as input images in the image pair, and the images in the original video are used as ground truth images in the image pair.

[0178] The image restoration model training module is used to train the image restoration model using the image pairs as training data.

[0179] Optionally, the device further includes:

[0180] The truth value acquisition module is used to acquire truth video frames before the image restoration model is trained using the image pair training data;

[0181] The mask generation module is used to randomly generate masked video frames based on the true video frames;

[0182] The base training module is used to train a base image restoration model using the masked video frame and the ground truth video frame.

[0183] The image restoration model training module includes:

[0184] The image restoration model training submodule is used to train the base image restoration model using the image as training data to obtain the image restoration model.

[0185] Optionally, the model enhancement module includes:

[0186] The model enhancement submodule is used to train the original scene reconstruction model using the original video and the repaired video of the new trajectory to obtain an enhanced scene reconstruction model.

[0187] Optionally, the new trajectory is the first sub-trajectory in the new complete trajectory, and the new complete trajectory includes m consecutively adjacent sub-trajectories. The model enhancement module includes:

[0188] The first model enhancement submodule is used to train the original scene reconstruction model using the repaired video of the first sub-trajectory to obtain the scene reconstruction model in the first training stage.

[0189] The device further includes:

[0190] The nth video generation module is used to generate a video of the (n+1)th sub-trajectory in the new complete trajectory using the scene reconstruction model of the nth training stage, where n is a positive integer from 1 to (m-1).

[0191] The repair module is used to repair the image in the generated video of the (n+1)th sub-trajectory using the image repair model, so as to obtain the repaired video of the (n+1)th sub-trajectory.

[0192] The training module is used to train the scene reconstruction model of the nth training stage using the repaired video of the (n+1)th sub-trajectory, so as to obtain the scene reconstruction model of the (n+1)th training stage.

[0193] The model determination module is used to take the scene reconstruction model of the m-th training stage as the final enhanced scene reconstruction model until n reaches m-1 from 1.

[0194] Optionally, the training module includes:

[0195] The training submodule is used to train the scene reconstruction model of the nth training stage using the original video and the repaired video of the (n+1)th sub-trajectory, so as to obtain the scene reconstruction model of the (n+1)th training stage.

[0196] Optionally, the video generation module includes:

[0197] The information acquisition submodule is used to acquire the structured information of the images in the original video;

[0198] The 3D information generation submodule is used to reconstruct the model from the original scene and generate 3D scene information.

[0199] The rendering submodule is used to render the 3D scene information through projection transformation based on the new trajectory and the structured information as needed, so as to obtain the generated video of the new trajectory.

[0200] According to an embodiment of the present invention, a generated video of a new trajectory is first generated using an original scene reconstruction model. This original scene reconstruction model is trained based on the original video, while the new trajectory refers to a trajectory different from the camera trajectory in the original video. Next, an image inpainting model is used to inpaint the images in the generated video of the new trajectory, resulting in a repaired video of the new trajectory. Then, the original scene reconstruction model is trained using the repaired video of the new trajectory, ultimately yielding an enhanced scene reconstruction model. Through this process, the repaired video of the new trajectory is used as expanded training data to retrain the scene reconstruction model. This allows the model to learn not only the original video corresponding to the original trajectory but also training data beyond the original trajectory, thereby enhancing the scene reconstruction capability of the scene reconstruction model. Specifically, the enhanced scene reconstruction model can effectively reduce the gap between the generated video and the original video and significantly reduce artifacts (such as ghosting, speckles, etc.) appearing in the generated video, thus improving the overall quality of scene reconstruction. Furthermore, this model also exhibits better performance when rendering complex scenes (such as multi-lane transitions or large viewpoint shifts).

[0201] Reference Figure 8 The diagram illustrates a structural block diagram of a video generation apparatus embodiment provided by another embodiment of the present invention, which may specifically include the following modules:

[0202] The trajectory input module 401 is used to input the target trajectory into the trained enhanced scene reconstruction model as described above;

[0203] The video generation module 402 is used to generate a video of the target trajectory based on the enhanced scene reconstruction model.

[0204] According to an embodiment of the present invention, the target trajectory is input into the pre-trained enhanced scene reconstruction model as described above. Based on the target trajectory, the enhanced scene reconstruction model generates a video of the target trajectory. Through this process, the repaired video of the new trajectory is used as extended training data to retrain the scene reconstruction model. This allows the model to learn not only the original video corresponding to the original trajectory, but also training data outside the original trajectory, thereby enhancing the scene reconstruction capability of the scene reconstruction model. Specifically, the enhanced scene reconstruction model can effectively reduce the gap between the generated video and the original video, and significantly reduce artifacts and ghosting phenomena in the generated video, thereby improving the overall quality of scene reconstruction. In addition, the model also shows better performance when rendering larger operations.

[0205] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0206] Figure 9This is a structural block diagram illustrating an enhanced electronic device 700 for scene reconstruction modeling, according to an exemplary embodiment. For example, the electronic device 700 may be an in-vehicle computer, a computer, a digital broadcasting terminal, a messaging device, a game console, a medical device, a fitness device, etc.

[0207] Reference Figure 9 The electronic device 700 may include one or more of the following components: a processing component 702, a memory 704, a power supply component 706, a multimedia component 708, an audio component 710, an input / output (I / O) interface 712, a sensor component 714, and a communication component 716.

[0208] Processing component 702 typically controls the overall operation of electronic device 700, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 702 may include one or more processors 720 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 702 may include one or more modules to facilitate interaction between processing component 702 and other components. For example, processing component 702 may include a multimedia module to facilitate interaction between multimedia component 708 and processing component 702.

[0209] Memory 704 is configured to store various types of data to support the operation of device 700. Examples of this data include instructions for any application or method operating on electronic device 700, contact data, phonebook data, messages, pictures, videos, etc. Memory 704 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0210] Power supply component 706 provides power to various components of electronic device 700. Power supply component 706 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 700.

[0211] Multimedia component 708 includes a screen that provides an output interface between the electronic device 700 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 708 includes a front-facing camera and / or a rear-facing camera. When the electronic device 700 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0212] Audio component 710 is configured to output and / or input audio signals. For example, audio component 710 includes a microphone (MIC) configured to receive external audio signals when electronic device 700 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 704 or transmitted via communication component 716. In some embodiments, audio component 710 also includes a speaker for outputting audio signals.

[0213] I / O interface 712 provides an interface between processing component 702 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0214] Sensor assembly 714 includes one or more sensors for providing state assessments of various aspects of electronic device 700. For example, sensor assembly 714 may detect the on / off state of device 700, the relative positioning of components such as the display and keypad of electronic device 700, changes in position of electronic device 700 or a component of electronic device 700, the presence or absence of user contact with electronic device 700, orientation or acceleration / deceleration of electronic device 700, and temperature changes of electronic device 700. Sensor assembly 714 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 714 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 714 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0215] Communication component 716 is configured to facilitate wired or wireless communication between electronic device 700 and other devices. Electronic device 700 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 716 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 716 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0216] In an exemplary embodiment, the electronic device 700 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0217] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 704 including instructions, which can be executed by a processor 720 of an electronic device 700 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0218] A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a terminal's processor, enables the terminal to perform an enhancement method for a scene reconstruction model, the method comprising:

[0219] A new trajectory is generated by using the original scene reconstruction model; wherein the original scene reconstruction model is trained using the original video, and the new trajectory is a trajectory different from the original trajectory, and the original trajectory is the camera trajectory of the original video;

[0220] An image restoration model is used to restore the images in the generated video of the new trajectory, resulting in a restored video of the new trajectory.

[0221] The original scene reconstruction model is trained using the repaired video with the new trajectory to obtain an enhanced scene reconstruction model.

[0222] Optionally, before generating a new trajectory video by reconstructing the model using the original scene, the method further includes:

[0223] Using the original scene reconstruction model, a generated video of the original trajectory is generated;

[0224] Based on the generated video of the original trajectory and the original video, image pair training data is determined, wherein the image in the generated video of the original trajectory is used as the input image in the image pair, and the image in the original video is used as the ground truth image in the image pair.

[0225] The image restoration model is trained using the image pairs as training data.

[0226] Optionally, before training the image restoration model using the image pairs as training data, the method further includes:

[0227] Obtain the true video frame;

[0228] Based on the true video frame, a masked video frame is randomly generated;

[0229] The pedestal image restoration model is trained using the masked video frames and the ground truth video frames.

[0230] The step of training the image restoration model using the image pairs as training data includes:

[0231] The image pairs are used as training data to train the base image restoration model, thus obtaining the image restoration model.

[0232] Optionally, the process of using the repaired video with the new trajectory to train the original scene reconstruction model to obtain an enhanced scene reconstruction model includes:

[0233] The original video and the repaired video with the new trajectory are used to train the original scene reconstruction model to obtain an enhanced scene reconstruction model.

[0234] Optionally, the new trajectory is the first sub-trajectory in the new complete trajectory, and the new complete trajectory includes m consecutive adjacent sub-trajectories. The process of training the original scene reconstruction model using the repaired video with the new trajectory to obtain an enhanced scene reconstruction model includes:

[0235] The repaired video of the first sub-trajectory is used to train the original scene reconstruction model to obtain the scene reconstruction model of the first training stage;

[0236] The method further includes:

[0237] Using the scene reconstruction model in the nth training stage, generate a video of the (n+1)th sub-trajectory in the new complete trajectory, where n is a positive integer from 1 to (m-1);

[0238] Using the image restoration model, the image in the generated video of the (n+1)th sub-trajectory is restored to obtain the restored video of the (n+1)th sub-trajectory;

[0239] The scene reconstruction model of the nth training stage is trained using the repaired video of the (n+1)th sub-trajectory to obtain the scene reconstruction model of the (n+1)th training stage.

[0240] Until n increases from 1 to m-1, the scene reconstruction model in the m-th training stage is used as the final enhanced scene reconstruction model.

[0241] Optionally, the step of using the repaired video of the (n+1)th sub-trajectory to train the scene reconstruction model of the nth training stage to obtain the scene reconstruction model of the (n+1)th training stage includes:

[0242] The scene reconstruction model in the nth training stage is trained using the original video and the repaired video of the (n+1)th sub-trajectory to obtain the scene reconstruction model in the (n+1)th training stage.

[0243] Optionally, the step of using the original scene reconstruction model to generate a new trajectory video includes:

[0244] Obtain the structured information of the images in the original video;

[0245] Reconstruct the model using the original scene to generate 3D scene information;

[0246] Based on the required new trajectory and the structured information, the 3D scene information is rendered through projection transformation to obtain the generated video of the new trajectory.

[0247] A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a terminal's processor, enables the terminal to perform a video generation method, the method comprising:

[0248] Input the target trajectory into the trained enhanced scene reconstruction model as described above;

[0249] Based on the target trajectory, the enhanced scene reconstruction model generates a video of the target trajectory.

[0250] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0251] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0252] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0253] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0254] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0255] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0256] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0257] The above provides a detailed description of a method for enhancing a scene reconstruction model, a device for enhancing a scene reconstruction model, an electronic device, and a readable storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for enhancing a scene reconstruction model, characterized in that, The method includes: A new trajectory is generated by using the original scene reconstruction model; wherein the original scene reconstruction model is trained using the original video, and the new trajectory is a trajectory different from the original trajectory, and the original trajectory is the camera trajectory of the original video; An image restoration model is used to restore the images in the generated video of the new trajectory, resulting in a restored video of the new trajectory. The original scene reconstruction model is trained using the repaired video with the new trajectory to obtain an enhanced scene reconstruction model.

2. The method according to claim 1, characterized in that, Before generating a new trajectory video by reconstructing the model using the original scene, the method further includes: Using the original scene reconstruction model, a generated video of the original trajectory is generated; Based on the generated video of the original trajectory and the original video, image pair training data is determined, wherein the image in the generated video of the original trajectory is used as the input image in the image pair, and the image in the original video is used as the ground truth image in the image pair. The image restoration model is trained using the image pairs as training data.

3. The method according to claim 2, characterized in that, Before training the image restoration model using the image pairs as training data, the method further includes: Obtain the true video frame; Based on the true video frame, a masked video frame is randomly generated; The pedestal image restoration model is trained using the masked video frames and the ground truth video frames. The step of training the image restoration model using the image pairs as training data includes: The image pairs are used as training data to train the base image restoration model, thus obtaining the image restoration model.

4. The method according to any one of claims 1-3, characterized in that, The restored video using the new trajectory is used to train the original scene reconstruction model to obtain an enhanced scene reconstruction model, including: The original video and the repaired video with the new trajectory are used to train the original scene reconstruction model to obtain an enhanced scene reconstruction model.

5. The method according to any one of claims 1-4, characterized in that, The new trajectory is the first sub-trajectory in the new complete trajectory, and the new complete trajectory includes m consecutive adjacent sub-trajectories. The restored video using the new trajectory is used to train the original scene reconstruction model to obtain an enhanced scene reconstruction model, including: The repaired video of the first sub-trajectory is used to train the original scene reconstruction model to obtain the scene reconstruction model of the first training stage; The method further includes: Using the scene reconstruction model in the nth training stage, generate a video of the (n+1)th sub-trajectory in the new complete trajectory, where n is a positive integer from 1 to (m-1); Using the image restoration model, the image in the generated video of the (n+1)th sub-trajectory is restored to obtain the restored video of the (n+1)th sub-trajectory; The scene reconstruction model of the nth training stage is trained using the repaired video of the (n+1)th sub-trajectory to obtain the scene reconstruction model of the (n+1)th training stage. Until n increases from 1 to m-1, the scene reconstruction model in the m-th training stage is used as the final enhanced scene reconstruction model.

6. The method according to claim 5, characterized in that, The step of using the repaired video of the (n+1)th sub-trajectory to train the scene reconstruction model of the nth training stage to obtain the scene reconstruction model of the (n+1)th training stage includes: The scene reconstruction model in the nth training stage is trained using the original video and the repaired video of the (n+1)th sub-trajectory to obtain the scene reconstruction model in the (n+1)th training stage.

7. The method according to any one of claims 1-6, characterized in that, The process of generating a new trajectory using a model reconstructed from the original scene includes: Obtain the structured information of the images in the original video; Reconstruct the model using the original scene to generate 3D scene information; Based on the required new trajectory and the structured information, the 3D scene information is rendered through projection transformation to obtain the generated video of the new trajectory.

8. A video generation method, characterized in that, The method includes: Input the target trajectory into the trained enhanced scene reconstruction model as described in any of claims 1-7; Based on the target trajectory, the enhanced scene reconstruction model generates a video of the target trajectory.

9. An enhancement device for a scene reconstruction model, characterized in that, The device includes: The video generation module is used to generate a new trajectory video using the original scene reconstruction model; wherein the original scene reconstruction model is trained using the original video, and the new trajectory is a trajectory different from the original trajectory, and the original trajectory is the camera trajectory of the original video; The video restoration module is used to restore the images in the generated video of the new trajectory using an image restoration model, so as to obtain the restored video of the new trajectory. The model enhancement module is used to train the original scene reconstruction model using the repaired video with the new trajectory to obtain an enhanced scene reconstruction model.

10. A video generation apparatus, characterized in that, The method includes: A trajectory input module is used to input the target trajectory into the trained enhanced scene reconstruction model as described in any one of claims 1-7; The video generation module is used to generate a video of the target trajectory based on the enhanced scene reconstruction model.

11. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method described in any one of claims 1-9.

12. A readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the method as described in any one of claims 1-9.

Images