Scene reconstruction method and device, electronic equipment, storage medium and program product

Abstract

Embodiments of the present disclosure disclose a scene reconstruction method and device, electronic equipment, storage medium and program product, wherein the method comprises: acquiring at least one first scene image of a driving scene under an initial trajectory; modeling based on the at least one first scene image to obtain an initial scene video; trajectory reconstruction based on the initial scene video to obtain at least one second scene video under a new trajectory; and checking the at least one second scene video through a motion model to obtain at least one target scene video that passes the check. According to the embodiments, various new trajectories that are not collected in the actual scene can be obtained through trajectory reconstruction, scene expansion is realized, and the motion of the second scene video corresponding to the new trajectory conforms to the physical law, is more in line with the actual situation, and is more suitable for training the automatic driving algorithm.

Description

Technical Field

[0001] This disclosure relates to scene reconstruction technology, and in particular to a scene reconstruction method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] In autonomous driving scene reconstruction tasks, the scene is typically reconstructed using a given segment of driving video as input. Compared to traditional 3D reconstruction, 3D reconstruction in autonomous driving scenes faces greater challenges. This is primarily due to the sparsity of viewpoints in autonomous driving data. Typically, unlike traditional 3D reconstruction, autonomous driving datasets lack rich multi-view images. Test sets are usually obtained by extracting frames from the original trajectory data, while the training set uses the remaining images after frame extraction. This forces existing methods to focus mainly on the reconstruction quality based on the original trajectory. Consequently, training end-to-end autonomous driving algorithms can only be performed based on the original trajectory, resulting in poor training results due to the limited training samples. Summary of the Invention

[0003] To address the aforementioned technical problems, this disclosure is proposed. Embodiments of this disclosure provide a scene reconstruction method, apparatus, electronic device, storage medium, and program product.

[0004] According to one aspect of the embodiments of this disclosure, a scene reconstruction method is provided, including:

[0005] Acquire at least one frame of the first scene image of the driving scenario under the initial trajectory;

[0006] Modeling is performed based on at least one frame of the first scene image to obtain an initial scene video;

[0007] Based on the initial scene video, trajectory reconstruction is performed to obtain at least one second scene video under the new trajectory;

[0008] The motion model is used to verify the at least one second scene video to obtain at least one target scene video that passes the verification.

[0009] Optionally, the step of modeling based on the at least one frame of the first scene image to obtain the initial scene video includes:

[0010] Dynamic separation processing is performed on the at least one frame of the first scene image to obtain a first static image and at least one group of first dynamic images; each group of first dynamic images includes at least one frame of dynamic image and each group of first dynamic images corresponds to a dynamic object.

[0011] Model each group of the first dynamic images to obtain at least one dynamic trajectory;

[0012] The at least one dynamic trajectory is transformed into the background coordinate system corresponding to the first static image to obtain the initial scene video.

[0013] Optionally, the dynamic object includes the vehicle itself and at least one other vehicle, and the step of modeling each group of the first dynamic image sets to obtain at least one dynamic trajectory includes:

[0014] Based on the dynamic object, the at least one group of first dynamic images is classified to determine the first dynamic image group corresponding to the vehicle and the first dynamic image group corresponding to related vehicles; the related vehicles are the other vehicles that are closest to the vehicle.

[0015] Based on predefined interaction rules and predefined behaviors corresponding to the relevant vehicles, the at least one dynamic trajectory corresponding to the vehicle is determined.

[0016] Optionally, acquiring at least one frame of the first scene image of the driving scene under the initial trajectory includes:

[0017] Acquire at least one frame of a third scene image of the driving scenario under the initial trajectory;

[0018] The first scene image is obtained based on the at least one frame of the third scene image.

[0019] Optionally, obtaining the at least one first scene image based on the at least one third scene image includes:

[0020] Interpolation processing is performed on the initial trajectory corresponding to the at least one frame of the third scene image to obtain the at least one frame of the first scene image; and / or,

[0021] For each position information corresponding to at least one frame of the third scene image in the initial trajectory, a horizontal translation operation is performed to obtain the at least one frame of the first scene image.

[0022] Optionally, the step of reconstructing the trajectory based on the initial scene video to obtain at least one second scene video under the new trajectory includes:

[0023] The initial scene video is reconstructed using a driving scene reconstruction model to obtain at least one second scene video under a new trajectory.

[0024] Optionally, the step of verifying the at least one second scene video using a motion model to obtain at least one target scene video that passes the verification includes:

[0025] For each of the second scene videos, the next frame estimated scene image is estimated from the current frame scene image in the second scene video using a motion model;

[0026] The next frame scene image in the second scene video is matched with the next frame estimated scene image to determine the verification result;

[0027] The at least one target scene video is determined based on the verification result.

[0028] Optionally, estimating the next frame estimated scene image from the current frame scene image in the second scene video using a motion model includes:

[0029] Determine the first pose information of the vehicle based on the current frame scene image;

[0030] Based on the motion model, pose estimation is performed on the first pose information to obtain the second estimated pose information corresponding to the estimated scene image in the next frame.

[0031] Optionally, the step of matching the next frame scene image in the second scene video with the next frame estimated scene image to determine the verification result includes:

[0032] Obtain the second pose information corresponding to the next frame scene image in the second scene video;

[0033] The second pose information is matched with the second estimated pose information to determine the verification result.

[0034] According to another aspect of the embodiments of this disclosure, a scene reconstruction apparatus is provided, comprising:

[0035] The image acquisition module is used to acquire at least one frame of the first scene image of the driving scene under the initial trajectory;

[0036] An initial modeling module is used to model based on the at least one frame of the first scene image to obtain an initial scene video;

[0037] The trajectory reconstruction module is used to reconstruct the trajectory based on the initial scene video to obtain at least one second scene video under the new trajectory.

[0038] The video verification module is used to verify the at least one second scene video through a motion model to obtain at least one target scene video that passes the verification.

[0039] Optionally, the initial modeling module is specifically used to perform dynamic separation processing on the at least one frame of the first scene image to obtain a first static image and at least one group of first dynamic images; each group of first dynamic images includes at least one frame of dynamic image, and each group of first dynamic images corresponds to a dynamic object; modeling is performed on each group of first dynamic images to obtain at least one dynamic trajectory; the at least one dynamic trajectory is converted to the background coordinate system corresponding to the first static image to obtain the initial scene video.

[0040] Optionally, the dynamic object includes the vehicle itself and at least one other vehicle. When the initial modeling module models each group of the first dynamic image sets to obtain at least one dynamic trajectory, it is used to classify the at least one group of the first dynamic image sets based on the dynamic object to determine the first dynamic image set corresponding to the vehicle and the first dynamic image set corresponding to the related vehicle. The related vehicle is the other vehicle that is closest to the vehicle. Based on predefined interaction rules and predefined behaviors corresponding to the related vehicle, the at least one dynamic trajectory corresponding to the vehicle is determined.

[0041] Optionally, the image acquisition module is specifically used to acquire at least one frame of a third scene image of the driving scene under the initial trajectory; and to obtain the at least one frame of a first scene image based on the at least one frame of the third scene image.

[0042] Optionally, when the image acquisition module obtains the at least one frame of the first scene image based on the at least one frame of the third scene image, it is used to perform interpolation processing on the initial trajectory corresponding to the at least one frame of the third scene image to obtain the at least one frame of the first scene image; and / or, respectively perform a lateral translation operation on at least one position information corresponding to the at least one frame of the third scene image in the initial trajectory to obtain the at least one frame of the first scene image.

[0043] Optionally, the trajectory reconstruction module is specifically used to reconstruct the trajectory of the initial scene video using a driving scene reconstruction model to obtain the at least one second scene video under the new trajectory.

[0044] Optionally, the video verification module is specifically configured to, for each second scene video, estimate the next frame estimated scene image from the current frame scene image in the second scene video using a motion model; match the next frame scene image in the second scene video with the next frame estimated scene image to determine the verification result; and determine the at least one target scene video based on the verification result.

[0045] Optionally, when the video verification module estimates the next estimated scene image based on the current frame scene image in the second scene video using the motion model, it is used to determine the first pose information corresponding to the vehicle based on the current frame scene image; and to perform pose estimation on the first pose information based on the motion model to obtain the second estimated pose information corresponding to the next estimated scene image.

[0046] Optionally, when the video verification module matches the next frame scene image in the second scene video with the next frame estimated scene image to determine the verification result, it is used to obtain the second pose information corresponding to the next frame scene image in the second scene video; and match the second pose information with the second estimated pose information to determine the verification result.

[0047] According to another aspect of the present disclosure, an electronic device is provided, comprising:

[0048] Memory, used to store computer program products;

[0049] A processor is configured to execute a computer program product stored in the memory, and when the computer program product is executed, to implement the scene reconstruction method described in any of the above embodiments.

[0050] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored, characterized in that, when the computer program instructions are executed by a processor, they implement the scene reconstruction method described in any of the above embodiments.

[0051] According to another aspect of the present disclosure, a computer program product is provided, including computer program instructions, characterized in that, when executed by a processor, the computer program instructions implement the scene reconstruction method described in any of the above embodiments.

[0052] Based on the scene reconstruction method, apparatus, electronic device, storage medium, and program product provided in the above embodiments of this disclosure, at least one frame of a first scene image of a driving scene under an initial trajectory is acquired; modeling is performed based on the at least one frame of the first scene image to obtain an initial scene video; trajectory reconstruction is performed based on the initial scene video to obtain at least one second scene video under a new trajectory; and the at least one second scene video is verified by a motion model to obtain at least one target scene video that passes the verification. This embodiment can obtain various new trajectories not collected in actual scenes through trajectory reconstruction, realizing scene expansion. Furthermore, the motion model verification of the reconstructed second scene video ensures that the motion of the second scene video corresponding to the new trajectory conforms to physical laws, is more consistent with actual conditions, and is more suitable for training autonomous driving algorithms.

[0053] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0054] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0055] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein:

[0056] Figure 1 This is a schematic flowchart of a scene reconstruction method provided in an exemplary embodiment of this disclosure;

[0057] Figure 2 This is a public announcement Figure 1 A flowchart illustrating step 104 in the illustrated embodiment;

[0058] Figure 3 This is a flowchart of an embodiment of the training method for the scene reconstruction model disclosed herein;

[0059] Figure 4 The overall process of training a scene reconstruction model using a training tool is shown.

[0060] Figure 5 This is a public announcement Figure 1 A flowchart illustrating step 108 in the illustrated embodiment;

[0061] Figure 6 This is a schematic diagram of the structure of a scene reconstruction apparatus provided in an exemplary embodiment of the present disclosure;

[0062] Figure 7 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0063] Hereinafter, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present disclosure, and not all embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0064] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0065] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0066] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0067] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0068] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship. The data referred to in this disclosure can include unstructured data such as text, images, and videos, as well as structured data.

[0069] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0070] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0071] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0072] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0073] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0074] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0075] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0076] Application Overview

[0077] In the process of realizing this disclosure, the inventors discovered that existing scene reconstruction methods have at least the following problems: Although existing scene reconstruction techniques can create realistic digital twins, these methods are limited by their training datasets and can usually only generate high-quality sensor outputs within the area covered by the recorded camera trajectory.

[0078] Exemplary methods

[0079] Figure 1 This is a schematic flowchart of a scene reconstruction method provided in an exemplary embodiment of this disclosure. This embodiment can be applied to electronic devices, such as... Figure 1 As shown, it includes the following steps:

[0080] Step 102: Obtain at least one frame of the first scene image of the driving scene under the initial trajectory.

[0081] The first scene image under the initial trajectory can be obtained based on data collected during the vehicle's journey over a certain distance. Based on real-world driving data, the image collected by the vehicle is also referred to as the ground truth (GT) video under the original trajectory. Optionally, the first scene image can be a directly acquired image, or an image obtained by processing the acquired image through at least one method (e.g., data augmentation).

[0082] Step 104: Model based on at least one frame of the first scene image to obtain the initial scene video.

[0083] In one embodiment, image-to-scene modeling can be achieved using any modeling method in the prior art.

[0084] Step 106: Reconstruct the trajectory based on the initial scene video to obtain at least one second scene video under the new trajectory.

[0085] Optionally, existing trajectory reconstruction methods can be used to reconstruct the trajectory of the initial scene video to obtain a second scene video under the new trajectory. For example, a low-quality video is first generated from the unknown new trajectory, and then the output is processed by a driving scene reconstruction model to obtain a high-quality restored video. The driving scene reconstruction model can be trained to generate a high-quality second scene video under the new trajectory.

[0086] Step 108: Verify at least one second scene video using a motion model to obtain at least one target scene video that has passed the verification.

[0087] In the fields of autonomous driving and computer vision, motion models are mathematical models that describe the motion of objects over time. They play a crucial role in target tracking, path prediction, and decision planning. Object motion is an objective phenomenon, while a motion model is an abstract description of this phenomenon. That is, the motion of a real object must conform to the motion model. In this embodiment, to verify whether the second scene video reconstructed from the trajectory conforms to the real motion laws of objects, the motion model is used to verify the second scene video. This ensures that the target scene video that passes the verification conforms to the motion laws of objects, making the data more reliable. When using the target scene video to train the autonomous driving algorithm, better training results can be achieved.

[0088] The scene reconstruction method provided in the above embodiments of this disclosure acquires at least one frame of a first scene image of a driving scene under an initial trajectory; models the scene based on the at least one frame of the first scene image to obtain an initial scene video; reconstructs the trajectory based on the initial scene video to obtain at least one second scene video under a new trajectory; and verifies the at least one second scene video using a motion model to obtain at least one target scene video that passes the verification. This embodiment can obtain various new trajectories not collected in the actual scene through trajectory reconstruction, achieving scene expansion. Furthermore, the motion model verification of the reconstructed second scene video ensures that the motion of the second scene video corresponding to the new trajectory conforms to physical laws, is more consistent with actual conditions, and is more suitable for training autonomous driving algorithms.

[0089] like Figure 2 As shown above, in the above Figure 1 Based on the illustrated embodiment, step 104 may include the following steps:

[0090] Step 1041: Perform dynamic separation processing on at least one frame of the first scene image to obtain a first static image and at least one group of first dynamic images.

[0091] Each first dynamic image group includes at least one dynamic image frame, and each first dynamic image group corresponds to one dynamic object. The dynamic object can be the vehicle that captured the image, or other moving objects such as other vehicles, people, and animals moving around the vehicle.

[0092] Step 1042: Model each group of first dynamic images to obtain at least one dynamic trajectory.

[0093] Optionally, at least one pose information corresponding to the dynamic object is determined based on the first dynamic image group; and the dynamic trajectory corresponding to the dynamic object is obtained based on the at least one pose information.

[0094] Step 1043: Transform at least one dynamic trajectory to the background coordinate system corresponding to the first static image to obtain the initial scene video.

[0095] This embodiment separates the modeling of the background and dynamic objects; it models the static background and dynamic objects separately (for example, identifying dynamic objects in an image, distinguishing them from the static background, modeling based on the static background, extracting dynamic regions based on motion residuals or pixel differences, and modeling the dynamic regions in consecutive frames to obtain the trajectory of the dynamic objects), thus clarifying the spatial relationships of objects in the scene. Then, it utilizes real-time dynamic object transformation technology to transform the local models of dynamic objects into the background coordinate system in real time, enabling dynamic updates and rendering of dynamic objects (such as vehicles) in the scene. It improves the image rendering quality from new perspectives through viewpoint synthesis and scene editing, and combines a driving scene reconstruction model to correct visual artifacts that occur during rendering, especially during dynamic operations such as lane changes.

[0096] Existing scene reconstruction methods suffer from insufficient data, making it impossible to reconstruct some long-tailed scenes, such as sudden braking by the vehicle in front, thus failing to provide rich scene data for reinforcement learning algorithms. This embodiment addresses this problem by proposing the following solution. In some optional embodiments, the dynamic objects include the vehicle itself and at least one other vehicle; step 1042 may include:

[0097] Based on the dynamic object, at least one group of first dynamic images is classified to determine the first dynamic image group corresponding to the vehicle and the first dynamic image group corresponding to related vehicles.

[0098] Among them, the relevant vehicles are the other vehicles that are closest to the vehicle itself.

[0099] Optionally, classification processing can be performed on the dynamic images in the first dynamic image group based on deep learning methods, and the dynamic images can be matched according to dynamic objects to obtain the first dynamic image group corresponding to the vehicle and the first dynamic image group corresponding to related vehicles.

[0100] Based on predefined interaction rules and predefined behaviors corresponding to relevant vehicles, at least one dynamic trajectory corresponding to the vehicle is determined.

[0101] In this embodiment, some interaction rules between the autonomous vehicle and related vehicles are predefined. When a related vehicle exhibits a predefined behavior, the trajectory information of the autonomous vehicle is processed accordingly by searching the predefined interaction rules to obtain at least one modified dynamic trajectory. To control the generation of long-tail data (such as sudden stops) in the trajectories of surrounding vehicles, this embodiment proposes a dynamic adversarial agent. The dynamic adversarial agent process includes two parts: target vehicle identification and interaction trajectory generation. Through motion trajectory analysis, the other vehicles closest to the autonomous vehicle are identified. Then, based on the predefined behaviors of related vehicles (such as braking or overtaking) specified in the predefined interaction rules, combined with the trajectory information of related vehicles and the autonomous vehicle, a new motion trajectory for the autonomous vehicle is generated. The generation of the new motion trajectory can be determined based on rules (e.g., setting braking to stop) or based on a large model (e.g., outputting corresponding trajectory information based on input text). The dynamic adversarial agent (DAA) proposed in this embodiment generates long-tail traffic scenes by autonomously adjusting the trajectories of surrounding vehicles (e.g., controlling related vehicles to suddenly brake or cut in), thereby compensating for the insufficient coverage of long-tail scenes in existing datasets and enabling reinforcement learning to learn in more complex situations.

[0102] In some alternative embodiments, based on the above embodiments, step 102 may include:

[0103] Acquire at least one frame of the third scene image of the driving scenario under the initial trajectory; based on the at least one frame of the third scene image, obtain at least one frame of the first scene image.

[0104] In this embodiment, the third scene image can be directly used as the first scene image. Alternatively, to enrich the diversity of the first scene image, at least one extended processing is performed on the third scene image. In this embodiment, the third scene image is data collected from real driving vehicles. This data collection process is very complex, wastes a lot of manpower and resources, and is limited by real driving scenarios, resulting in a relatively simple first scene image. To solve this problem, optionally, interpolation processing is performed on the initial trajectory corresponding to at least one frame of the third scene image to obtain at least one frame of the first scene image. By performing interpolation processing on the initial trajectory determined by the position information corresponding to the initial acquisition of at least one frame of the third scene image, more position information can be obtained, which is suitable for enriching scarce driving behavior scenarios.

[0105] And / or, respectively, perform a lateral translation operation on at least one position information corresponding to at least one frame of the third scene image in the initial trajectory to obtain at least one frame of the first scene image.

[0106] By laterally shifting at least one positional information in the initial trajectory, a new trajectory can be generated based on the initial trajectory, thereby expanding the data coverage.

[0107] Initial trajectory recovery can be achieved using existing technologies, such as comprehensively utilizing visual features, IMU data (provided by the vehicle's onboard IMU device), and GPS information (provided by the vehicle's onboard positioning device), through four core steps: feature matching, motion estimation, multi-sensor fusion, and trajectory optimization. The CousinTrajectory Generator (CTG) proposed in this embodiment is a trajectory data augmentation method. Addressing the problem of overly concentrated and lacking diversity of straight-line driving scenarios in existing training datasets, CTG generates richer expert trajectory data through interpolation and lateral offset, resulting in better imitation learning performance.

[0108] Autonomous driving data collection typically involves gathering data from a segment of the vehicle's driving process, with the trajectory of this process known as the original trajectory. If the vehicle travels only in one lane during this process, the collected data may only represent one lane, resulting in an autonomous driving dataset usually consisting only of data from the original trajectory and lacking rich multi-view images, leading to sparse viewpoints in the autonomous driving data. When rendering videos from new perspectives, such as those depicting gradual lane changes or lane shifts, the generated videos exhibit significant ghosting phenomena, including blurring, tearing, and other artifacts. Existing methods produce poor rendering quality for videos generated from new trajectories. Therefore, this embodiment proposes using a driving scene reconstruction model to reconstruct the trajectory of the initial scene video, obtaining at least one second scene video under the new trajectory.

[0109] The scene reconstruction model is trained, and the training process includes: first, training a video restoration model using an insufficiently trained scene reconstruction model and acquired images under the original trajectory; then, using the trained video restoration model to assist the reconstruction model to obtain restored images under the new trajectory; using the restored images under the new trajectory and the acquired images under the original trajectory to construct a reconstruction dataset; and using the reconstruction dataset to train the scene reconstruction model. The reconstruction dataset includes restored images under the new trajectory, addressing the problem of viewpoint sparsity in autonomous driving data. Therefore, using the trained scene reconstruction model can improve the rendering quality of images under the new trajectory, thereby improving the performance of the scene reconstruction model and solving the technical problem of poor rendering effects of videos generated under the new trajectory mentioned in existing technologies. Therefore, based on the trained scene reconstruction model, videos under the new trajectory with better rendering effects can be generated.

[0110] In some alternative embodiments, Figure 3This is a flowchart of one embodiment of the training method for the scene reconstruction model disclosed herein. For example... Figure 3 As shown, the method may specifically include:

[0111] Step 310: Obtain the captured image under the original trajectory in the driving scene.

[0112] The images captured under the original trajectory can be data collected during the vehicle's journey, representing real-world driving data, also known as ground truth (GT) videos under the original trajectory. The scene reconstruction model training tool, which is the main execution unit for the training method of the disclosed scene reconstruction model, first acquires the GT video of the original trajectory, and then uses this video to train the video restoration model and the scene reconstruction model in subsequent steps.

[0113] Step 320: Train the video restoration model using the poorly trained scene reconstruction model and the acquired images under the original trajectory.

[0114] First, a scene reconstruction model is trained using ground truth (GT) video of the original trajectory. During training, the original trajectory is downsampled, and the scene reconstruction model renders an image from the original viewpoint. Rendering can include the process of extracting images from the scene given a viewpoint. When the scene reconstruction model is not sufficiently trained, i.e., when the model has not fully converged, the output data is usually a poorly rendered image with ghosting. This poorly rendered image and the GT video of the original trajectory are used as a video pair to train a video inpainting model. In subsequent steps, the trained video inpainting model is used to assist the scene reconstruction model, enabling it to render videos from new viewpoints effectively, thereby improving the performance of the scene reconstruction model.

[0115] Step 330: Render the image under the new trajectory using the scene reconstruction model to obtain the rendered image under the new trajectory.

[0116] In this step, a new trajectory can be sampled first. For example, based on the original trajectory, switch to a viewpoint that shifts the lane 1.5 meters to the left, and sample a new trajectory from that viewpoint. The scene reconstruction model is then used to render the image under the new trajectory. Because the scene reconstruction model is not sufficiently trained for the new trajectory, the rendering quality of the image under the new trajectory generated by the model is poor, requiring restoration using a video insulation model.

[0117] Step 340: Use the trained video restoration model to restore the rendered image under the new trajectory to obtain the restored image under the new trajectory.

[0118] In step 320, an image with poor rendering quality from the original viewpoint generated by an insufficiently trained scene reconstruction model is used. This poorly rendered image and the ground truth video of the original trajectory are used as video pairs to train the video inpainting model. After the video inpainting model is trained, its parameters are locked. The model is then used to inpaint the rendered image under the new trajectory generated in step 330, resulting in an inpainted image under the new trajectory.

[0119] Step 350: Train the scene reconstruction model using the repaired image under the new trajectory and the acquired image under the original trajectory.

[0120] In this step, the scene reconstruction model is trained using the repaired image under the new trajectory obtained in step 340 and the acquired image under the original trajectory obtained in step 310.

[0121] Figure 4 This illustrates the overall process of training a scene reconstruction model using a training tool. For example... Figure 4 As shown, "Dynamic Scene Reconstruction" refers to the scene reconstruction model. At the start of training, the dataset only contains ground truth videos of the original trajectories; see [link to training data]. Figure 4 The "Original Trajectory GT Video" is used as the initial reference. First, the reconstruction model is trained using the original trajectory's GT video for a period of time. After the scene reconstruction model achieves good processing results with the original trajectory's GT video, a new trajectory can be sampled, for example, shifting the original trajectory slightly to the left by 1.5m. This instructs the scene reconstruction model to render the image under the new trajectory. A progressive trajectory sampling strategy can be employed, gradually increasing the lane offset and the offset viewpoint of the new trajectory relative to the original trajectory during multiple sampling processes, obtaining progressive trajectory samples—a series of new trajectories with different offsets. Then, the scene reconstruction model renders the new trajectory, obtaining the rendered image under the new trajectory. Figure 4 The Novel Trajectory RenderedVideo is generated at this stage. Because the scene reconstruction model is trained only on the original trajectory, the quality of the rendered images on the new trajectory may be poor. See also... Figure 4"Online Restoration" refers to the online restoration process performed by the video restoration model. A projection module is pre-set in the scene reconstruction model training tool. This module is used to reproject other vehicles and lane lines (excluding the vehicle itself) onto the rendered image based on the new trajectory. This projection yields a "3D Box Sequence" and a "HDMap Sequence." Then, the pre-trained, parameter-locked video restoration model, DriveRestorer, is used to restore the rendered image, resulting in a restored image under the new trajectory. Specifically, Figure 3 In this context, "Enc" represents encoding and "Video Decoder" represents decoding. The "Noisy Images," control condition c, and the encoded "3D Box Sequence" and "HDMap Sequence" are input into the video restoration model DriveRestorer. After processing by this model, the result is decoded (Video Decoder) to obtain the restored image under the new trajectory. Figure 3 The "Novel TrajectoryRestored Video" is mentioned. The restored image is then added to the reconstructed dataset, and the reconstructed dataset is updated. See [link to documentation]. Figure 4 The "Dataset Update" option ensures that the updated dataset includes the repaired images with the new trajectory. This completes one iteration of the process scene reconstruction model training.

[0122] In the next iteration, the trajectory is resampled from the updated dataset, ensuring that images under the new trajectory constitute a certain proportion. This process is repeated, using a scene reconstruction model to render images under the new trajectory, and a trained video restoration model to restore the rendered images under the new trajectory. The restored images are then added to the reconstruction dataset, which is then updated. Because images under the new trajectory constitute a certain proportion when sampling from the reconstruction dataset, multiple iterations can improve the rendering quality of images under the new trajectory by the reconstruction model.

[0123] In summary, based on the embodiments of this disclosure, a video restoration model is first trained using an insufficiently trained scene reconstruction model and acquired images under the original trajectory. Then, the trained video restoration model is used to assist the reconstruction model, resulting in restored images under a new trajectory. A reconstruction dataset can be constructed using the restored images under the new trajectory and the acquired images under the original trajectory. This reconstruction dataset is then used to train the scene reconstruction model. The reconstruction dataset includes restored images under the new trajectory, and the sampling ensures that images under the new trajectory constitute a certain proportion. Therefore, using the trained scene reconstruction model can improve the rendering quality of images under the new trajectory, thereby improving the performance of the scene reconstruction model.

[0124] like Figure 5 As shown above, in the above Figure 1 Based on the illustrated embodiment, step 108 may include the following steps:

[0125] Step 1081: For each second scene video, estimate the next frame estimated scene image from the current frame scene image in the second scene video using a motion model.

[0126] The first pose information of the vehicle is determined based on the current frame scene image; the pose is estimated based on the first pose information using the motion model to obtain the second estimated pose information corresponding to the next frame estimated scene image.

[0127] In a digital twin environment corresponding to the real world, the motion trajectories of both the ego vehicle and other vehicles in the scene need to conform to a motion model. Specifically, in the world coordinate system, the complete pose of the vehicle at time t is represented as: in, This represents the rotation matrix describing the vehicle's orientation. It is the three-dimensional position information of the vehicle center in the world coordinate system.

[0128] At each time step, the vehicle's pose will change according to the linear velocity v. t and steering angle δ t The update is performed according to the defined kinematic bicycle model. Specifically, the update formula for the vehicle position is shown in formula (1):

[0129]

[0130] in, This indicates the three-dimensional position information of the vehicle center in the world coordinate system at the next time step; From the rotation matrix The forward direction vector is extracted from the vector; Δt represents the duration of a time step.

[0131] The vehicle orientation is updated by rotation about the vertical axis, as shown in Equation (2):

[0132]

[0133] in, Rot represents the rotation matrix indicating the vehicle's orientation at the next time step; y (Δθ t ) represents the rotation matrix; Δθ t The incremental rotation angle can be calculated based on the exercise bike module, for example, based on the following formula (3):

[0134]

[0135] Where L represents the vehicle wheelbase (a calibrated value); v t Indicates the vehicle's forward velocity; δ t This represents the steering angle input. The corresponding rotation matrix is ​​Rot. y (Δθ t The specific expression for ) is shown in formula (4) below:

[0136]

[0137] The linear velocity, steering angle, and other information involved in the above calculation process can be obtained based on vehicle sensors installed on the vehicle.

[0138] Step 1082: Match the next frame scene image in the second scene video with the next frame estimated scene image to determine the verification result.

[0139] Optionally, the second pose information corresponding to the next frame scene image in the second scene video is obtained; the second pose information is matched with the second estimated pose information to determine the verification result.

[0140] In this embodiment, to ensure that the second scene video generated under the new trajectory conforms to the law of object motion, the pose information of the next time step (next frame) is inferred based on the motion model to obtain the second estimated pose information. The second pose information corresponding to the scene image of the next frame in the second scene video (which can be determined according to the vehicle's built-in IMU and positioning devices) is matched with the second estimated pose information. If the difference between the second pose information and the second estimated pose information is less than a preset threshold, it indicates that the two match. At this time, it is determined that the second pose information conforms to the law of kinematics, that is, the verification result is passed. However, if the difference between the second pose information and the second estimated pose information is greater than or equal to the preset threshold, it indicates that the two do not match. That is, there are trajectory points in the second scene video that do not conform to the law of kinematics, and the verification result is failed. The trajectory points in the second scene video that fail the verification can be replaced with the second estimated trajectory points, or the motion trajectory corresponding to the trajectory point can be deleted.

[0141] Step 1083: Determine at least one target scene video based on the verification results.

[0142] This embodiment uses a motion model to verify the kinematic laws of the trajectory, avoiding the problem of unusable generated trajectories. The vehicle's own position and pose information, along with that of other vehicles, are updated in real time based on the vehicle's kinematic model, ensuring realistic and reliable dynamic interactions between vehicles in the virtual environment and achieving closed-loop simulation.

[0143] Developed based on the ReconDreamer module (which constructs a realistic 4D world from single-view input video), this module integrates world model knowledge to achieve accurate representation of dynamic scenes, ensuring high-quality data acquisition from sensors even outside the recorded camera trajectory coverage area. Simultaneously, it utilizes kinematic models to model all vehicles, enabling real-time closed-loop simulation and reinforcement learning training.

[0144] The target scene video obtained by the scene reconstruction method provided in this embodiment can be used to train an end-to-end autonomous driving algorithm. The autonomous driving algorithm trained by the target scene video provided in this embodiment is more in line with the real scene because the scene video is more realistic. Furthermore, since the training data effectively covers complex and sparse long-tail scenes, the trained autonomous driving algorithm can cope with various driving scenarios.

[0145] Any scene reconstruction method provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, any scene reconstruction method provided in this disclosure can be executed by a processor, such as by a processor executing any scene reconstruction method mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated below.

[0146] Exemplary device

[0147] Figure 6 This is a schematic diagram of the structure of a scene reconstruction apparatus provided in an exemplary embodiment of this disclosure. Figure 6 As shown, the apparatus provided in this embodiment includes:

[0148] Image acquisition module 61 is used to acquire at least one frame of the first scene image of the driving scene under the initial trajectory.

[0149] The initial modeling module 62 is used to model based on at least one frame of the first scene image to obtain the initial scene video.

[0150] The trajectory reconstruction module 63 is used to reconstruct the trajectory based on the initial scene video to obtain at least one second scene video under the new trajectory.

[0151] The video verification module 64 is used to verify at least one second scene video through a motion model to obtain at least one target scene video that has passed the verification.

[0152] The scene reconstruction apparatus provided in the above embodiments of this disclosure acquires at least one frame of a first scene image of a driving scene under an initial trajectory; models the scene based on the at least one frame of the first scene image to obtain an initial scene video; reconstructs the trajectory based on the initial scene video to obtain at least one second scene video under a new trajectory; and verifies the at least one second scene video using a motion model to obtain at least one target scene video that passes the verification. This embodiment can obtain various new trajectories not collected in the actual scene through trajectory reconstruction, achieving scene expansion. Furthermore, the motion model verification of the reconstructed second scene video ensures that the motion of the second scene video corresponding to the new trajectory conforms to physical laws, is more consistent with actual conditions, and is more suitable for training autonomous driving algorithms.

[0153] In some optional embodiments, the image acquisition module 61 is specifically used to acquire at least one frame of a third scene image of the driving scene under the initial trajectory; and to obtain at least one frame of a first scene image based on the at least one frame of the third scene image.

[0154] Optionally, when the image acquisition module 61 obtains at least one first scene image based on at least one third scene image, it performs interpolation processing on the initial trajectory corresponding to the at least one third scene image to obtain at least one first scene image; and / or performs a lateral translation operation on at least one position information corresponding to the at least one third scene image in the initial trajectory to obtain at least one first scene image.

[0155] In some optional embodiments, the initial modeling module 62 is specifically used to perform dynamic separation processing on at least one frame of the first scene image to obtain a first static image and at least one group of first dynamic images; each group of first dynamic images includes at least one frame of dynamic image, and each group of first dynamic images corresponds to a dynamic object; each group of first dynamic images is modeled to obtain at least one dynamic trajectory; the at least one dynamic trajectory is converted to the background coordinate system corresponding to the first static image to obtain the initial scene video.

[0156] Optionally, the dynamic objects include the vehicle itself and at least one other vehicle. When the initial modeling module 62 models each group of first dynamic images to obtain at least one dynamic trajectory, it is used to classify at least one group of first dynamic images based on the dynamic objects, determine the first dynamic image group corresponding to the vehicle and the first dynamic image group corresponding to the related vehicle; the related vehicle is the other vehicle closest to the vehicle; and determine at least one dynamic trajectory corresponding to the vehicle based on predefined interaction rules and predefined behaviors corresponding to the related vehicle.

[0157] In some optional embodiments, the trajectory reconstruction module 63 is specifically used to reconstruct the trajectory of the initial scene video using a driving scene reconstruction model to obtain at least one second scene video under the new trajectory.

[0158] In some optional embodiments, the video verification module 64 is specifically used to estimate the next frame estimated scene image for each second scene video using a motion model; match the next frame scene image in the second scene video with the next frame estimated scene image to determine the verification result; and determine at least one target scene video based on the verification result.

[0159] Optionally, when the video verification module 64 estimates the next estimated scene image based on the current frame scene image in the second scene video using the motion model, it determines the first pose information corresponding to the vehicle based on the current frame scene image; and performs pose estimation based on the first pose information using the motion model to obtain the second estimated pose information corresponding to the next estimated scene image.

[0160] Optionally, when the video verification module 64 matches the next frame scene image in the second scene video with the next frame estimated scene image to determine the verification result, it is used to obtain the second pose information corresponding to the next frame scene image in the second scene video; and match the second pose information with the second estimated pose information to determine the verification result.

[0161] Exemplary electronic devices

[0162] Below, for reference Figure 7This describes an electronic device according to embodiments of the present disclosure. The electronic device may be either or both of a first device and a second device, or a standalone device independent of them, which may communicate with the first device and the second device to receive acquired input signals from them.

[0163] Figure 7 A block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0164] like Figure 7 As shown, the electronic device includes one or more processors and memory.

[0165] A processor can be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and can control other components in an electronic device to perform desired functions.

[0166] The memory can store one or more computer program products, and the memory can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program products can be stored on the computer-readable storage medium, and the processor can run the computer program products to implement the scene reconstruction methods of the various embodiments of this disclosure described above and / or other desired functions.

[0167] In one example, the electronic device may also include input devices and output devices, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0168] In addition, the input device may also include, for example, a keyboard, a mouse, etc.

[0169] This output device can output various information to the outside, including determined distance information, direction information, etc. The output device may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0170] Of course, for the sake of simplicity, Figure 7 Only some of the components of the electronic device relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0171] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products, including computer program instructions that, when executed by a processor, cause the processor to perform the steps in the scene reconstruction methods according to various embodiments of this disclosure as described in the foregoing portion of this specification.

[0172] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0173] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the scene reconstruction methods according to various embodiments of this disclosure as described in the foregoing portion of this specification.

[0174] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0175] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0176] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0177] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0178] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0179] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0180] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0181] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A scene reconstruction method, characterized by, The method comprises the following steps: acquiring at least one first scene image of a driving scene under an initial trajectory; modeling based on the at least one first scene image to obtain an initial scene video; trajectory reconstruction based on the initial scene video to obtain at least one second scene video under a new trajectory; checking the at least one second scene video through a motion model to obtain at least one target scene video that passes the check; the checking the at least one second scene video through a motion model to obtain at least one target scene video that passes the check comprises: for each second scene video, estimating a next frame of estimated scene image from a current frame of scene image in the second scene video through a motion model; matching the next frame of scene image in the second scene video with the next frame of estimated scene image to determine a check result; determining the at least one target scene video according to the check result.

2. The method of claim 1, wherein, the modeling based on the at least one first scene image to obtain an initial scene video comprises: performing dynamic separation processing on the at least one first scene image to obtain a first static image and at least one first dynamic image group; each first dynamic image group comprises at least one dynamic image, and each first dynamic image group corresponds to a dynamic object; modeling each first dynamic image group respectively to obtain at least one dynamic trajectory; converting the at least one dynamic trajectory to a background coordinate system corresponding to the first static image to obtain the initial scene video.

3. The method of claim 2, wherein, The dynamic object comprises a self vehicle and at least one other vehicle, and the modeling each first dynamic image group respectively to obtain at least one dynamic trajectory comprises: classifying the at least one first dynamic image group based on the dynamic object to determine the first dynamic image group corresponding to the self vehicle and the first dynamic image group corresponding to a related vehicle; the related vehicle is the other vehicle closest to the self vehicle; determining the at least one dynamic trajectory corresponding to the self vehicle based on a predefined interaction rule and a predefined behavior corresponding to the related vehicle.

4. The method according to any of claims 1 to 3, characterized in that, The method comprises the following steps: acquiring at least one third scene image of a driving scene under an initial trajectory; obtaining the at least one first scene image based on the at least one third scene image.

5. The method of claim 4, wherein, The method comprises the following steps: performing interpolation processing on the initial trajectory corresponding to the at least one third scene image to obtain the at least one first scene image; and / or performing a horizontal translation operation on at least one position information corresponding to the at least one third scene image in the initial trajectory to obtain the at least one first scene image.

6. The method according to any one of claims 1 to 5, characterized in that, The method comprises the following steps: trajectory reconstruction of the initial scene video by using a driving scene reconstruction model to obtain the at least one second scene video under a new trajectory.

7. The method according to any of claims 1 to 6, characterized in that The estimating, by the motion model, a next frame estimated scene image from a current frame scene image in the second scene video comprises: determining first pose information corresponding to the ego vehicle based on the current frame scene image; performing pose estimation on the first pose information based on the motion model to obtain second estimated pose information corresponding to the next frame estimated scene image.

8. The method of claim 7, wherein, The matching, of the next frame scene image in the second scene video with the next frame estimated scene image, to determine a verification result comprises: obtaining second pose information corresponding to the next frame scene image in the second scene video; matching the second pose information with the second estimated pose information to determine the verification result.

9. A scene reconstruction apparatus, characterized by comprising: comprise: an image acquisition module configured to acquire at least one first scene image of a driving scene under an initial trajectory; an initial modeling module configured to model based on the at least one first scene image to obtain an initial scene video; a trajectory reconstruction module configured to reconstruct a trajectory based on the initial scene video to obtain at least one second scene video under a new trajectory; a video verification module configured to verify the at least one second scene video by a motion model to obtain at least one target scene video that passes the verification; The video verification module is specifically configured to, for each second scene video, estimate a next frame estimated scene image from a current frame scene image in the second scene video by a motion model; match the next frame scene image in the second scene video with the next frame estimated scene image to determine a verification result; and determine the at least one target scene video according to the verification result. comprise:

10. An electronic device, comprising: a memory configured to store a computer program product; a processor configured to execute the computer program product stored in the memory, and when the computer program product is executed, implement the scene reconstruction method of any one of claims 1-8. The computer program instructions, when executed by the processor, implement the scene reconstruction method of any one of claims 1-8.

11. A computer-readable storage medium having stored thereon computer program instructions, wherein, The computer program instructions, when executed by the processor, implement the scene reconstruction method of any one of claims 1-8.

12. A computer program product comprising computer program instructions, characterised in that, The computer program instructions, when executed by the processor, implement the scene reconstruction method of any one of claims 1-8.

Images