A dynamic video generation method and device, a storage medium and an electronic device

Abstract

The application provides a dynamic video generation method and device, a storage medium and electronic equipment, which perform denoising processing on a conditional image, delete noise points in the conditional image by using a pre-trained denoising network, process an output image of the denoising network by using a time sequence smoothing algorithm, eliminate image fluctuation and instability caused by noise, and obtain a denoised image; the conditional image is an image generated after point rasterization processing is performed on a target point cloud set collected by a laser radar arranged on a vehicle; according to a movement trajectory of the vehicle and illumination intensity and an illumination direction in historical denoised images, the illumination intensity and the illumination direction in the denoised image are adjusted to obtain an illumination adjusted image, the historical denoised images are a plurality of denoised images before the current processed denoised image; the illumination adjusted image and a corresponding driving trajectory thereof are input into a time sequence joint optimization framework to generate a dynamic controllable video, so that a user can observe and intelligently identify.

Description

Technical Field

[0001] This invention relates to the field of images, and more specifically, to a method, apparatus, storage medium, and electronic device for generating dynamic video. Background Technology

[0002] With the deep integration of artificial intelligence and digital twin technology, the accuracy and real-time performance of environmental perception and scene modeling have become core elements for improving the reliability of autonomous driving systems. Especially in dynamic urban scenarios with dense traffic participants, variable lighting, and frequent perspective shifts, achieving efficient fusion of multimodal data and instance-level spatiotemporal consistency remains a key bottleneck restricting the implementation of autonomous driving technology. Vector maps, as the core carrier connecting environmental perception and decision-making, directly affect the depth of understanding and responsiveness of autonomous driving systems to complex scenarios. Existing technologies still have significant shortcomings when facing practical challenges.

[0003] The shortcomings include how to construct dynamic video images to facilitate user observation and intelligent recognition. Summary of the Invention

[0004] The purpose of this invention is to provide a dynamic video generation method, apparatus, storage medium, and electronic device to improve the above-mentioned problems.

[0005] To achieve the above objectives, the technical solutions adopted in the embodiments of the present invention are as follows:

[0006] In a first aspect, embodiments of the present invention provide a method for generating dynamic video, the method comprising:

[0007] The denoising process for the conditional image is as follows: noise points in the conditional image are removed using a pre-trained denoising network, and the output image of the denoising network is processed using a temporal smoothing algorithm to eliminate image fluctuations and instabilities caused by noise, so as to obtain a denoised image.

[0008] Among them, the conditional image is an image generated by rasterizing a set of target point clouds collected by a lidar deployed on a vehicle.

[0009] Based on the vehicle's trajectory and the light intensity and direction in the historical denoised images, the light intensity and direction in the denoised images are adjusted to obtain an illumination-adjusted image. The historical denoised images are multiple frames of denoised images prior to the currently processed denoised image.

[0010] The illumination-adjusted image and its corresponding driving trajectory are input into a joint optimization framework for time series analysis to generate dynamic and controllable video.

[0011] Secondly, embodiments of the present invention provide a dynamic video generation apparatus, the apparatus comprising:

[0012] The first processing unit is used to denoise the conditional image to obtain a denoised image, including: using a pre-trained denoising network to delete noise points in the conditional image, and using a temporal smoothing algorithm to process the output image of the denoising network to eliminate image fluctuations and instabilities caused by noise, so as to obtain a denoised image.

[0013] Among them, the conditional image is an image generated by rasterizing a set of target point clouds collected by a lidar deployed on a vehicle.

[0014] The first processing unit is further configured to adjust the light intensity and light direction in the denoised image according to the vehicle's movement trajectory and the light intensity and light direction in the historical denoised image to obtain an illumination-adjusted image, wherein the historical denoised image is a series of denoised images prior to the currently processed denoised image.

[0015] The second processing unit is used to input the illumination adjustment image and its corresponding driving trajectory into a joint optimization framework of time series to generate dynamic and controllable video.

[0016] Thirdly, embodiments of the present invention provide a storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method.

[0017] Fourthly, embodiments of the present invention provide an electronic device, the electronic device comprising: a processor and a memory, the memory being used to store one or more programs; when the one or more programs are executed by the processor, the above-described method is implemented.

[0018] Compared to existing technologies, the present invention provides a dynamic video generation method, apparatus, storage medium, and electronic device that denoises a conditional image. It utilizes a pre-trained denoising network to remove noise points from the conditional image and employs a temporal smoothing algorithm to process the output image of the denoising network, eliminating image fluctuations and instabilities caused by noise to obtain a denoised image. The conditional image is generated by rasterizing a set of target point clouds collected by a LiDAR deployed on a vehicle. Based on the vehicle's trajectory and the illumination intensity and direction in historical denoised images, the illumination intensity and direction in the denoised image are adjusted to obtain an illumination-adjusted image. The historical denoised images are multiple frames of denoised images preceding the currently processed denoised image. The illumination-adjusted image and its corresponding driving trajectory are input into a joint optimization framework of time series data to generate a dynamic and controllable video for user observation and intelligent recognition.

[0019] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0022] Figure 2 This is one of the flowcharts illustrating the dynamic video generation method provided in this embodiment of the invention.

[0023] Figure 3 This is a second flowchart illustrating the dynamic video generation method provided in an embodiment of the present invention.

[0024] Figure 4 This is a schematic diagram of a dynamic video generation device provided in an embodiment of the present invention.

[0025] In the diagram: 10-Processor; 11-Memory; 12-Bus; 13-Communication interface; 501-First processing unit; 502-Second processing unit. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0027] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0028] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this invention, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0029] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0030] The following detailed description of some embodiments of the present invention is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0031] This invention provides an electronic device, which may be a vehicle computer or a server device communicatively connected to a vehicle computer. Please refer to... Figure 1 This is a schematic diagram of the structure of an electronic device. The electronic device includes a processor 10, a memory 11, and a bus 12. The processor 10 and the memory 11 are connected via the bus 12. The processor 10 is used to execute executable modules, such as computer programs, stored in the memory 11.

[0032] Processor 10 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the dynamic video generation method can be completed through integrated logic circuits in the hardware or software instructions within processor 10. The processor 10 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0033] The memory 11 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device.

[0034] Bus 12 can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. Figure 1 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus 12 or one type of bus 12.

[0035] The memory 11 is used to store programs, such as programs corresponding to a dynamic video generation device. The dynamic video generation device includes at least one software functional module that can be stored in the memory 11 in the form of software or firmware or embedded in the operating system (OS) of the electronic device. After receiving an execution instruction, the processor 10 executes the program to implement the dynamic video generation method.

[0036] The electronic device provided in this embodiment of the invention may further include a communication interface 13. The communication interface 13 is connected to the processor 10 via a bus.

[0037] It should be understood that, Figure 1 The structure shown is only a partial schematic diagram of the electronic device; the electronic device may also include components that are larger than... Figure 1The more or fewer components shown, or having the same Figure 1 The different configurations shown. Figure 1 The components shown can be implemented using hardware, software, or a combination thereof.

[0038] The dynamic video generation method provided in this embodiment of the invention can be applied to, but is not limited to, [various applications]. Figure 1 For the specific process of the electronic devices shown, please refer to [link / reference]. Figure 2 The dynamic video generation method includes S11, S12 and S13, which are described in detail below.

[0039] S11, Denoising the conditional image to obtain a denoised image, including: using a pre-trained denoising network to remove noise points in the conditional image, and using a temporal smoothing algorithm to process the output image of the denoising network to eliminate image fluctuations and instabilities caused by noise, so as to obtain a denoised image.

[0040] Among them, the conditional image is an image generated by rasterizing a set of target point clouds collected by a lidar deployed on a vehicle.

[0041] This denoising process is particularly suitable for dynamic urban scenes, effectively preventing visual artifacts caused by camera movement or object motion, and improving image quality stability by avoiding artifacts and blurring during generation. The denoising network employs a deep learning framework combining convolutional neural networks (CNNs) and residual connections. Its core task is to backpropagate noise from LiDAR conditional images, progressively optimizing image details. Through multiple iterative learning iterations, the network finely corrects the residual features of different image content and continuously optimizes the depth and texture effects of each video frame, ensuring high-quality visual effects.

[0042] S12, based on the vehicle's trajectory and the light intensity and direction in the historical denoised image, adjust the light intensity and direction in the denoised image to obtain the adjusted image.

[0043] Among them, the historical denoised images are the denoised images from multiple frames before the currently processed denoised image.

[0044] Video generation involves not only geometric and structural consistency but also the optimization of lighting effects. In complex urban scenes, lighting conditions often fluctuate significantly over time, especially considering variations in weather, time of day, and light source location. To overcome this issue, an optimization mechanism based on a lighting control module is introduced to adjust the light intensity and direction in the denoised image, resulting in a video with natural lighting transitions and avoiding lighting distortion or visual inconsistencies.

[0045] S13, input the illumination-adjusted image and its corresponding driving trajectory (the vehicle's attitude trajectory when acquiring the target point cloud set) into the joint optimization framework of the time series to generate dynamic and controllable video.

[0046] Among them, the driving trajectory can be understood as the attitude trajectory of the vehicle when acquiring the target point cloud set, including vehicle position information and vehicle orientation information.

[0047] In the dynamic video generation method provided in this embodiment of the invention, dynamic video images are constructed to facilitate user observation and intelligent recognition.

[0048] In one alternative implementation, the training process of the joint optimization framework for time series includes S21 and S22, which are described in detail below.

[0049] S21, the conditional images and their corresponding driving trajectories from the training phase are input into the joint optimization framework of the time series to obtain controllable videos from the training phase.

[0050] S22, construct a first-class loss function based on the decomposed images of the controllable video during the training phase and the real environment images during the training phase, in order to optimize the joint optimization framework of the time series.

[0051] Optionally, the first type of loss function includes the reconstruction loss function, the SSIM loss function, and the LPIPS loss function;

[0052] The reconstruction loss function is used to measure the pixel-level difference between the disassembled images of the controlled video during the training phase and the real environment images during the training phase.

[0053] The SSIM loss function is used to assess the structural consistency between the decomposed images of the controlled video during the training phase and the real-world images during the training phase.

[0054] The LPIPS loss function is used to measure the high-level semantic consistency between the decomposed images of the controllable video during the training phase and the real environment images during the training phase, further improving the realism of the visual quality.

[0055] Optionally, during the training or actual operation phase, the process of generating controllable videos using the joint optimization framework of time series includes: S31 and S32, which are described in detail below.

[0056] S31, perform interpolation processing based on the vehicle trajectory to obtain the trajectory connection line between the current input image and the next input image.

[0057] Ensure a smooth transition, guarantee that the generated image is consistent with the physical trajectory, and reduce visual inconsistencies caused by irregular trajectory changes.

[0058] S32, based on the current input image, uses the trajectory connecting line as a geometric constraint to adjust the perspective in order to obtain the controllable video corresponding to the trajectory connecting line.

[0059] In one alternative implementation, please refer to Figure 3 After inputting the illumination adjustment image and its corresponding driving trajectory into the joint optimization framework of the time series to generate dynamic and controllable video, the dynamic video generation method also includes S14, S15, S16 and S17, which are described in detail below.

[0060] S14, Three-dimensional scene modeling of sequence images in controllable video based on Gaussian distribution;

[0061] S15, Based on the depth information and optical flow estimation information of the sequence images in the controllable video, determine the target foreground object (in motion) in the 3D scene modeling data.

[0062] Among them, optical flow estimation information is defined as estimating the motion information of objects in an image by calculating the pixel displacement between adjacent frames, and the target foreground object is the object to be identified in motion.

[0063] S16. Spatiotemporal modeling is performed on each dynamic target region to obtain the corresponding spatiotemporal Gaussian cluster, which serves as the Gaussian foreground representation of the target foreground object.

[0064] The dynamic target region is a Gaussian ellipsoid centered on the target foreground object.

[0065] S17 uses controllable video as a supervision signal, and combines Gaussian foreground representation and Gaussian background representation to perform 3D scene image reconstruction to obtain 3D reconstructed video.

[0066] Gaussian background representation is a Gaussian representation of objects to be identified in 3D scene modeling, excluding Gaussian foreground representation.

[0067] It can display large-scale, dynamically changing 3D city scenes in real time and supports interactive operation from different perspectives. Users can freely switch between different perspectives, and the system will display the complete scene structure and dynamic changes while maintaining high-precision image quality.

[0068] Please continue to refer to this. Figure 3 In one optional implementation, after obtaining the 3D reconstructed video, the dynamic video generation method further includes: S18, as follows.

[0069] S18 renders the video based on the Gaussian weighted average of each pixel in the 3D reconstructed video to ensure the consistency and coherence of the video over time.

[0070] In one alternative implementation, after the training phase, the dynamic video generation method further includes: S19, as follows.

[0071] S19, obtain the evaluation and optimization metrics for controllable videos during the training phase.

[0072] The evaluation optimization refers to the use of PSNR and FID metrics to optimize the joint optimization framework for time series data. The specific formula is as follows.

[0073]

[0074] Here, R is the maximum possible pixel value of the preset image (255 for 8-bit images), and MSE (mean squared error) is the mean squared error between the decomposed images of the controllable video and the real environment image. By optimizing the PSNR metric, the system can ensure that the detail and sharpness of the generated images meet the expected standards, especially for high-quality reproduction in complex scenes.

[0075] PSNR is used to evaluate the sharpness of each frame in the generated scene, and the generation model is optimized by comparing the PSNR values ​​of each frame to reduce noise in the image and ensure high-quality transitions between frames.

[0076] The Field Image Distinction (FID) metric is an important indicator for measuring the visual difference between decomposed images of a controllable video and real-world images, and it is widely used in performance evaluation of Generative Adversarial Networks (GANs) and diffusion models. It assesses the realism of the generated result by calculating the difference between the feature distributions of the decomposed images of the controllable video and the feature distributions of the real-world images. Specifically, FID calculates the distance between the mean and covariance matrices of the feature vectors extracted by the pre-trained Inception network and the real image.

[0077] The formula is as follows:

[0078] FID = ||μ r -μ g || 2 +Tr(∑r+∑g-2(∑r∑g) 1 / 2 )

[0079] Where, μ r μ is the feature mean of the decomposed image of the controllable video. g Let ∑r be the feature mean of the real environment image, ∑g be the covariance matrix of the feature distribution of the decomposed images of the controllable video, and Tr be the trace of the product of the sum and square root of the covariance matrices, thus providing a more comprehensive assessment of the degree of difference between the two distributions. A lower FID value indicates that the generated image is closer to the feature distribution of the real image, meaning that the generated result is more visually realistic and diverse.

[0080] Regarding the generation method of the conditional image, this embodiment of the invention also provides an optional implementation method, please refer to the following.

[0081] The target point cloud set is colored based on the environmental image corresponding to the target time.

[0082] The target point cloud set is the set of point cloud data collected by the LiDAR deployed on the vehicle at the target time. The coloring process refers to adding the color value corresponding to the environment image to each point in the target point cloud set. The environment image is the two-dimensional visual image acquired by the vehicle's image acquisition system at the target time, and the environment image includes the color value of each pixel.

[0083] The target point cloud set after coloring is subjected to foreground-background separation processing to determine the dynamic foreground region and the static background region.

[0084] To highlight the salience of target objects (moving objects) in dynamic foreground regions within point cloud conditional images, a foreground-background separation mechanism combining geometric segmentation and semantic guidance (annotation) is introduced to perform foreground-background separation processing on the colored target point cloud set. This foreground-background separation process ensures the overall structural coherence of the conditional image while enhancing the expressive power of local regions, enabling the vector map generation model to maintain structural consistency for key target objects even when considering changes in viewpoint.

[0085] By combining the historical point cloud set within the time window, the incomplete areas are filled in.

[0086] The time window is a window of a preset length between target times, and the incomplete area is a dynamic foreground area in the target point cloud set that is occluded or incomplete.

[0087] Using perspective projection mapping, the completed target point cloud set is projected onto the vehicle's image coordinate system to obtain the first projected image.

[0088] The raster region in the first projected image is rendered to obtain a point rasterized conditional image.

[0089] Among them, the point rasterization conditional image is used as a training reference image for the vector map generation model to improve the training effect of the model and improve the quality of the vector map it generates.

[0090] Optionally, the target point cloud set is colored based on the environmental image corresponding to the target time, including: projecting the target point cloud set onto the vehicle's image coordinate system using a perspective projection mapping relationship to obtain a second projected image; wherein, the perspective projection mapping relationship is the mapping relationship between the lidar coordinate system and the vehicle's image coordinate system; obtaining the second projected image through projection to complete the mapping from a three-dimensional spatial structure to a two-dimensional image structure; using the color value of the i-th pixel in the environmental image as the color value of the i-th pixel in the second projected image; wherein, 1≤i≤I, and I is the total number of pixels in the environmental image; determining the corresponding point of the i-th pixel in the second projected image in the target point cloud set according to the perspective projection mapping relationship; and adding the color value of the i-th pixel in the second projected image to its corresponding point in the target point cloud set.

[0091] Optionally, the environment image is an image with dynamic object annotations (e.g., cars, motorcycles, pedestrians, and bicycles) and static object annotations (e.g., roads, buildings, and trees). Based on this, foreground-background separation processing is performed on the colored target point cloud set to determine the dynamic foreground and static background regions. This includes: using spatial density, depth gradient, and normal vector as clustering reference factors (low-order geometric features), preliminary clustering of the colored target point cloud set is performed to obtain preliminary clustered regions. The preliminary clustered regions are those with spatial density greater than a density threshold, depth gradient fluctuation values ​​greater than a gradient threshold, and normal vector change values ​​greater than a vector threshold. When the proportion of dynamic points in the preliminary clustered regions is greater than or equal to a preset ratio, it is determined to be a dynamic foreground region. The coordinates of points in the preliminary clustered regions projected onto the vehicle's image coordinate system are called projected coordinates. When the pixel annotation corresponding to the projected coordinates in the environment image is a dynamic object annotation, the point corresponding to the projected coordinates is a dynamic point. When the proportion of dynamic points in the preliminary clustered regions is less than a preset ratio, it is determined to be a static background region.

[0092] Because single-frame LiDAR point cloud sets may suffer from problems such as sparse sampling, severe occlusion, and missing textures, it is difficult to achieve a continuous and complete geometric representation in image space. Therefore, this invention introduces a cross-frame point cloud aggregation mechanism to complete the target point cloud set, uniformly constructing a dense point cloud that integrates observation information from multiple time points. Optionally, by combining historical point cloud sets within a time window, incomplete regions are completed, including: comparing the foreground dynamic region and background static region of historical point cloud frames within the time window with the foreground dynamic region and background static region of the target point cloud set, based on vehicle motion trajectory, to determine incomplete regions; combining pose change reference information and vehicle motion trajectory, obtaining supplementary content for the incomplete regions from historical point cloud frames within the time window; and adding the supplementary content to the incomplete regions of the target point cloud set to complete the incomplete region completion.

[0093] Optionally, the raster region in the first projected image is rendered to obtain a point rasterized conditional image, including: constructing a target frustum model by combining the pose information of the LiDAR at the target time; projecting the raster points onto the vehicle's image coordinate system to obtain raster pixels, where the raster points are points in the target point cloud set that fall into the target frustum model; constructing a two-dimensional Gaussian kernel at the position of the raster pixels, and determining the scale range of the two-dimensional Gaussian kernel based on the depth of the raster points; using the area within the scale range corresponding to the two-dimensional Gaussian kernel as the raster region; using the RGB, depth, point density, and normal vector corresponding to the raster region as the rendering target, and performing weighted fusion on a raster region basis to complete the raster region rendering, thereby generating a conditional image that integrates geometric, texture, and shape information. By rendering the raster region in the first projected image, the final output point rasterized conditional image significantly outperforms traditional point cloud visualization results in terms of resolution, spatial consistency, and information density. It can be directly input into the vector map generation model as a high-quality prior condition, playing a key role in controlling viewpoint changes and lighting optimization.

[0094] Please see Figure 4 , Figure 4 A dynamic video generation device is provided as an embodiment of the present invention. Optionally, the dynamic video generation device is applied to the electronic device described above.

[0095] A dynamic video generation device includes: a first processing unit 501 and a second processing unit 502.

[0096] The first processing unit 501 is used to denoise the conditional image to obtain a denoised image, including: using a pre-trained denoising network to delete noise points in the conditional image, using a temporal smoothing algorithm to process the output image of the denoising network, and eliminating image fluctuations and instabilities caused by noise to obtain a denoised image.

[0097] Among them, the conditional image is an image generated by rasterizing a set of target point clouds collected by a lidar deployed on a vehicle.

[0098] The first processing unit 501 is further configured to adjust the light intensity and light direction in the denoised image according to the vehicle's movement trajectory and the light intensity and light direction in the historical denoised image, so as to obtain an illumination-adjusted image, wherein the historical denoised image is a series of denoised images prior to the currently processed denoised image.

[0099] The second processing unit 502 is used to input the illumination adjustment image and its corresponding driving trajectory into a joint optimization framework of the time series to generate dynamic and controllable video.

[0100] Optionally, the training process of the joint optimization framework for time series includes: inputting the conditional images and their corresponding driving trajectories from the training phase into the joint optimization framework for time series to obtain controllable videos from the training phase; and constructing a first-class loss function based on the decomposed images of the controllable videos from the training phase and the real environment images from the training phase to optimize the joint optimization framework for time series.

[0101] The second processing unit 502 can execute S13 as described above, and the first processing unit 501 can execute other steps in the above method embodiment.

[0102] It should be noted that the dynamic video generation device provided in this embodiment can execute the method flow shown in the above method flow embodiment to achieve the corresponding technical effects. For the sake of brevity, any parts not mentioned in this embodiment can be referred to the corresponding content in the above embodiments.

[0103] This invention also provides a storage medium storing computer instructions and programs, which, when read and run, execute the dynamic video generation method described above. The storage medium may include memory, flash memory, registers, or a combination thereof.

[0104] The following provides an electronic device, which may be a vehicle computer device or a server device communicatively connected to a vehicle computer device. This electronic device is as follows: Figure 1 As shown, the above-described dynamic video generation method can be implemented. Specifically, the electronic device includes: a processor 10, a memory 11, and a bus 12. The processor 10 may be a CPU. The memory 11 is used to store one or more programs, which, when executed by the processor 10, execute the dynamic video generation method of the above embodiment.

[0105] In summary, the present invention provides a dynamic video generation method, apparatus, storage medium, and electronic device that denoises a conditional image to obtain a denoised image. The method includes: deleting noise points from the conditional image using a pre-trained denoising network; processing the output image of the denoising network using a temporal smoothing algorithm to eliminate image fluctuations and instabilities caused by noise, thereby obtaining a denoised image; wherein the conditional image is an image generated by point rasterization processing of a target point cloud set collected by a LiDAR deployed on a vehicle; adjusting the light intensity and light direction in the denoised image based on the vehicle's trajectory and the light intensity and light direction in historical denoised images to obtain an illumination-adjusted image, wherein the historical denoised images are multiple frames of denoised images prior to the currently processed denoised image; and inputting the illumination-adjusted image and its corresponding driving trajectory into a joint optimization framework of time series to generate a dynamic and controllable video for user observation and intelligent recognition.

[0106] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0107] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A method for generating dynamic video, characterized in that, The method includes: The denoising process for the conditional image is as follows: noise points in the conditional image are removed using a pre-trained denoising network, and the output image of the denoising network is processed using a temporal smoothing algorithm to eliminate image fluctuations and instabilities caused by noise, so as to obtain a denoised image. Among them, the conditional image is an image generated by rasterizing a set of target point clouds collected by a lidar deployed on a vehicle. Based on the vehicle's trajectory and the light intensity and direction in the historical denoised images, the light intensity and direction in the denoised images are adjusted to obtain an illumination-adjusted image. The historical denoised images are multiple frames of denoised images prior to the currently processed denoised image. The illumination adjustment image and its corresponding driving trajectory are input into a joint optimization framework for time series analysis to generate dynamic and controllable video. The process of generating controllable video using the joint optimization framework of the time series includes: performing interpolation processing based on the driving trajectory to obtain the trajectory connection line between the current input image and the next input image; and adjusting the perspective based on the current input image, using the trajectory connection line as a geometric constraint, to obtain the controllable video corresponding to the trajectory connection line.

2. The dynamic video generation method as described in claim 1, characterized in that, The training process of the joint optimization framework for the time series includes: The conditional images and their corresponding driving trajectories from the training phase are input into a joint optimization framework for the time series to obtain controllable videos from the training phase. A first type of loss function is constructed based on the decomposed images of the controllable video during the training phase and the real environment images during the training phase, in order to optimize the joint optimization framework of the time series.

3. The dynamic video generation method as described in claim 2, characterized in that, The first type of loss function includes the reconstruction loss function, the SSIM loss function, and the LPIPS loss function; The reconstruction loss function is used to measure the pixel-level difference between the disassembled images of the controllable video during the training phase and the real environment images during the training phase. The SSIM loss function is used to assess the structural consistency between the decomposed images of the controllable video during the training phase and the real-world images during the training phase. The LPIPS loss function is used to measure the high-level semantic consistency between the decomposed images of the controllable video during the training phase and the real-world environment images during the training phase.

4. The dynamic video generation method as described in claim 1, characterized in that, After the method incorporates the illumination-adjusted image and its corresponding driving trajectory into a joint optimization framework for a time series to generate a dynamic and controllable video, the method further includes: Three-dimensional scene modeling is performed on the sequence of images in the controllable video based on Gaussian distribution; Based on the depth information and optical flow estimation information of the sequence images in the controllable video, the target foreground object in the 3D scene modeling data is determined; Spatiotemporal modeling is performed on each dynamic target region to obtain the corresponding spatiotemporal Gaussian cluster, which serves as the Gaussian foreground representation of the target foreground object; wherein, the dynamic target region is a Gaussian ellipsoid centered on the target foreground object; Using the controllable video as a supervision signal, and combining Gaussian foreground representation and Gaussian background representation, a three-dimensional scene image is reconstructed to obtain a three-dimensional reconstructed video.

5. The dynamic video generation method as described in claim 4, characterized in that, After obtaining the 3D reconstructed video, the method further includes: The video is rendered based on the Gaussian weighted average of each pixel in the 3D reconstructed video to ensure the consistency and coherence of the video over time.

6. A dynamic video generation device, characterized in that, The device includes: The first processing unit is used to denoise the conditional image to obtain a denoised image, including: using a pre-trained denoising network to delete noise points in the conditional image, and using a temporal smoothing algorithm to process the output image of the denoising network to eliminate image fluctuations and instabilities caused by noise, so as to obtain a denoised image. Among them, the conditional image is an image generated by rasterizing a set of target point clouds collected by a lidar deployed on a vehicle. The first processing unit is further configured to adjust the light intensity and light direction in the denoised image according to the vehicle's movement trajectory and the light intensity and light direction in the historical denoised image to obtain an illumination-adjusted image, wherein the historical denoised image is a series of denoised images prior to the currently processed denoised image. The second processing unit is used to input the illumination adjustment image and its corresponding driving trajectory into a joint optimization framework of time series to generate dynamic and controllable video. The process of generating controllable video using the joint optimization framework of the time series includes: performing interpolation processing based on the driving trajectory to obtain the trajectory connection line between the current input image and the next input image; and adjusting the perspective based on the current input image, using the trajectory connection line as a geometric constraint, to obtain the controllable video corresponding to the trajectory connection line.

7. The dynamic video generation apparatus as described in claim 6, characterized in that, The training process of the joint optimization framework for the time series includes: inputting the conditional images and their corresponding driving trajectories from the training phase into the joint optimization framework for the time series to obtain controllable videos from the training phase; and constructing a first type of loss function based on the decomposed images of the controllable videos from the training phase and the real environment images from the training phase to optimize the joint optimization framework for the time series.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method as described in any one of claims 1-5.

9. An electronic device, characterized in that, include: Processor and memory, the memory being used to store one or more programs; When the one or more programs are executed by the processor, the method as described in any one of claims 1-5 is implemented.

Images