Static image to dynamic video generation method and system based on diffusion model

Abstract

The application discloses a still image to dynamic video generation method and system based on a diffusion model, and relates to the technical field of image processing. The method comprises the following steps: acquiring a still image, performing depth estimation on the still image to obtain an initial depth map, back-projecting pixel points in the still image to a three-dimensional space to obtain a three-dimensional point cloud, acquiring a target view sequence, performing spatial transformation and projection processing on the three-dimensional point cloud according to the target view sequence to obtain corresponding view geometric information, extracting three-dimensional geometric constraint features according to the view geometric information, constructing and training a video generation model serving as the diffusion model, and constituting a target video by a video frame sequence. The application can improve the continuity of spatial structures, perspective relationships and interframe changes in the process of converting a still image into a dynamic video.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically a method and system for generating still images to dynamic videos based on a diffusion model. Background Technology

[0002] The technology for converting still life images into dynamic videos has wide applications in e-commerce displays, virtual reality, and digital content creation. It can transform static product images into interactive, dynamic content, significantly improving user experience and business conversion efficiency. With the breakthrough progress of diffusion models in image generation, applying them to still life image-to-video generation has become an important research direction in this field.

[0003] Existing still-life image-to-video methods primarily rely on temporal interpolation of 2D image sequences or simple motion prediction, often resulting in videos lacking a realistic sense of 3D space. When processing the spatial structure of objects, these methods fail to accurately capture the geometric shape and occlusion relationships of objects in 3D space, leading to unnatural object deformation and incorrect perspective in the generated videos during viewpoint changes. For example, when showing the process of a ceramic vase rotating from the front to the side, videos generated by traditional methods often show curvature of the vase's surface and elliptical changes at the mouth that do not conform to the laws of realistic 3D projection, and the changes in light and shadow on the object's surface also lack spatial consistency.

[0004] More importantly, the 3D geometric information recovered from still images itself has depth fuzziness and structural uncertainty. When the camera viewpoint undergoes complex changes such as panning, tilting, and tilting, ensuring that each generated video frame conforms to the projection geometry constraints under that viewpoint becomes the core bottleneck restricting the technical implementation. This will directly lead to problems such as jumps in the generated surround video on the camera motion trajectory, jitter of object edges between different frames, and misalignment of the relative positional relationship between foreground objects and background space.

[0005] Therefore, how to accurately generate continuous video frames that conform to the laws of perspective projection while maintaining the structural consistency and smooth motion of objects in space has become a key issue in achieving high-quality still life surround video generation. Summary of the Invention

[0006] To address the above problems, this invention provides a method and system for generating still images to dynamic videos based on a diffusion model, which solves the problems of insufficient consistency of object structure and unstable video smoothness in existing image-generated videos.

[0007] To achieve the above objectives, the technical solution adopted by the present invention includes the following steps:

[0008] The still life image to dynamic video generation method based on the diffusion model includes the following steps:

[0009] A still life image is acquired, and depth estimation is performed on the still life image to obtain an initial depth map corresponding to the still life image. Based on the camera intrinsic parameters corresponding to the still life image and the initial depth map, the pixels in the still life image are back-projected into three-dimensional space to obtain a three-dimensional point cloud. The three-dimensional point cloud includes a number of three-dimensional points, and each three-dimensional point corresponds to a pixel in the still life image.

[0010] Obtain the target view sequence corresponding to the view change of the video frame to be generated. The target view sequence includes several consecutive viewpoints. Perform spatial transformation and projection processing on the three-dimensional point cloud according to the target view sequence to obtain the view geometric information corresponding to each video frame to be generated. Extract three-dimensional geometric constraint features according to the view geometric information.

[0011] A video generation model is constructed, which is a diffusion model. The video generation model is trained to obtain a trained video generation model. The still image, the target view sequence, and the three-dimensional geometric constraint features are input into the trained video generation model. The trained video generation model outputs a video frame sequence. The video composed of the video frame sequence is the target video.

[0012] Preferably, the depth estimation of the still life image is specifically implemented in the following ways:

[0013] The still image is input into a pre-trained monocular depth estimation model to obtain the depth value corresponding to each pixel in the still image. The depth value is then normalized to obtain the initial depth map.

[0014] Preferably, the specific implementation of back-projecting the pixels in the still life image into three-dimensional space includes:

[0015] The camera intrinsic parameters include focal length parameters and principal point offset parameters, and the two-dimensional coordinates of each pixel in the still image and the depth value corresponding to the pixel in the initial depth map are obtained.

[0016] For each pixel, the offset coordinates of the two-dimensional pixel coordinates of the pixel are calculated according to the principal point offset parameter in the camera intrinsic parameters. According to the focal length parameter and the depth value, the offset coordinates are converted into horizontal and vertical spatial coordinates in the camera coordinate system. The depth value is used as the optical axis direction coordinate in the camera coordinate system. The three-dimensional coordinates of the pixel in the camera coordinate system are determined according to the horizontal spatial coordinates, the vertical spatial coordinates and the optical axis direction coordinates.

[0017] The above processing is performed on all pixels in the still image to obtain the three-dimensional point cloud composed of several three-dimensional coordinates.

[0018] Preferably, the specific implementation of obtaining the viewpoint geometric information corresponding to each video frame to be generated includes:

[0019] For each target view in the target view sequence, the 3D point cloud is transformed to the camera coordinate system corresponding to the target view according to the camera pose parameters of the target view. The transformed 3D point cloud is then filtered to retain the 3D points within the imaging range of the target view. The retained 3D points are then projected onto the 2D imaging plane. When multiple 3D points are projected to the same position, the depth value of the 3D point closest to the target view is retained to obtain the target view depth map.

[0020] An effective projection mask is generated based on the pixel positions with effective depth values ​​in the target view depth map, and the target view depth map and the effective projection mask are used as the view geometry information corresponding to the target view.

[0021] Preferably, after obtaining the viewpoint geometry information corresponding to each video frame to be generated, the method further includes:

[0022] For each video frame to be generated, the viewpoint geometry information is used to obtain the target viewpoint depth map and effective projection mask from the viewpoint geometry information.

[0023] The camera intrinsic parameters include lens distortion parameters. Based on the lens distortion parameters, a mapping relationship is established between the pixel positions before correction and the pixel positions after correction in the target view depth map.

[0024] The edge distortion region in the target view depth map is determined according to the mapping relationship, and the depth value in the edge distortion region is resampled according to the mapping relationship to obtain the corrected target view depth map.

[0025] Preferably, the extraction of three-dimensional geometric constraint features based on the viewpoint geometric information is specifically implemented in the following ways:

[0026] Based on the effective projection mask in the view geometry information, the effective pixels in the target view depth map are determined. For each effective pixel, a local three-dimensional point set is constructed based on the three-dimensional points corresponding to the surrounding effective pixels.

[0027] The local three-dimensional point set is fitted with a local plane. The normal direction of the fitted local plane is the surface normal vector corresponding to the effective pixel. The curvature value is calculated based on the spatial deviation of the local three-dimensional point set relative to the fitted plane. The three-dimensional geometric constraint feature is formed by the surface normal vector and curvature value corresponding to each effective pixel.

[0028] Preferably, the training of the video generation model is specifically implemented in the following ways:

[0029] Acquire training samples, which include several still images of samples and their corresponding target viewpoint sequences and video frame sequences;

[0030] For each still life image, the three-dimensional geometric constraint features of the still life image are extracted according to the target view sequence;

[0031] A noise scheduling table is generated based on the number of time steps in each training session and the noise intensity corresponding to each time step.

[0032] The video frame sequence is feature-encoded to obtain video features, and noise is added to the video features according to the noise scheduling table to obtain noisy video features;

[0033] During each training session, the sample still image and its corresponding target view sequence, the three-dimensional geometric constraint features, and the noisy video features are input into the video generation model, and the video generation model outputs the noise prediction result.

[0034] The denoising loss is calculated based on the noise prediction result and the added noise. The model parameters of the video generation model are updated based on the denoising loss. When the denoising loss is less than a preset loss threshold or the number of training iterations reaches a preset number, training is stopped, and the trained video generation model is obtained.

[0035] Preferably, after the trained video generation model outputs a video frame sequence, the method further includes:

[0036] Dense optical flow estimation is performed on adjacent video frames in the video frame sequence to obtain the dense optical flow field between adjacent video frames, and a pixel displacement vector field is generated based on the dense optical flow field.

[0037] A two-dimensional spatial frequency domain transformation is performed on the pixel displacement vector field to obtain a frequency domain displacement representation. Based on the frequency radius corresponding to each frequency component in the frequency domain displacement representation, the frequency components are sorted. Frequency components with a frequency radius greater than the median value of the frequency radius are determined as high-frequency components. The sum of squares of the amplitudes of the high-frequency components is calculated to obtain the high-frequency component energy. The sum of squares of the amplitudes of all frequency components in the frequency domain displacement representation is calculated to obtain the total frequency domain energy.

[0038] The high-frequency component energy ratio of adjacent video frames is obtained based on the ratio between the high-frequency component energy and the total frequency domain energy. When the high-frequency component energy ratio of any adjacent video frame is greater than the preset energy ratio benchmark, it is determined that there is a geometric discontinuity between the adjacent video frames.

[0039] Preferably, after determining that there is a geometric discontinuity between adjacent video frames, the method further includes:

[0040] Based on the proportion of high-frequency component energy corresponding to adjacent video frames with geometric discontinuities, the abnormal motion region in the adjacent video frames is determined, and a region constraint mask is generated based on the abnormal motion region.

[0041] The still image, the target view sequence, the three-dimensional geometric constraint features, and the region constraint mask are used as model update inputs to update the video generation model. The updated video generation model is then used to regenerate the video frame sequence corresponding to the target view sequence, and the regenerated video frame sequence is used as the updated video frame sequence.

[0042] Through the above technical solutions, this invention combines the processes of depth estimation of still images, 3D point cloud construction, target view projection, and diffusion model generation. This enables the video generation model to generate a video frame sequence under the combined influence of image content, viewpoint changes, and 3D geometric constraints, thereby improving the continuity of object spatial structure, perspective relationships, and inter-frame motion during the conversion of still images into dynamic videos. At the same time, by detecting geometric discontinuities in adjacent video frames and updating the model based on abnormal regions, edge jitter, local deformation, and structural jumps in the generated video are reduced, improving the stability and realism of the target video. Attached Figure Description

[0043] Figure 1 This is a flowchart illustrating the still image to dynamic video generation method based on a diffusion model provided in an embodiment of this application.

[0044] Figure 2 This is a schematic diagram of the still life to dynamic video generation system based on the diffusion model provided in an embodiment of this application. Detailed Implementation

[0045] To enable those skilled in the art to better understand the technical solution, the present invention will be described in detail below with reference to embodiments. The description in this part is only exemplary and explanatory, and should not be used to limit the scope of protection of the present invention in any way.

[0046] like Figure 1 As shown, the still life image to dynamic video generation method based on the diffusion model includes the following steps:

[0047] The still life image to dynamic video generation method based on the diffusion model includes the following steps:

[0048] A still life image is acquired, and depth estimation is performed on the still life image to obtain an initial depth map corresponding to the still life image. Based on the camera intrinsic parameters corresponding to the still life image and the initial depth map, the pixels in the still life image are back-projected into three-dimensional space to obtain a three-dimensional point cloud. The three-dimensional point cloud includes a number of three-dimensional points, and each three-dimensional point corresponds to a pixel in the still life image.

[0049] Obtain the target view sequence corresponding to the view change of the video frame to be generated. The target view sequence includes several consecutive viewpoints. Perform spatial transformation and projection processing on the three-dimensional point cloud according to the target view sequence to obtain the view geometric information corresponding to each video frame to be generated. Extract three-dimensional geometric constraint features according to the view geometric information.

[0050] A video generation model is constructed, which is a diffusion model. The video generation model is trained to obtain a trained video generation model. The still image, the target view sequence, and the three-dimensional geometric constraint features are input into the trained video generation model. The trained video generation model outputs a video frame sequence. The video composed of the video frame sequence is the target video.

[0051] Existing still-life image-to-video methods mostly rely on 2D image features for inter-frame motion prediction or texture interpolation. When generating videos with changing viewpoints, they easily overlook the real spatial structure of still objects, leading to problems such as inaccurate perspective relationships, jittery object outlines, local deformation, and discontinuities in the structure of consecutive frames. Especially when only a single still-life image is input, the video generation model lacks 3D geometric information that can be used to constrain changes in spatial layout under different viewpoints, making it difficult to stably generate video frame sequences that conform to the changing patterns of the target viewpoint.

[0052] To this end, after acquiring a still life image, this invention first performs depth estimation on the still life image to obtain an initial depth map, and then backprojects the pixels in the still life image into three-dimensional space using camera intrinsic parameters to obtain a three-dimensional point cloud. Next, it acquires a target viewpoint sequence corresponding to the viewpoint changes of the video frames to be generated, and performs spatial transformation and projection processing on the three-dimensional point cloud based on the target viewpoint sequence to obtain the viewpoint geometric information corresponding to each video frame to be generated, and extracts three-dimensional geometric constraint features from it. Subsequently, it constructs and trains a diffusion model as a video generation model, inputting the still life image, target viewpoint sequence, and three-dimensional geometric constraint features into the trained video generation model to generate a video frame sequence and form the target video. Through the above technical means, this invention can introduce three-dimensional geometric constraints corresponding to the target viewpoint in the process of generating dynamic video from a single still life image, enabling the diffusion model to generate video under the joint constraints of image content, viewpoint changes, and spatial structure, thereby improving the perspective consistency, object structure stability, and inter-frame continuity of the generated video, and reducing structural drift and visual distortion caused by the lack of three-dimensional constraints.

[0053] In an optional embodiment, the depth estimation of the still life image is specifically implemented in the following ways:

[0054] The still image is input into a pre-trained monocular depth estimation model to obtain the depth value corresponding to each pixel in the still image. The depth value is then normalized to obtain the initial depth map.

[0055] In existing still-life image-to-video generation technologies, the input information is typically a single 2D still-life image. These images mainly contain 2D visual information such as color, texture, edges, and brightness variations, but do not directly contain the spatial distance relationships between pixels and the camera. Therefore, when relying solely on 2D image features for video generation, the video generation model is prone to problems such as jittery object outlines, local deformation, confused foreground and background depth relationships, and perspective changes that do not conform to realistic imaging rules when inferring changes in the scene from different viewpoints. Especially when generating videos with viewpoint changes such as rotation, zooming, and translation from still-life images, the lack of depth constraints makes it difficult for the model to accurately determine the distance relationships between different regions in the image in 3D space.

[0056] To address the aforementioned issues, this embodiment first performs depth estimation on the still life image after acquisition. Specifically, the still life image is input into a pre-trained monocular depth estimation model to obtain the depth value corresponding to each pixel in the still life image. The monocular depth estimation model is a neural network model pre-trained using training data containing image and its corresponding depth information. It can infer the relative depth of each pixel based on texture changes, object contours, occlusion relationships, light and shadow distribution, and perspective cues in the still life image. Thus, even if the input is only a single ordinary still life image, depth data representing the distance relationships between different regions in the image can be obtained.

[0057] Since different monocular depth estimation models may output depth values ​​at different scales, and the same model may output different depth value ranges for different images, this embodiment further normalizes the depth values ​​to obtain the initial depth map. The normalization process maps the depth values ​​of each pixel to a uniform depth value range based on the numerical distribution of the depth values, ensuring a consistent scale representation of the depth data. The initial depth map is a two-dimensional depth data formed by arranging the normalized depth values ​​of each pixel according to their pixel positions in the still image, with each pixel position corresponding to a depth value representing its spatial proximity.

[0058] In this embodiment, the pre-trained monocular depth estimation model includes an input preprocessing layer, a feature encoding network, a multi-scale feature fusion network, a depth decoding network, and a depth regression layer. The input preprocessing layer receives a still image, scales the still image to 512×512 pixels, and normalizes the pixel values ​​to obtain the model input image features.

[0059] The feature encoding network employs a convolutional neural network structure to progressively downsample and extract semantic features from the input image features of the model. The feature encoding network comprises four encoding stages: the first stage has 64 output channels, the second stage has 128, the third stage has 256, and the fourth stage has 512. Each encoding stage includes a convolutional layer, a normalization layer, and a non-linear activation layer. The convolutional kernel size of the convolutional layer is 3×3, and the stride is 1 or 2. The non-linear activation layer uses the ReLU activation function. After these four encoding stages, image features at different resolution levels are obtained.

[0060] The multi-scale feature fusion network receives image features at different resolution levels output by the feature encoding network and fuses high-level semantic features and low-level edge features. Specifically, the high-level features output by the fourth encoding stage are upsampled and sequentially concatenated with the features output by the third, second, and first encoding stages to obtain multi-scale fused features.

[0061] The depth decoding network upsamples the multi-scale fused features stepwise to restore the spatial resolution corresponding to the input still image. The depth decoding network includes four upsampling modules, each comprising a bilinear upsampling layer and a convolutional layer. The number of output channels for the convolutional layers in the four upsampling modules are 256, 128, 64, and 32 respectively, and the kernel size is 3×3 for each module. A depth regression layer is connected after the depth decoding network. The depth regression layer includes a 1×1 convolutional layer, used to map the decoded features to a single-channel depth prediction result and output the depth value corresponding to each pixel using the Sigmoid function.

[0062] The monocular depth estimation model has an input size of 512×512×3 and an output size of 512×512×1; the depth value is a normalized depth value, ranging from 0 to 1. After the model outputs the depth value, the predicted depth map is upsampled using bilinear interpolation based on the original resolution of the still image to ensure its size matches that of the still image. The upsampled depth value is then normalized to obtain the initial depth map.

[0063] When training the monocular depth estimation model, the training samples include sample images and corresponding real depth maps or pseudo-depth labels. The sample images are input into the monocular depth estimation model to obtain a predicted depth map. A depth estimation loss is constructed based on the difference between the predicted depth map and the real depth map or pseudo-depth label. The depth estimation loss includes a depth value regression loss and an edge smoothing loss. The depth value regression loss constrains the numerical difference between the depth value and the depth label, while the edge smoothing loss constrains the smoothing of depth changes within the same continuous surface region and reduces cross-edge smoothing at object edges. The Adam optimizer is used during training, with an initial learning rate of 0.0001, a batch size of 8, and 80 training epochs. Training stops when the depth estimation loss on the validation set no longer decreases or reaches the set number of training epochs, resulting in the pre-trained monocular depth estimation model.

[0064] Through the above processing, this embodiment can extend still-life images from a simple two-dimensional texture representation to an image representation containing pixel-level depth information, providing basic data for subsequent back-projection based on camera intrinsic parameters, construction of 3D point clouds, and extraction of 3D geometric constraint features. Compared with directly inputting still-life images into the video generation model, this embodiment provides spatial structure priors to the subsequent generation process through an initial depth map, which helps improve the consistency of object contours, perspective changes, and foreground / background depth relationships in the generated video, and reduces structural drift and local deformation caused by changes in viewpoint.

[0065] In an optional embodiment, the specific implementation of back-projecting the pixels in the still life image into three-dimensional space includes:

[0066] The camera intrinsic parameters include focal length parameters and principal point offset parameters, and the two-dimensional coordinates of each pixel in the still image and the depth value corresponding to the pixel in the initial depth map are obtained.

[0067] For each pixel, the offset coordinates of the two-dimensional pixel coordinates of the pixel are calculated according to the principal point offset parameter in the camera intrinsic parameters. According to the focal length parameter and the depth value, the offset coordinates are converted into horizontal and vertical spatial coordinates in the camera coordinate system. The depth value is used as the optical axis direction coordinate in the camera coordinate system. The three-dimensional coordinates of the pixel in the camera coordinate system are determined according to the horizontal spatial coordinates, the vertical spatial coordinates and the optical axis direction coordinates.

[0068] The above processing is performed on all pixels in the still image to obtain the three-dimensional point cloud composed of several three-dimensional coordinates.

[0069] In this embodiment, after obtaining the initial depth map, the pixels in the still life image are back-projected into three-dimensional space based on the camera intrinsic parameters corresponding to the still life image. The camera intrinsic parameters include a focal length parameter and a principal point offset parameter. The focal length parameter represents the scaling relationship between spatial coordinates and image coordinates during camera imaging, and the principal point offset parameter represents the position of the camera's optical axis center in the image plane. For each pixel in the still life image, the two-dimensional coordinates of that pixel and the corresponding depth value in the initial depth map are obtained.

[0070] Specifically, based on the principal point offset parameter in the camera intrinsic parameters, the offset coordinates of the pixel's two-dimensional pixel coordinates relative to the camera's optical axis center are calculated. Then, based on the focal length parameter and the depth value, the offset coordinates are converted into horizontal and vertical spatial coordinates in the camera coordinate system. Simultaneously, the depth value is used as the optical axis direction coordinate in the camera coordinate system. The three-dimensional coordinates of the pixel in the camera coordinate system are jointly determined by the horizontal spatial coordinates, the vertical spatial coordinates, and the optical axis direction coordinates. After performing the above processing on all pixels in the still image, a three-dimensional point cloud composed of several three-dimensional coordinates is obtained.

[0071] Back projection refers to the process of restoring a pixel to a three-dimensional spatial point based on its position in a two-dimensional image plane, its corresponding depth value, and camera intrinsic parameters. The camera coordinate system is a three-dimensional coordinate system established with the camera's optical center as the reference, typically used to represent the spatial position of pixels in an image. A three-dimensional point cloud is a data set composed of multiple discrete three-dimensional points, each with corresponding three-dimensional coordinates, used to approximately represent the spatial distribution of still objects and their surrounding scene.

[0072] In one specific embodiment, the still life image is a ceramic cup image with a resolution of 1024×1024 pixels. The horizontal and vertical focal lengths in the camera's intrinsic parameters are both 980 pixels, and the principal point is located at the image center, i.e., the horizontal principal point offset parameter is 512 pixels, and the vertical principal point offset parameter is 512 pixels. The two-dimensional coordinates of a pixel at the edge of the cup rim in the image are 620 horizontally and 410 vertically. The normalized depth value of this pixel in the initial depth map is 0.48, and after depth scale mapping, the actual calculated depth value is 2.13 meters. Based on the horizontal offset of 108 pixels and the vertical offset of -102 pixels relative to the principal point position of this pixel, combined with the focal length parameter and the actual calculated depth value, the three-dimensional coordinates of this pixel in the camera coordinate system are calculated, where the horizontal spatial coordinates are approximately 0.235 meters, the vertical spatial coordinates are approximately -0.222 meters, and the optical axis direction coordinates are 2.13 meters. The above back projection processing is performed on all valid pixels in the ceramic cup image to obtain a three-dimensional point cloud composed of multiple three-dimensional coordinates.

[0073] Therefore, this embodiment can convert the depth information in the initial depth map into three-dimensional geometric data that can be used for spatial transformation and target viewpoint projection, so that the subsequent video generation process no longer relies solely on two-dimensional image texture, but can constrain viewpoint changes based on the spatial structure of still objects, thereby improving the accuracy of object outlines, foreground-background occlusion relationships, and perspective changes in the generated video.

[0074] In an optional embodiment, the specific implementation of obtaining the viewpoint geometry information corresponding to each video frame to be generated includes:

[0075] For each target view in the target view sequence, the 3D point cloud is transformed to the camera coordinate system corresponding to the target view according to the camera pose parameters of the target view. The transformed 3D point cloud is then filtered to retain the 3D points within the imaging range of the target view. The retained 3D points are then projected onto the 2D imaging plane. When multiple 3D points are projected to the same position, the depth value of the 3D point closest to the target view is retained to obtain the target view depth map.

[0076] An effective projection mask is generated based on the pixel positions with effective depth values ​​in the target view depth map, and the target view depth map and the effective projection mask are used as the view geometry information corresponding to the target view.

[0077] After obtaining the 3D point cloud, a target view sequence corresponding to the viewpoint changes of the video frame to be generated is acquired. The target view sequence includes several target views arranged in chronological order, each target view corresponding to a video frame to be generated and having corresponding camera pose parameters. The camera pose parameters include rotation and translation parameters, used to represent the spatial observation position and observation direction of the video frame to be generated relative to the original viewpoint of the still image.

[0078] The 3D point cloud is a set of spatial points constructed in the original camera coordinate system. To obtain the geometric constraint information corresponding to each video frame to be generated, the 3D point cloud needs to be transformed to different target viewpoints based on the target viewpoint sequence. Specifically, for each target viewpoint in the target viewpoint sequence, a coordinate transformation relationship is constructed based on the camera pose parameters corresponding to that target viewpoint, and this coordinate transformation relationship is used to transform each 3D point in the 3D point cloud to the camera coordinate system corresponding to that target viewpoint, resulting in a transformed point cloud under the target viewpoint.

[0079] After the coordinate transformation is completed, the transformed point cloud undergoes field-of-view filtering. This filtering involves retaining 3D points located within the near clipping plane, far clipping plane, and the horizontal and vertical field of view, while discarding 3D points outside the imaging range, based on the camera's imaging range corresponding to the target's viewpoint. This process reduces invalid points in subsequent projection calculations and ensures that the generated target viewpoint depth map only contains spatial points observable by the camera from that target viewpoint.

[0080] Subsequently, the selected 3D points are projected onto the 2D imaging plane to obtain the projected position and corresponding depth value of each 3D point in the target viewpoint image plane. When multiple 3D points are projected onto the same pixel position, the depth value of the 3D point closest to the optical center of the camera corresponding to that target viewpoint is retained as the effective depth value of that pixel position, while the depth values ​​of 3D points located behind it are discarded. This process is equivalent to depth buffering, used to simulate the occlusion relationship in the real imaging process. After completing the projection processing of all retained 3D points, a target viewpoint depth map corresponding to that target viewpoint is obtained. The target viewpoint depth map is used to represent the spatial depth distribution corresponding to each pixel position under that target viewpoint.

[0081] In practical processing, when the target viewpoint changes significantly, the 3D point cloud obtained by backprojecting a single still image is insufficient to cover the back or rear side areas of the object, easily resulting in large invalid projection areas in the target viewpoint depth map. To address this issue, this embodiment uses an effective projection mask to mark the locations of pixels without effective depth values, enabling the subsequent video generation model to distinguish between reliable geometric regions and areas requiring completion. For invalid projection areas, the video generation model, when generating video frames, primarily relies on the image content features of the still image, the viewpoint conditions of the target viewpoint sequence, and the geometric constraints of adjacent effective projection areas to complete the content, thereby reducing image holes and edge breaks caused by missing point clouds. When generating the effective projection mask, if a small invalid hole exists in the target viewpoint depth map and is surrounded by an effective projection area, the invalid hole is usually caused by sparse point clouds or projection rounding. This embodiment performs neighborhood interpolation compensation based on the depth values ​​of effective pixels surrounding the invalid hole and updates the effective projection mask accordingly. If the invalid region is located at the edge of an object or in a newly revealed area, its invalidity marker is retained so that the subsequent video generation model can generate content for that region, thus avoiding mistaking unreliable depth as a true geometric constraint.

[0082] In an optional embodiment, after obtaining the viewpoint geometry information corresponding to each video frame to be generated, the method further includes:

[0083] For each video frame to be generated, the viewpoint geometry information is used to obtain the target viewpoint depth map and effective projection mask from the viewpoint geometry information.

[0084] The camera intrinsic parameters include lens distortion parameters. Based on the lens distortion parameters, a mapping relationship is established between the pixel positions before correction and the pixel positions after correction in the target view depth map.

[0085] The edge distortion region in the target view depth map is determined according to the mapping relationship, and the depth value in the edge distortion region is resampled according to the mapping relationship to obtain the corrected target view depth map.

[0086] After obtaining the viewpoint geometry information corresponding to each video frame to be generated, distortion correction processing is further performed on the target viewpoint depth map in the viewpoint geometry information. Since the target viewpoint depth map is obtained by projecting a 3D point cloud onto a 2D imaging plane, when the imaging device has lens distortion, there is an offset between the projection position of the image edge region and the ideal pinhole imaging position, which can easily lead to stretching, compression, or positional deviation in the edge region of the depth map. If the uncorrected target viewpoint depth map is directly used as the subsequent geometric constraint input to the diffusion model, the object edge region in the generated video is prone to contour drift and perspective inconsistency.

[0087] Specifically, the camera intrinsic parameters include lens distortion parameters, which include radial distortion parameters and tangential distortion parameters. Based on these lens distortion parameters, a mapping relationship is established between the pixel positions before and after correction in the target view depth map. This mapping relationship describes the coordinate correspondence of each pixel position in the target view depth map before and after distortion correction. After establishing the mapping relationship, the edge distortion region in the target view depth map is determined based on the coordinate offset between the pixel positions before and after correction. The edge distortion region is the image edge region in the target view depth map that is significantly affected by lens distortion, and it is typically located far from the principal point. For pixels within the edge distortion region, their corresponding positions in the target view depth map before correction are found according to the mapping relationship, and resampling is performed based on the depth values ​​with valid projection markers in the neighborhood of the corresponding position to obtain the corrected depth value of that pixel. After resampling each pixel within the edge distortion region, the corrected target view depth map is obtained.

[0088] Among them, lens distortion parameters refer to calibration parameters used to describe the deviation of actual lens imaging from the ideal pinhole imaging model; radial distortion parameters are used to describe the radial offset of image points from the principal point outward or inward; tangential distortion parameters are used to describe the tangential offset caused by non-ideal alignment between the lens and the imaging plane; resampling refers to the process of re-determining the depth value corresponding to the corrected pixel position based on the mapping relationship between the pixel positions before and after correction.

[0089] In one specific embodiment, the target view depth map has a resolution of 1024×1024 pixels, the camera principal point position is 512 pixels horizontally and 512 pixels vertically, and the lens distortion parameters include radial distortion parameters k1 = -0.18 and k2 = 0.035, and tangential distortion parameters p1 = 0.001 and p2 = -0.0008. Based on the above lens distortion parameters, the correction mapping relationship of each pixel position in the target view depth map is calculated, and the region within 80 pixels of the image edge and with a coordinate offset greater than 0.5 pixels before and after correction is determined as the edge distortion region. For pixels within this region, the corresponding position in the target view depth map before correction is read according to the mapping relationship; when the corresponding position falls at a non-integer pixel coordinate, bilinear interpolation is performed based on the depth values ​​of its four neighboring valid pixels to obtain the corrected depth value.

[0090] In an optional embodiment, the extraction of three-dimensional geometric constraint features based on the viewpoint geometric information is specifically implemented in the following ways:

[0091] Based on the effective projection mask in the view geometry information, the effective pixels in the target view depth map are determined. For each effective pixel, a local three-dimensional point set is constructed based on the three-dimensional points corresponding to the surrounding effective pixels.

[0092] The local three-dimensional point set is fitted with a local plane. The normal direction of the fitted local plane is the surface normal vector corresponding to the effective pixel. The curvature value is calculated based on the spatial deviation of the local three-dimensional point set relative to the fitted plane. The three-dimensional geometric constraint feature is formed by the surface normal vector and curvature value corresponding to each effective pixel.

[0093] After obtaining the target view depth map and the effective projection mask, three-dimensional geometric constraint features are further extracted based on the view geometry information. Since the target view depth map mainly represents the depth distribution of each pixel position, and the diffusion model also needs to understand the orientation changes, surface undulations, and local geometric continuity of the object surface when generating video frames, this embodiment further extracts surface normal vectors and curvature values ​​from the target view depth map to form three-dimensional geometric constraint features that can constrain the spatial structure of the video frame.

[0094] Specifically, firstly, effective pixels in the target view depth map are determined based on the effective projection mask. Effective pixels refer to pixels in the target view depth map that have effective depth values ​​and are obtained by projecting from a 3D point cloud. For each effective pixel, a neighborhood window of a preset size is selected in the target view depth map, centered on that effective pixel, and surrounding effective pixels with the same effective projection markers are filtered from within the neighborhood window. Based on the 3D points corresponding to these surrounding effective pixels, a local 3D point set is constructed.

[0095] The target view depth map has a resolution of 1024×1024 pixels. A 5×5 pixel neighborhood window is selected centered on each valid pixel. When the number of pixels with valid projection marks within this neighborhood window is no less than 8, the 3D coordinates corresponding to these valid pixels are read and a local 3D point set is formed. When the number of valid pixels is less than 8, the neighborhood window is expanded to 7×7 pixels to obtain enough local 3D points to participate in geometric calculations. This process avoids instability in the calculation of local geometric features due to sparse point clouds or projection holes.

[0096] Subsequently, a local plane fitting is performed on the local 3D point set. Local plane fitting refers to determining the plane that best approximates the distribution trend of the local point set based on the spatial distribution of each 3D point in the local 3D point set. The normal direction of the fitted local plane is the surface normal vector corresponding to the central effective pixel. The surface normal vector represents the orientation of the object's surface at that pixel position from the target viewpoint, reflecting the tilt direction and spatial orientation changes of the object's surface.

[0097] For each valid pixel, its corresponding local 3D point set is obtained, and a local plane fitting is performed on the local 3D point set to obtain a fitting plane; the vertical distance from each 3D point in the local 3D point set to the fitting plane is calculated; the squares of each vertical distance are summed and averaged to obtain the mean square deviation value; the square root of the mean square deviation value is taken to obtain the root mean square deviation value; the root mean square deviation value is used as the curvature value corresponding to the valid pixel.

[0098] In an optional embodiment, the training of the video generation model is specifically implemented in the following ways:

[0099] Acquire training samples, which include several still images of samples and their corresponding target viewpoint sequences and video frame sequences;

[0100] For each still life image, the three-dimensional geometric constraint features of the still life image are extracted according to the target view sequence;

[0101] A noise scheduling table is generated based on the number of time steps in each training session and the noise intensity corresponding to each time step.

[0102] The video frame sequence is feature-encoded to obtain video features, and noise is added to the video features according to the noise scheduling table to obtain noisy video features;

[0103] During each training session, the sample still image and its corresponding target view sequence, the three-dimensional geometric constraint features, and the noisy video features are input into the video generation model, and the video generation model outputs the noise prediction result.

[0104] The denoising loss is calculated based on the noise prediction result and the added noise. The model parameters of the video generation model are updated based on the denoising loss. When the denoising loss is less than a preset loss threshold or the number of training iterations reaches a preset number, training is stopped, and the trained video generation model is obtained.

[0105] The video generation model employs a latent space conditional diffusion structure, which is a diffusion model that performs conditional generation within a compressed feature space. Unlike directly adding or removing noise to an image or video in pixel space, the latent space conditional diffusion model first compresses the image or video frame sequence into lower-dimensional latent space features using an encoding network. Then, it performs diffusion noise addition and progressive denoising on these latent space features. Finally, it uses a decoding network to restore the denoised latent space features back to the image or video frame sequence. Here, "latent space" refers to the low-dimensional feature representation space obtained by the encoding network, which retains the main structural and semantic information of the image or video while reducing the data dimensionality. "Conditions" refers to the constraint information input during the diffusion denoising process, which in this embodiment includes still images, target viewpoint sequences, and three-dimensional geometric constraint features. "Diffusion" refers to the process of progressively adding noise to sample video features during the training phase and learning denoising rules, and then progressively restoring the target video features from the noise features during the generation phase. The video generation model used in this invention includes a video feature encoding network, a conditional encoding network, a noise addition module, a spatiotemporal denoising network, and a video decoding network. The video feature encoding network is used to encode the sample video frame sequence into video features. The conditional encoding network is used to conditionally encode the sample still image, target view sequence, and three-dimensional geometric constraint features. The noise addition module is used to add noise to the video features according to the noise scheduling table. The spatiotemporal denoising network is used to output noise prediction results based on the denoised video features and conditional features. The video decoding network is used to decode the denoised video features into a video frame sequence during the generation stage.

[0106] The sample video frame sequence consists of 16 frames, each with a resolution of 512×512 pixels. The video feature encoding network uses the encoding part of a pre-trained video autoencoder to encode the sample video frame sequence into video latent space features of size 16×64×64×4, where 16 represents the frame number, 64×64 represents the latent space resolution, and 4 represents the number of latent space channels. The video decoding network uses the decoding part corresponding to the video feature encoding network to restore the 16×64×64×4 video latent space features into a 16-frame sequence of 512×512 pixels. During training, the parameters of the video feature encoding network and the video decoding network remain unchanged to reduce training complexity and maintain the stability of video reconstruction.

[0107] The conditional coding network includes an image conditional coding branch, a viewpoint conditional coding branch, and a geometric conditional coding branch. The image conditional coding branch receives a sample still life image of size 512×512×3 and extracts image conditional features through a four-level convolutional coding structure. The output channels of the four-level convolutional coding structure are 64, 128, 256, and 512 respectively, with a kernel size of 3×3 and a stride of 2, ultimately outputting image conditional features of size 64×64×512. The viewpoint conditional coding branch receives a target viewpoint sequence, where each target viewpoint is represented by 3 rotation parameters and 3 translation parameters, forming a viewpoint parameter sequence of length 16 and 6 dimensions per frame. This viewpoint parameter sequence is mapped to 16×256 viewpoint conditional features through two fully connected layers. The geometric condition encoding branch receives three-dimensional geometric constraint features, which include surface normal vector features and curvature features. Its input size is 16×64×64×4, and the four channels include three surface normal vector channels and one curvature channel. The geometric condition encoding branch includes three three-dimensional convolutional layers with a kernel size of 3×3×3 and output channels of 64, 128 and 320 respectively, to obtain the geometric condition features.

[0108] The noise-adding module generates a noise scheduling table based on the number of time steps in the diffusion training. In this embodiment, the total number of time steps in the diffusion training is 1000, and the noise intensity increases from 0.00085 to 0.012, distributed linearly to each time step. During each training session, a training time step is selected from the 1000 time steps, and Gaussian noise is added to the video latent space features according to the noise intensity corresponding to that training time step, resulting in noisy video features. The added Gaussian noise serves as the supervision target during the training process.

[0109] The spatiotemporal denoising network adopts a U-shaped spatiotemporal denoising network structure, including a downsampling encoding part, an intermediate processing part, and an upsampling decoding part. The downsampling encoding part includes four scale levels, with the number of channels at each scale level being 320, 640, 1280, and 1280 respectively; each scale level includes a spatial convolutional layer, a temporal attention layer, and a conditional injection layer. The spatial convolutional layer is used to process spatial features within a single frame, with a kernel size of 3×3; the temporal attention layer is used to handle temporal correlations between different video frames; the conditional injection layer injects image conditional features, viewpoint conditional features, and geometric conditional features into the intermediate features of the current scale level. The intermediate processing part includes two sets of residual convolutional layers and one set of spatiotemporal attention layers. The upsampling decoding part is configured correspondingly to the downsampling encoding part and receives features of the same scale from the downsampling encoding part through skip connections.

[0110] During training, the noisy video features, training time steps, image conditional features, viewpoint conditional features, and geometric conditional features are input into the spatiotemporal denoising network. The spatiotemporal denoising network outputs a noise prediction result with the same size as the noisy video features, i.e., a predicted noise feature of 16×64×64×4. The difference between the predicted noise feature and the actual added Gaussian noise is calculated element-wise according to frame dimension, spatial dimension, and channel dimension. The sum of the squares of each element's difference and the result is divided by the total number of elements to obtain the mean squared error loss. This mean squared error loss is used as the denoising loss, and backpropagation is used to update the parameters of the conditional coding network and the spatiotemporal denoising network based on this denoising loss. The AdamW optimizer is used during training with a learning rate of 0.0001, a batch size of 2, and 800 training epochs. Training stops when the denoising loss is less than the loss threshold of 0.01, or when the number of training epochs reaches 800, resulting in a trained video generation model.

[0111] With the above structure, the video generation model can complete video generation in the latent space, reducing the computational cost of diffusion denoising directly in the pixel space. At the same time, the model introduces still images, target viewpoint sequences, and three-dimensional geometric constraint features during the denoising process, so that the generated video frame sequence is simultaneously constrained by image content, viewpoint changes, and spatial structure, thereby improving the structural stability, viewpoint continuity, and image realism of the generated video.

[0112] In an optional embodiment, after the trained video generation model outputs a sequence of video frames, the method further includes:

[0113] Dense optical flow estimation is performed on adjacent video frames in the video frame sequence to obtain the dense optical flow field between adjacent video frames, and a pixel displacement vector field is generated based on the dense optical flow field.

[0114] A two-dimensional spatial frequency domain transformation is performed on the pixel displacement vector field to obtain a frequency domain displacement representation. Based on the frequency radius corresponding to each frequency component in the frequency domain displacement representation, the frequency components are sorted. Frequency components with a frequency radius greater than the median value of the frequency radius are determined as high-frequency components. The sum of squares of the amplitudes of the high-frequency components is calculated to obtain the high-frequency component energy. The sum of squares of the amplitudes of all frequency components in the frequency domain displacement representation is calculated to obtain the total frequency domain energy.

[0115] The high-frequency component energy ratio of adjacent video frames is obtained based on the ratio between the high-frequency component energy and the total frequency domain energy. When the high-frequency component energy ratio of any adjacent video frame is greater than the preset energy ratio benchmark, it is determined that there is a geometric discontinuity between the adjacent video frames.

[0116] After the trained video generation model outputs a sequence of video frames, geometric discontinuities are detected in the sequence. Since the diffusion model may produce local jitter and structural jumps at object edges, in areas of occlusion, or in regions with significant depth changes when generating videos with varying viewpoints, it is necessary to detect motion continuity between adjacent video frames.

[0117] Specifically, dense optical flow estimation is performed on adjacent video frames in the video frame sequence to obtain a dense optical flow field between adjacent video frames, and a pixel displacement vector field is generated based on the dense optical flow field. The dense optical flow field is used to represent the displacement relationship of each pixel in adjacent video frames, and the pixel displacement vector field is used to represent the overall motion distribution between adjacent video frames.

[0118] Subsequently, a two-dimensional spatial frequency domain transformation is performed on the pixel displacement vector field to obtain a frequency domain displacement representation, and high-frequency components are divided according to the frequency radius corresponding to each frequency component. The high-frequency components correspond to local abrupt movements in the pixel displacement vector field, while the low-frequency components correspond to overall smooth movements. The sum of squares of the amplitudes of the high-frequency components is calculated to obtain the high-frequency component energy, and the sum of squares of the amplitudes of all frequency components is calculated to obtain the total frequency domain energy. The ratio of the two is then calculated to obtain the proportion of high-frequency component energy.

[0119] When the energy proportion of the high-frequency component corresponding to any adjacent video frame is greater than a preset energy proportion benchmark, a geometric discontinuity is determined to exist between the adjacent video frames. The preset energy proportion benchmark is determined based on the frequency domain statistical results of normal optical flow in continuous video samples. In this way, edge jitter, local deformation, and structural abrupt changes that occur during video generation can be identified, providing a basis for subsequent model updates or video frame corrections.

[0120] In one specific embodiment, the Horn-Schunck optical flow algorithm is used to estimate dense optical flow between adjacent video frames. The Horn-Schunck optical flow algorithm calculates the pixel motion relationship between adjacent video frames based on constant brightness constraints and global smoothness constraints, and is suitable for obtaining a dense optical flow field covering the entire frame image.

[0121] Specifically, adjacent video frames in the video frame sequence are acquired and converted into grayscale images. Then, the brightness gradients of the two grayscale images are calculated in the horizontal, vertical, and temporal directions, yielding the horizontal gradient, vertical gradient, and temporal gradient, respectively. The horizontal gradient represents the change in pixel brightness along the horizontal direction of the image, the vertical gradient represents the change in pixel brightness along the vertical direction of the image, and the temporal gradient represents the brightness change at the same pixel position between adjacent video frames.

[0122] After obtaining the horizontal, vertical, and temporal gradients, pixel motion equations are established based on constant brightness constraints, and global smoothing constraints are introduced to ensure spatial continuity of optical flow vectors at adjacent pixel locations. The horizontal and vertical optical flow components of each pixel are initialized and updated iteratively. In each iteration, the average value of the optical flow components within the current pixel's neighborhood is calculated first, and then the horizontal, vertical, and temporal gradients at that pixel location are used to update the horizontal and vertical optical flow components. Ultimately, each pixel obtains a horizontal displacement component and a vertical displacement component, and the horizontal and vertical displacement components of all pixels constitute a dense optical flow field between adjacent video frames.

[0123] In an optional embodiment, after determining that there is a geometric discontinuity between adjacent video frames, the method further includes:

[0124] Based on the proportion of high-frequency component energy corresponding to adjacent video frames with geometric discontinuities, the abnormal motion region in the adjacent video frames is determined, and a region constraint mask is generated based on the abnormal motion region.

[0125] The still image, the target view sequence, the three-dimensional geometric constraint features, and the region constraint mask are used as model update inputs to update the video generation model. The updated video generation model is then used to regenerate the video frame sequence corresponding to the target view sequence, and the regenerated video frame sequence is used as the updated video frame sequence.

[0126] Once it is determined that there is a geometric discontinuity between adjacent video frames, the video generation model is further constrained and updated to reduce local jitter, edge jumps, and structural deformation problems when regenerating the video frame sequence.

[0127] Specifically, a high-frequency motion response map is generated based on the high-frequency components corresponding to adjacent video frames exhibiting geometric discontinuities. This high-frequency motion response map is obtained by retaining the high-frequency components in the frequency domain displacement representation and performing an inverse transform, and is used to characterize the location distribution of local motion abrupt changes in adjacent video frames. Abnormal motion regions are identified based on the pixel regions with high response values ​​in the high-frequency motion response map, and these abnormal motion regions undergo connected component merging and boundary smoothing to generate a region constraint mask. This region constraint mask is used to mark image regions that require focused constraint or regeneration.

[0128] Furthermore, the directional distribution of pixel displacement vectors is statistically analyzed within the region corresponding to the region constraint mask to obtain a motion direction histogram. The directional interval with the highest proportion in the motion direction histogram is determined as the dominant motion mode. The dominant motion mode is used to represent the main local motion trend within the abnormal motion region. Through this processing, it is possible to distinguish whether the abnormal region is caused by horizontal jitter, vertical jump, or local rotational deformation, thereby providing a motion constraint basis for subsequent model updates.

[0129] During model updates, the still image, the target view sequence, the 3D geometric constraint features, and the region constraint mask are used as model update inputs. The temporal attention layer related to motion generation and the conditional injection layer related to 3D geometric constraint feature fusion in the video generation model are identified as the layers to be updated. The temporal attention layer is used to establish temporal relationships between different video frames, and the conditional injection layer is used to inject 3D geometric constraint features into the diffusion denoising process. The parameters of the video feature encoding network and the video decoding network are kept unchanged; only the parameters of the layers to be updated are updated to avoid significant changes in the overall model parameters that could damage the appearance of the still image.

[0130] A structural continuity loss is constructed based on the region constraint mask, dominant motion mode, and 3D geometric constraint features. The structural continuity loss includes region motion smoothing loss, geometric constraint preservation loss, and content preservation loss. The region motion smoothing loss is determined based on the difference between the displacement vectors of adjacent pixels within the region constraint mask; the geometric constraint preservation loss is determined based on the difference between the local structure corresponding to the regenerated video frame and the 3D geometric constraint features; and the content preservation loss is determined based on the difference between the video frame content outside the region constraint mask and the content of the video frame before the update.

[0131] Subsequently, backpropagation is performed on the layer to be updated based on the structural continuity loss to calculate the gradients of the parameters in the temporal attention layer and the conditional injection layer. When the gradient magnitude exceeds the gradient clipping range, the gradient is clipped to obtain the clipped gradient, and the parameters of the layer to be updated are updated based on the clipped gradient. Simultaneously, a parameter regularization term is added to the loss function to limit the magnitude of parameter changes in the layer to be updated during a single update process, preventing new structural deviations in other video frame regions from occurring due to local anomaly corrections.

[0132] In summary, this invention constructs a 3D point cloud by performing depth estimation and backprojection on still life images, and generates frame-by-frame corresponding perspective geometric information and 3D geometric constraint features by combining the target perspective sequence. This allows the diffusion model to generate a video frame sequence under the joint constraints of image content, perspective changes, and spatial geometric structure, thereby improving the consistency of object contours, perspective relationships, and inter-frame structures during the conversion of still life images into dynamic videos. Simultaneously, by detecting geometric discontinuities through optical flow frequency domain analysis and updating the video generation model based on abnormal motion regions, it can reduce edge jitter, local deformation, and structural jumps in the generated video, improving the continuity, stability, and realism of the target video.

[0133] like Figure 2 As shown, the still-life image to motion video generation system based on the diffusion model includes:

[0134] The point cloud creation module is used to acquire still life images, perform depth estimation on the still life images to obtain an initial depth map corresponding to the still life images, and back-project the pixels in the still life images to three-dimensional space according to the camera intrinsic parameters corresponding to the still life images and the initial depth map to obtain a three-dimensional point cloud. The three-dimensional point cloud includes a number of three-dimensional points, and each three-dimensional point corresponds to a pixel in the still life image.

[0135] The constraint extraction module is used to obtain the target view sequence corresponding to the view change of the video frame to be generated. The target view sequence includes several consecutive viewpoints. The three-dimensional point cloud is spatially transformed and projected according to the target view sequence to obtain the view geometric information corresponding to each video frame to be generated. The three-dimensional geometric constraint features are extracted according to the view geometric information.

[0136] The video generation module is used to construct a video generation model, which is a diffusion model. The video generation model is trained to obtain a trained video generation model. The still image, the target viewpoint sequence, and the three-dimensional geometric constraint features are input into the trained video generation model. The trained video generation model outputs a video frame sequence, and the video composed of the video frame sequence is the target video.

[0137] It should be noted that, in this document, the terms "comprising," "including," and any other variations are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Specific examples have been used in this document to illustrate the principles and implementation methods of the present invention. These examples are merely for the purpose of helping to understand the method and core ideas of the present invention. The above descriptions are only preferred embodiments of the present invention. It should be pointed out that, due to the limitations of written expression and the objective existence of infinite specific structures, those skilled in the art can make several improvements, modifications, or variations without departing from the principles of the present invention, and can also combine the above technical features in an appropriate manner. These improvements, modifications, variations, or combinations, or the direct application of the concept and technical solution of the present invention to other situations without modification, should all be considered within the scope of protection of the present invention.

Claims

1. A method for generating still life images to dynamic videos based on a diffusion model, characterized in that, Includes the following steps: A still life image is acquired, and depth estimation is performed on the still life image to obtain an initial depth map corresponding to the still life image. Based on the camera intrinsic parameters corresponding to the still life image and the initial depth map, the pixels in the still life image are back-projected into three-dimensional space to obtain a three-dimensional point cloud. The three-dimensional point cloud includes a number of three-dimensional points, and each three-dimensional point corresponds to a pixel in the still life image. Obtain the target view sequence corresponding to the view change of the video frame to be generated. The target view sequence includes several consecutive viewpoints. Perform spatial transformation and projection processing on the three-dimensional point cloud according to the target view sequence to obtain the view geometric information corresponding to each video frame to be generated. Extract three-dimensional geometric constraint features according to the view geometric information. A video generation model is constructed, which is a diffusion model. The video generation model is trained to obtain a trained video generation model. The still image, the target view sequence, and the three-dimensional geometric constraint features are input into the trained video generation model. The trained video generation model outputs a video frame sequence. The video composed of the video frame sequence is the target video.

2. The method for generating still life images to dynamic videos based on a diffusion model according to claim 1, characterized in that, The specific implementation of depth estimation for the still image includes: The still image is input into a pre-trained monocular depth estimation model to obtain the depth value corresponding to each pixel in the still image. The depth value is then normalized to obtain the initial depth map.

3. The method for generating still life images to dynamic videos based on a diffusion model according to claim 1, characterized in that, The specific implementation of back-projecting the pixels in the still image into three-dimensional space includes: The camera intrinsic parameters include focal length parameters and principal point offset parameters, and the two-dimensional coordinates of each pixel in the still image and the depth value corresponding to the pixel in the initial depth map are obtained. For each pixel, the offset coordinates of the two-dimensional pixel coordinates of the pixel are calculated according to the principal point offset parameter in the camera intrinsic parameters. According to the focal length parameter and the depth value, the offset coordinates are converted into horizontal and vertical spatial coordinates in the camera coordinate system. The depth value is used as the optical axis direction coordinate in the camera coordinate system. The three-dimensional coordinates of the pixel in the camera coordinate system are determined according to the horizontal spatial coordinates, the vertical spatial coordinates and the optical axis direction coordinates. The above processing is performed on all pixels in the still image to obtain the three-dimensional point cloud composed of several three-dimensional coordinates.

4. The method for generating still life images to dynamic videos based on a diffusion model according to claim 1, characterized in that, The specific implementation of obtaining the viewpoint geometric information corresponding to each video frame to be generated includes: For each target view in the target view sequence, the 3D point cloud is transformed to the camera coordinate system corresponding to the target view according to the camera pose parameters of the target view. The transformed 3D point cloud is then filtered to retain the 3D points within the imaging range of the target view. The retained 3D points are then projected onto the 2D imaging plane. When multiple 3D points are projected to the same position, the depth value of the 3D point closest to the target view is retained to obtain the target view depth map. An effective projection mask is generated based on the pixel positions with effective depth values ​​in the target view depth map, and the target view depth map and the effective projection mask are used as the view geometry information corresponding to the target view.

5. The method for generating still images to dynamic videos based on a diffusion model according to claim 4, characterized in that, After obtaining the viewpoint geometric information corresponding to each video frame to be generated, the process further includes: For each video frame to be generated, the viewpoint geometry information is used to obtain the target viewpoint depth map and effective projection mask from the viewpoint geometry information. The camera intrinsic parameters include lens distortion parameters. Based on the lens distortion parameters, a mapping relationship is established between the pixel positions before correction and the pixel positions after correction in the target view depth map. The edge distortion region in the target view depth map is determined according to the mapping relationship, and the depth value in the edge distortion region is resampled according to the mapping relationship to obtain the corrected target view depth map.

6. The method for generating still images to dynamic videos based on a diffusion model according to claim 5, characterized in that, The specific implementation of extracting three-dimensional geometric constraint features based on the viewpoint geometric information includes: Based on the effective projection mask in the view geometry information, the effective pixels in the target view depth map are determined. For each effective pixel, a local three-dimensional point set is constructed based on the three-dimensional points corresponding to the surrounding effective pixels. The local three-dimensional point set is fitted with a local plane. The normal direction of the fitted local plane is the surface normal vector corresponding to the effective pixel. The curvature value is calculated based on the spatial deviation of the local three-dimensional point set relative to the fitted plane. The three-dimensional geometric constraint feature is formed by the surface normal vector and curvature value corresponding to each effective pixel.

7. The method for generating still images to dynamic videos based on a diffusion model according to claim 1, characterized in that, The specific implementation of training the video generation model includes: Acquire training samples, which include several still images of samples and their corresponding target viewpoint sequences and video frame sequences; For each still life image, the three-dimensional geometric constraint features of the still life image are extracted according to the target view sequence; A noise scheduling table is generated based on the number of time steps in each training session and the noise intensity corresponding to each time step. The video frame sequence is feature-encoded to obtain video features, and noise is added to the video features according to the noise scheduling table to obtain noisy video features; During each training session, the sample still image and its corresponding target view sequence, the three-dimensional geometric constraint features, and the noisy video features are input into the video generation model, and the video generation model outputs the noise prediction result. The denoising loss is calculated based on the noise prediction result and the added noise. The model parameters of the video generation model are updated based on the denoising loss. When the denoising loss is less than a preset loss threshold or the number of training iterations reaches a preset number, training is stopped, and the trained video generation model is obtained.

8. The method for generating still images to dynamic videos based on a diffusion model according to claim 1, characterized in that, After the trained video generation model outputs a sequence of video frames, the following steps are also included: Dense optical flow estimation is performed on adjacent video frames in the video frame sequence to obtain the dense optical flow field between adjacent video frames, and a pixel displacement vector field is generated based on the dense optical flow field. A two-dimensional spatial frequency domain transformation is performed on the pixel displacement vector field to obtain a frequency domain displacement representation. Based on the frequency radius corresponding to each frequency component in the frequency domain displacement representation, the frequency components are sorted. Frequency components with a frequency radius greater than the median value of the frequency radius are determined as high-frequency components. The sum of squares of the amplitudes of the high-frequency components is calculated to obtain the high-frequency component energy. The sum of squares of the amplitudes of all frequency components in the frequency domain displacement representation is calculated to obtain the total frequency domain energy. The high-frequency component energy ratio of adjacent video frames is obtained based on the ratio between the high-frequency component energy and the total frequency domain energy. When the high-frequency component energy ratio of any adjacent video frame is greater than the preset energy ratio benchmark, it is determined that there is a geometric discontinuity between the adjacent video frames.

9. The method for generating still images to dynamic videos based on a diffusion model according to claim 8, characterized in that, After determining that there is a geometric discontinuity between adjacent video frames, the process also includes: Based on the proportion of high-frequency component energy corresponding to adjacent video frames with geometric discontinuities, the abnormal motion region in the adjacent video frames is determined, and a region constraint mask is generated based on the abnormal motion region. The still image, the target view sequence, the three-dimensional geometric constraint features, and the region constraint mask are used as model update inputs to update the video generation model. The updated video generation model is then used to regenerate the video frame sequence corresponding to the target view sequence, and the regenerated video frame sequence is used as the updated video frame sequence.

10. A still life image to dynamic video generation system based on a diffusion model, characterized in that, include: The point cloud creation module is used to acquire still life images, perform depth estimation on the still life images to obtain an initial depth map corresponding to the still life images, and back-project the pixels in the still life images to three-dimensional space according to the camera intrinsic parameters corresponding to the still life images and the initial depth map to obtain a three-dimensional point cloud. The three-dimensional point cloud includes a number of three-dimensional points, and each three-dimensional point corresponds to a pixel in the still life image. The constraint extraction module is used to obtain the target view sequence corresponding to the view change of the video frame to be generated. The target view sequence includes several consecutive viewpoints. The three-dimensional point cloud is spatially transformed and projected according to the target view sequence to obtain the view geometric information corresponding to each video frame to be generated. The three-dimensional geometric constraint features are extracted according to the view geometric information. The video generation module is used to construct a video generation model, which is a diffusion model. The video generation model is trained to obtain a trained video generation model. The still image, the target viewpoint sequence, and the three-dimensional geometric constraint features are input into the trained video generation model. The trained video generation model outputs a video frame sequence, and the video composed of the video frame sequence is the target video.

Images