Method and device for generating video based on three-dimensional scene, equipment, medium and product

Abstract

Embodiments of the present application provide a three-dimensional scene-based video generation method and device, equipment, medium and product, the method comprising: receiving an input panorama; determining scene geometry information corresponding to the input panorama and a target camera pose sequence; for each target camera pose, based on the scene geometry information, re-projecting pixels in the input panorama to an image space corresponding to the target camera pose to obtain re-projection condition information corresponding to the target camera pose; based on the re-projection condition information corresponding to each target camera pose, generating multiple video frames corresponding to each target camera pose through a video generation model, and arranging the multiple video frames according to the order of each target camera pose in the target camera pose sequence to obtain a scene video. The embodiments of the present application can balance the controllability of camera motion, the consistency of scene structure and the integrity of image content of the scene video.

Description

Technical Field

[0001] This application relates to the fields of computer vision and data processing technology, and in particular to video generation methods, apparatus, devices, media, and products based on three-dimensional scenes. Background Technology

[0002] With the development of technologies such as virtual reality, augmented reality, digital twins, immersive content display, indoor and outdoor scene browsing, and intelligent content generation, the demand for generating video content based on images or scene data is constantly increasing. Among related technologies, one approach is to directly generate video from a single image or a small number of images. Another approach is to perform image synthesis or new perspective generation based on 3D scene representation. However, video generation models often struggle to simultaneously ensure the controllability of camera motion, the consistency of scene structure, and the integrity of image content in scene videos.

[0003] Based on this, embodiments of this application provide a video generation method, apparatus, device, medium, and product based on a three-dimensional scene to improve related technologies. Summary of the Invention

[0004] The purpose of this application is to provide a method, apparatus, device, medium, and product for generating video based on a three-dimensional scene, which takes into account the controllability of camera motion, the consistency of scene structure, and the integrity of image content in scene video.

[0005] In a first aspect, embodiments of this application provide a video generation method based on a three-dimensional scene. The method includes: receiving an input panoramic image; determining scene geometric information and a target camera pose sequence corresponding to the input panoramic image, the target camera pose sequence including multiple target camera poses; for each target camera pose, based on the scene geometric information, reprojecting pixels in the input panoramic image to an image space corresponding to the corresponding target camera pose to obtain reprojection condition information corresponding to the target camera pose, the reprojection condition information being used to characterize a reprojected image obtained by reprojecting pixels in the input panoramic image and a target generation region, the target generation region being a region in the image space that has not obtained valid reprojected image content provided by pixels in the input panoramic image; based on the reprojection condition information corresponding to each target camera pose, generating multiple video frames corresponding to each target camera pose through a video generation model, and arranging the multiple video frames according to the order of each target camera pose in the target camera pose sequence to obtain a scene video, wherein the video generation model generates image content in each video frame corresponding to the corresponding target generation region.

[0006] Secondly, embodiments of this application provide a video generation apparatus based on a three-dimensional scene. The apparatus includes: a data receiving module for receiving an input panoramic image; an information determining module for determining scene geometric information and a target camera pose sequence corresponding to the input panoramic image, the target camera pose sequence including multiple target camera poses; and a reprojection module for reprojecting pixels in the input panoramic image to the image space corresponding to the corresponding target camera pose based on the scene geometric information, for each target camera pose, to obtain reprojection condition information corresponding to the target camera pose, the reprojection condition information being used to characterize the input panoramic image. The image space contains a reprojected image obtained by pixel reprojection and a target generation region, wherein the target generation region is a region in the image space that does not obtain effective reprojected image content provided by pixels in the input panoramic image; a video generation module is used to generate multiple video frames corresponding to each of the target camera poses based on the reprojection condition information corresponding to each of the target camera poses, and to arrange the multiple video frames according to the order of each of the target camera poses in the target camera pose sequence to obtain a scene video, wherein the image content corresponding to the corresponding target generation region in each of the video frames is generated by the video generation model.

[0007] Thirdly, embodiments of this application provide a computer device, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of any of the above methods.

[0008] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above methods.

[0009] Fifthly, embodiments of this application provide a computer program product, the computer program product including a computer program, which, when executed by a processor, implements the steps of any of the above methods.

[0010] This application provides a method, apparatus, device, medium, and product for generating video based on a three-dimensional scene. It adopts the concept of converting the target camera pose into reprojection condition information and generating video frames based on the reprojection condition information, thereby taking into account the controllability of camera motion, the consistency of scene structure, and the integrity of image content in scene video. Attached Figure Description

[0011] The embodiments of this application are further described below with reference to the accompanying drawings and specific implementation details.

[0012] Figure 1This is a flowchart illustrating a video generation method based on a three-dimensional scene provided in an embodiment of this application.

[0013] Figure 2 This is a schematic diagram of scene geometry restoration, camera motion path planning, and reprojection condition generation provided in an embodiment of this application.

[0014] Figure 3 This is a schematic diagram illustrating the collaborative processing of reprojection condition information and potential video generation network provided in an embodiment of this application.

[0015] Figure 4 This is a schematic diagram of a long-short mixed training and consistency training provided in an embodiment of this application.

[0016] Figure 5 This is a structural block diagram of a video generation device based on a three-dimensional scene provided in an embodiment of this application.

[0017] Figure 6 This is a structural block diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0018] With the development of technologies such as virtual reality, augmented reality, digital twins, immersive content display, indoor and outdoor scene browsing, and intelligent content generation, the demand for generating video content based on images or scene data is constantly increasing. Among related technologies, one approach is to directly generate video based on a single image or a small number of images, while another approach is to perform image synthesis or generate new perspectives based on 3D scene representation.

[0019] When generating video based on camera motion information, the camera motion information can be represented by velocity information, motion vectors, or motion features obtained by network encoding. However, there is a lack of clear correspondence between this type of abstract motion information and the scene geometry and image pixels from the target's perspective. This makes it difficult to ensure that the camera motion is within a passable area and also makes it difficult to ensure that the generated video frames accurately reflect the position and attitude changes of the target camera.

[0020] Related geometric reprojection methods can generate new perspective images based on scene geometry and target camera pose. However, when the target camera pose changes significantly relative to the input image acquisition pose, the size of the newly exposed area increases, and the reprojection result is prone to pixel loss, image holes, or structural incompleteness. Simply using the reprojection result as the final image output makes it difficult to adequately fill in areas where the input image does not provide valid image content.

[0021] Furthermore, video generation models can be trained using training segments with a fixed trajectory span. When the trajectory span is small, the reprojection results are more complete, but there are fewer samples available for training the content generation capability of the target generation region; when the trajectory span is large, the target generation region is larger, but the known image content is reduced. Training with a single trajectory span makes it difficult to simultaneously ensure both appearance preservation under small viewpoint changes and image content generation capability under large viewpoint changes.

[0022] See Figure 1 , Figure 1 This is a flowchart illustrating a video generation method based on a three-dimensional scene provided in an embodiment of this application.

[0023] To balance the controllability of camera motion, the consistency of scene structure, and the integrity of image content in scene videos, this application provides a video generation method based on a three-dimensional scene, the method including steps S101 to S104.

[0024] Step S101: Receive the input panoramic image.

[0025] Step S102: Determine the scene geometry information and target camera pose sequence corresponding to the input panoramic image, wherein the target camera pose sequence includes multiple target camera poses.

[0026] Step S103: For each target camera pose, based on the scene geometry information, reproject the pixels in the input panoramic image to the image space corresponding to the target camera pose to obtain reprojection condition information corresponding to the target camera pose. The reprojection condition information is used to characterize the reprojected image obtained by reprojecting the pixels in the input panoramic image and the target generation region. The target generation region is the region in the image space that has not obtained effective reprojected image content provided by the pixels in the input panoramic image.

[0027] Step S104: Based on the reprojection condition information corresponding to the poses of each target camera, a plurality of video frames corresponding to the poses of each target camera are generated by a video generation model, and the plurality of video frames are arranged in the order of the poses of each target camera in the target camera pose sequence to obtain a scene video, wherein the image content corresponding to the corresponding target generation region in each video frame is generated by the video generation model.

[0028] The input panoramic image can be a panoramic image used to provide scene appearance content and a basis for spatial analysis. The input panoramic image can include a 360-degree panoramic image represented by an equidistant cylindrical projection method or a panoramic image represented by other spherical projection methods. Each pixel in the image is used to represent the color, texture, and object appearance information in the corresponding viewing direction.

[0029] A pixel can be the basic image unit in an input panoramic image. Each pixel can be associated with pixel location, color information, and information related to its location in the scene space. During reprojection, pixels in the input panoramic image are mapped to 3D space based on scene geometry, and then projected onto the image space corresponding to the target camera's pose.

[0030] Scene geometry information can be used to characterize the spatial structure of the scene corresponding to the input panoramic image, and provides spatial location information for pixel reprojection. Scene geometry information can include depth information, 3D point clouds, scene mesh models, voxel models, or other 3D data that can represent the position and spatial relationships of the scene surface.

[0031] A target camera pose sequence can be an ordered set of multiple target camera poses used to define the motion of a virtual camera in a scene video. The target camera pose sequence can be obtained by sampling the camera motion path or determined based on multiple positions and viewing directions specified by the user. The order of the target camera pose sequence is used to determine the temporal order of multiple video frames.

[0032] The target camera pose can be information used to characterize the position and orientation of a virtual camera when generating a video frame. The target camera pose can include 3D position information and 3D orientation information, or it can be represented by a combination of pose transformation matrix, translation information, and rotation information.

[0033] Image space can be a two-dimensional image coordinate space corresponding to the target camera pose, used to carry the image content formed when observing the scene from that target camera pose. Image space can be a panoramic projection space, a cube mapping space, a fisheye projection space, or a perspective projection space.

[0034] Reprojection is a process that maps pixels in an input panoramic image to a 3D space based on scene geometry, and then projects the mapped pixels into the corresponding image space according to the target camera's pose. Reprojection can convert the abstract target camera position and pose into pixel-level image content corresponding to the target's viewing perspective.

[0035] A reprojected image is an image formed in the image space corresponding to the pose of the target camera after the pixels of the input panoramic image are reprojected. The reprojected image can include complete reprojected regions and pixel-missing regions, where the complete reprojected regions can directly inherit the color and texture information of the input panoramic image.

[0036] Effective reprojected image content can be image content that can be formed in the image space corresponding to the target camera pose by the pixels in the input panoramic image, and that meets the requirements of pixel mapping and visibility. When pixels are not covered, pixel mapping is invalid, or the corresponding scene surface is newly exposed from the target's viewing angle, the corresponding area does not have effective reprojected image content.

[0037] The target generation region can be any region in the image space corresponding to the target camera pose where no effective reprojection image content has been obtained, and where image content needs to be generated by the video generation model. The target generation region can include regions newly exposed due to viewpoint changes, regions not recorded in the input panoramic image, regions with missing reprojection pixels, and regions where the reprojection results do not meet the usage conditions.

[0038] Reprojection condition information can be used to constrain the video generation model to generate video frames, and can characterize the reprojected image and the target generation region. Reprojection condition information can be a combination of the reprojected image and occlusion mask, a combination of the reprojected image and validity map, or conditional features obtained by jointly encoding the reprojected image and the target generation region.

[0039] A video generation model can be a model that generates multiple video frames based on reprojection condition information. The video generation model can preserve the scene appearance that can be determined by the input panoramic image based on the reprojected image, and generate missing image content based on the target generation region, so that the generated video frames have both the target camera pose correspondence and content integrity.

[0040] A video frame can be a single image from a scene video, with each video frame corresponding to a target camera pose in the target camera pose sequence. The region in a video frame corresponding to the reprojected image can retain the image content provided by the input panoramic image, while the image content corresponding to the target generation region is generated by the video generation model.

[0041] Scene video can be a video formed by arranging multiple video frames according to the order of the target camera poses in the target camera pose sequence. Scene video can be panoramic video, cubemap video, multi-view video, or other videos that can show the movement of a virtual camera in a 3D scene.

[0042] In this embodiment, an input panoramic image is first received, and the scene geometry information and target camera pose sequence corresponding to the input panoramic image are determined. Then, for each target camera pose, the pixels in the input panoramic image are reprojected to the corresponding image space using the scene geometry information, forming reprojection condition information that reflects the known image content and the target generation region. Based on the reprojection condition information corresponding to each target camera pose, the video generation model retains the image content that can be determined through reprojection and generates the image content in the target generation region, thereby obtaining multiple video frames corresponding to multiple target camera poses. Finally, the multiple video frames are arranged in the order corresponding to the target camera pose sequence to obtain a scene video that reflects the movement process of the target cameras.

[0043] This embodiment determines the scene geometry information corresponding to the input panoramic image and the target camera pose sequence, giving the video generation process a clear scene space foundation and camera motion basis. By performing pixel reprojection on each target camera pose, the target camera pose is converted into pixel-level conditions corresponding to the target image space, improving the correspondence between video frames and target camera poses. By using reprojection condition information to simultaneously characterize the reprojected image and the target generation region, the video generation model can distinguish between image content that can be determined by the input panoramic image and the image content that needs to be generated. The video generation model generates image content in the target generation region, improving the problem of pixel loss and image holes caused by viewpoint changes in pure reprojection methods. By arranging multiple video frames in the order corresponding to the target camera pose sequence, the motion continuity in the scene video is improved. This embodiment addresses the problem in related video generation methods that lack an explicit geometric correspondence between camera motion control and image content, and the difficulty in filling in missing regions caused by viewpoint changes. It adopts the concept of converting the target camera pose into reprojection condition information and generating video frames based on the reprojection condition information, thereby balancing the controllability of camera motion, the consistency of scene structure, and the integrity of image content in the scene video.

[0044] For example, a panoramic image of a living room with a resolution of 4096×2048 is received, and the scene geometry information corresponding to the living room and a sequence of target camera poses containing 81 target camera poses are determined. The 81 target camera poses are used to represent the process of the virtual camera moving from the entrance of the living room towards the balcony, and the average positional interval between adjacent target camera poses is 0.05 meters.

[0045] For the pose of the 41st target camera, the pixels in the input panoramic image are reprojected onto an image space with a resolution of 2048×1024 to obtain a reprojected image. Since the pose of the 41st target camera has shifted relative to the acquisition position of the input panoramic image, no effective reprojected image content is obtained for the ground behind the sofa and the inside of the door frame; the corresponding target generation area occupies approximately 18% of the image space. The reprojected image and the target generation area are used as the reprojection condition information corresponding to the pose of the 41st target camera and input into the video generation model. The video generation model generates the image content of the ground behind the sofa and the inside of the door frame, resulting in the 41st video frame.

[0046] 81 video frames corresponding to the poses of 81 target cameras were generated in the same way, and the 81 video frames were arranged in the order of the target camera pose sequence to obtain a living room scene video with a frame rate of 10 frames per second and a duration of about 8.1 seconds.

[0047] In some embodiments, determining the scene geometric information corresponding to the input panoramic image may include: determining the depth information corresponding to the pixels in the input panoramic image based on the input panoramic image; and mapping the scene content represented by the input panoramic image to a three-dimensional space based on the pixels in the input panoramic image and the depth information to obtain the scene geometric information.

[0048] Depth information can be used to characterize the spatial distance between the scene location corresponding to a pixel in the input panoramic image and the panoramic image acquisition location. Depth information can include metric depth values, relative depth values, or scale-calibrated depth values ​​for each pixel, or it can be recorded using a depth map with the same or corresponding resolution as the input panoramic image. Depth information can be obtained by processing the input panoramic image using a panoramic depth estimation model.

[0049] Determining the depth information corresponding to a pixel can be a process of estimating the spatial distance of each pixel based on image features such as color, texture, edges, object shapes, and scene layout in the input panoramic image. Methods for determining the depth information corresponding to a pixel can include pixel-by-pixel depth estimation, region-by-region depth estimation, or estimating relative depth first and then performing scale calibration.

[0050] Scene content can be the scene appearance and spatial structure recorded in the input panoramic image. Scene content can include image areas corresponding to the ground, walls, ceiling, furniture and other objects, and can also include the color, texture, boundary and spatial relationship of each image area.

[0051] Mapping scene content to 3D space can be achieved by determining the spatial orientation of pixels based on their positions in the input panoramic image, and then determining the pixel's position in that spatial orientation based on its depth information. This converts the scene content in the 2D image into geometric data in 3D space. The mapping result can be represented as a 3D point cloud composed of multiple 3D points, a scene mesh model formed by connecting 3D points, or a set of 3D coordinates.

[0052] Three-dimensional space can be a range of spatial coordinates used to represent the positional relationships within a scene. This three-dimensional space can use a camera coordinate system with the input panoramic image acquisition location as the origin, or it can use a scene coordinate system obtained through coordinate transformation. A location in three-dimensional space can be represented by three spatial coordinates.

[0053] In this embodiment, the depth information corresponding to each pixel is first determined based on the input panoramic image. Then, the corresponding spatial direction is determined according to the pixel position of each pixel in the input panoramic image. The three-dimensional spatial position corresponding to each pixel is determined by combining the corresponding depth information. In this way, the two-dimensional scene content represented by the input panoramic image is mapped to three-dimensional space to form scene geometric information that can represent the position, shape and spatial relationship of the scene surface.

[0054] This embodiment determines the depth information corresponding to each pixel based on the input panoramic image, enabling each pixel in the 2D panoramic image to obtain corresponding spatial distance information. By combining pixel position and depth information, the scene content is mapped to 3D space, allowing the colors, textures, and object boundaries in the input panoramic image to establish a correspondence with their 3D spatial positions. By determining the mapping result as scene geometric information, a 3D geometric foundation is provided for subsequent image processing based on the scene's spatial structure. This embodiment addresses the problem that a single panoramic image can only directly represent 2D image content and cannot directly provide the 3D spatial structure of the scene. It adopts the concept of mapping panoramic scene content to 3D space based on pixel depth information, thereby realizing the recovery of scene geometric information from the input panoramic image and reducing reliance on pre-constructed 3D scene resources.

[0055] For example, receiving an indoor panoramic image with a resolution of 4096×2048 and performing pixel-by-pixel depth estimation on the input panoramic image yields a depth map with a resolution of 4096×2048. The depth values ​​in the depth map range from 0.6 meters to 10 meters.

[0056] Taking a single pixel in the input panoramic image as an example, the horizontal and vertical azimuth angles corresponding to the pixel are determined based on its horizontal and vertical positions within the image. Then, the pixel's 3D coordinates in 3D space are calculated based on its corresponding depth value of 3.2 meters. Multiple pixels in the input panoramic image are processed in the same way to obtain a 3D point cloud composed of multiple 3D points. This 3D point cloud is then used to determine the scene geometry information corresponding to the input panoramic image.

[0057] In some embodiments, a scene mesh model can be constructed based on a 3D point cloud. Specifically, the 3D point cloud can be triangulated according to the spatial adjacency relationship between the 3D points, connecting adjacent 3D points into multiple triangular patches, and forming a scene mesh model from these triangular patches. The scene mesh model can be used to represent the spatial location and surface shape of the ground, walls, ceiling, furniture surfaces, and other scene structures.

[0058] Triangulation can include connecting adjacent 3D points based on pixel adjacency relationships in the input panoramic image, or determining triangular patches based on spatial distance, normal difference, or surface continuity between 3D points. For adjacent 3D points with depth variations exceeding a set threshold, triangular patches may not be created to reduce erroneous connections across different scene surfaces.

[0059] For example, a 3D point cloud is generated based on a depth map with a resolution of 4096×2048. This 3D point cloud is then triangulated to obtain a scene mesh model comprising approximately 1.5 million triangular faces. Scene geometric information can include at least one of the following: depth map, 3D point cloud, and scene mesh model. The scene mesh model can be used for subsequent traversable area determination, camera occupancy detection, and pixel reprojection.

[0060] In some embodiments, the process of determining the target camera pose sequence may include: determining a passable area based on the scene geometry information and according to passability constraints; generating a camera motion path within the passable area; determining the poses of the plurality of target cameras based on the camera motion path to obtain the target camera pose sequence; wherein the passability constraints may include at least one of collision constraints and clearance constraints.

[0061] Access constraints can be used to limit the position and range of movement of a virtual camera in a 3D scene. Access constraints allow you to exclude areas from the spatial representation of the scene's geometry where the virtual camera cannot safely pass.

[0062] Collision constraints are conditions used to limit the spatial overlap between a virtual camera and walls, furniture, pillars, door frames, and other scene structures. Collision constraints can be implemented using camera dimensions, camera bounding spheres, camera bounding boxes, or the minimum safe distance between the camera and obstacles. Positions that satisfy collision constraints ensure that the virtual camera does not overlap with scene obstacles or meets the set safe distance.

[0063] Clearance constraints are conditions used to limit the height of a virtual camera's location to a sufficient amount of usable space. Clearance constraints can be determined based on the vertical distance between the ground and ceilings, beams, chandeliers, overhead cabinets, or other overhead obstacles. A location that satisfies clearance constraints has vertical space that matches the camera's installation height and the space it occupies, reducing the probability of the camera's movement path entering low-ceilinged spaces or overlapping with overhead obstacles.

[0064] A traversable region can be a 3D scene represented by scene geometry that satisfies traversal constraints and allows a virtual camera to determine its position or generate a camera motion path. A traversable region can be represented as a continuous ground region, a traversable polygon in the navigation mesh, a traversable grid in the occupied grid, or 3D free space. A traversable region can be obtained by excluding collision risk areas and areas with insufficient clearance from candidate ground regions.

[0065] In some embodiments, a navigation grid or occupation grid can be constructed based on a scene mesh model to represent passable areas. Specifically, candidate ground areas can be identified based on the normal direction, spatial height, and connectivity of each triangular facet in the scene mesh model; walls, furniture, columns, door frames, and other non-ground structures are projected onto the candidate ground areas along the direction of gravity to obtain obstacle areas; the obstacle areas are expanded according to the size of the space occupied by the camera and a set safety distance, and the expanded obstacle areas are excluded from the candidate ground areas.

[0066] In some embodiments, the vertical available space above the candidate ground area can be detected, and areas with insufficient clearance can be excluded based on the camera installation height and the height of the space occupied by the camera. After obstacle area exclusion and clearance filtering, the ground area can be divided into polygons or rasterized to obtain a navigation grid or occupancy grid for camera motion path planning.

[0067] For example, if the horizontal radius of the space occupied by the camera is 0.25 meters and the safety distance is set at 0.15 meters, the obstacle area can be expanded according to a comprehensive distance of 0.40 meters; if the camera installation height is 1.55 meters and the height of the space occupied by the camera is 0.30 meters, areas with a vertical clearance of less than 1.85 meters can be identified as areas with insufficient clearance. A navigation mesh is constructed based on the remaining ground area, and the camera motion path is generated within the connected polygons of the navigation mesh.

[0068] A camera motion path can be a path located within a passable area, used to describe the changes in the virtual camera's position. Camera motion paths can include sector exploration paths, circular paths, paths formed by user-specified key locations, random movement paths, or paths leading to a specified scene area. Camera motion paths can be represented as continuous curves, sequentially connected path points, or sequences of passable nodes.

[0069] Generating camera motion paths within a passable area can be achieved by determining the spatial route that allows the virtual camera to move continuously, based on the connectivity between different locations within the passable area. When generating camera motion paths, a start and end point can be selected within the passable area, and a path search can be performed between locations that satisfy collision and clearance constraints; alternatively, camera motion paths can be generated according to the scene exploration range or closed-loop motion requirements.

[0070] Determining the pose of multiple target cameras based on camera motion paths can be a process of determining multiple camera positions along the camera motion path and then determining the camera pose for each camera position. The multiple camera positions can be sampled at set distance intervals, set time intervals, or set number of video frames; the camera pose can be determined based on the extension direction of the camera motion path, the direction between adjacent camera positions, or the target viewing direction.

[0071] In this embodiment, the spatial locations that the virtual camera can pass through are identified based on scene geometric information. Locations that may overlap with the scene structure or have insufficient available space height are excluded based on collision constraints and clearance constraints, thus obtaining a passable area. Subsequently, a continuously moving camera path is generated within the passable area, and multiple camera positions and corresponding camera poses are determined along the camera movement path, thereby obtaining a target camera pose sequence arranged in the order of camera movement.

[0072] This embodiment determines the passable area based on scene geometry information and access constraints, enabling the camera's motion range to match the scene's spatial structure. By utilizing collision constraints to limit the spatial relationship between the camera's occupied space and scene obstacles, the probability of the camera's motion path passing through walls, furniture, or other scene structures can be reduced. By using clearance constraints to filter locations with insufficient available space height, areas where the camera enters are less likely to have low ceilings or overhead obstacles can be minimized. Generating camera motion paths within passable areas improves the spatial executability of these paths. Determining multiple target camera poses along the camera motion path provides continuous camera control information with clearly defined spatial positions for multiple video frames. This embodiment addresses the problem of mismatch between camera motion and scene geometry when generating camera poses solely based on abstract motion information or discrete positions. It adopts a concept of first determining the passable area based on scene geometry information and access constraints, and then generating the camera motion path and target camera pose sequence, thereby improving the geometric compliance and motion executability of the target camera pose sequence.

[0073] For example, the scene geometry represents an indoor scene that is 8 meters long, 6 meters wide, and 3 meters high. The virtual camera's occupied space is set to a spherical space with a radius of 0.25 meters, the minimum safe distance between the virtual camera and walls and furniture is set to 0.15 meters, and the minimum vertical clearance is set to 1.8 meters. After determining passage within the indoor scene based on collision and clearance constraints, a passable area of ​​24 square meters is obtained.

[0074] A 6.4-meter-long camera motion path is generated within the passable area, and 81 camera positions are determined along the camera motion path at 0.08-meter intervals. The camera attitude is determined based on the extension direction of the camera motion path at each camera position, and the 81 target camera poses form a target camera pose sequence.

[0075] In some embodiments, determining the poses of the plurality of target cameras based on the camera motion path may include: generating a candidate camera pose sequence based on the camera motion path, wherein the candidate camera poses in the candidate camera pose sequence include camera position information and camera attitude information; performing continuity processing on the candidate camera pose sequence based on a pose continuity condition to obtain an intermediate camera pose sequence, wherein the continuity processing includes adjusting the candidate camera pose sequence if the position changes and / or attitude changes between adjacent candidate camera poses do not meet the pose continuity condition; detecting whether the camera occupancy space corresponding to each intermediate camera pose in the intermediate camera pose sequence meets the passage constraint based on scene geometry information, and obtaining a detection result; and adjusting, deleting, or replacing intermediate camera poses that do not meet the passage constraint based on the detection result to obtain the target camera pose sequence.

[0076] The candidate camera pose sequence can be an ordered set of multiple candidate camera poses initially generated based on the camera motion path. The candidate camera pose sequence has not yet completed pose continuity processing and passage constraint review, and can be obtained by sampling along the camera motion path at set distance intervals, set time intervals, or set numbers.

[0077] A candidate camera pose can be a camera space state within a sequence of candidate camera poses. The candidate camera pose includes camera position information and camera attitude information, and can be represented by a combination of 3D position and 3D rotation angle, a combination of translation vector and rotation matrix, a combination of translation vector and quaternion, or a pose transformation matrix.

[0078] Camera position information can be information used to characterize the location of a virtual camera in three-dimensional space. Camera position information can include three-dimensional spatial coordinates, or translation information relative to the starting position of the camera's motion path, the location of the input panoramic image acquisition, or the scene reference position.

[0079] Camera attitude information can be used to characterize the virtual camera's viewing direction and spatial orientation. Camera attitude information can include yaw angle, pitch angle, and roll angle, and can also be represented by rotation matrices, quaternions, or rotation vectors.

[0080] Adjacent candidate camera poses can be two candidate camera poses that are sequentially adjacent in the candidate camera pose sequence. Adjacent candidate camera poses can correspond to video frames that are sequentially adjacent in time. The positional and attitude changes between adjacent candidate camera poses are used to characterize the motion amplitude of the virtual camera between adjacent time points.

[0081] Position changes can be the differences between camera positions corresponding to adjacent candidate camera poses. Position changes can be represented by the spatial distance between two camera positions, the path distance along the camera motion path, or the differences in position components in different coordinate directions.

[0082] Attitude changes can be the differences between camera poses corresponding to adjacent candidate camera poses. Attitude changes can be represented by differences in yaw angle changes, pitch angle changes, roll angle changes, relative rotation angles, or rotation matrices.

[0083] Pose continuity conditions can be used to limit position and attitude changes between adjacent candidate camera poses. These conditions may include position changes not exceeding a position change threshold, attitude changes not exceeding an attitude change threshold, differences between adjacent position intervals not exceeding an interval difference threshold, and differences between adjacent attitude changes not exceeding an attitude smoothing threshold. Pose continuity conditions can also be used to make the movement step size and rotation changes between adjacent camera poses approximately uniform.

[0084] Continuity processing can be a process of processing candidate camera pose sequences based on pose continuity conditions. When the candidate camera pose sequence does not meet the pose continuity conditions, continuity processing may include adjusting the candidate camera position, adjusting the candidate camera pose, adding candidate camera poses, merging candidate camera poses that are too close together, or redetermining the sampling interval of the candidate camera poses; when the candidate camera pose sequence already meets the pose continuity conditions, the candidate camera pose sequence can be retained.

[0085] Intermediate camera pose sequences can be camera pose sequences obtained after performing continuity processing on candidate camera pose sequences. Intermediate camera pose sequences can be candidate camera pose sequences that have undergone position or attitude adjustments, or they can be candidate camera pose sequences that have already met the pose continuity condition and are directly retained.

[0086] The camera's occupied space can be used to represent the spatial range occupied by a virtual camera or the object supporting the virtual camera in the camera's pose. The camera's occupied space can be represented by a sphere, cylinder, bounding box, or a three-dimensional geometric shape that matches the camera's shape, and its position and orientation can change according to the camera's position and orientation information.

[0087] The detection results can be information obtained by detecting whether the space occupied by the intermediate camera pose, based on scene geometry information, meets the passage constraints. The detection results can include the judgment result of whether the passage constraints are met or not, and can also include the type of constraint not met, the degree of overlap between the camera space and the scene structure, the distance to obstacles, or the clearance height.

[0088] Adjusting, deleting, or replacing intermediate camera poses can be different ways of processing intermediate camera poses based on detection results. Adjustment can change the camera position information and / or camera orientation information of the intermediate camera pose; deletion can remove intermediate camera poses that are difficult to satisfy passage constraints; replacement can regenerate new camera poses based on adjacent intermediate camera poses, camera motion paths, and scene geometry information to replace intermediate camera poses that do not satisfy passage constraints.

[0089] In this embodiment, firstly, a candidate camera pose sequence including camera position and pose information is generated based on the camera motion path. Then, the candidate camera pose sequence is processed according to the pose continuity condition to obtain an intermediate camera pose sequence with relatively continuous position and pose changes. Subsequently, based on the scene geometry information, it is checked whether the space occupied by each intermediate camera pose meets the passage constraints. Based on the detection results, intermediate camera poses that do not meet the passage constraints are adjusted, deleted, or replaced, finally obtaining a target camera pose sequence that balances pose continuity and spatial accessibility.

[0090] This embodiment generates a candidate camera pose sequence that includes both camera position and pose information based on the camera motion path. This can convert continuous or discrete camera motion paths into camera poses that can control the viewpoint of video frames. By performing continuity processing on the candidate camera pose sequence based on pose continuity conditions, excessive position jumps or abrupt pose changes between adjacent camera poses can be limited, improving the continuity of camera motion. By detecting whether the camera space occupied by each intermediate camera pose meets the passage constraints, camera poses that have resulted in spatial collisions, insufficient obstacle distances, or insufficient clearance after path processing can be identified. By adjusting, deleting, or replacing intermediate camera poses that do not meet the passage constraints, the matching degree between the target camera pose sequence and the scene spatial structure can be improved. This embodiment addresses the problem that discontinuous pose changes and the failure of specific camera poses after path processing to meet scene passage requirements may occur when directly determining camera poses based on camera motion paths. It adopts the concept of first performing pose continuity processing and then re-checking the passage constraints of the camera space occupied, thereby improving the motion smoothness and spatial executability of the target camera pose sequence.

[0091] For example, the camera motion path is 6.4 meters long, and 81 candidate camera poses are generated along the camera motion path. The target position interval is 0.08 meters, and the attitude change threshold between adjacent candidate camera poses is 3 degrees. The position interval between the 30th and 31st candidate camera poses is detected to be 0.17 meters, and the attitude change between the 45th and 46th candidate camera poses is 6 degrees. Therefore, the corresponding candidate camera poses are adjusted so that the interval between adjacent positions is no greater than 0.10 meters and the attitude change between adjacent poses is no greater than 3 degrees, resulting in an intermediate camera pose sequence including 81 intermediate camera poses.

[0092] The camera's occupied space was further set to a spherical space with a radius of 0.25 meters, and the safe distance between the camera's occupied space and the scene structure was set to 0.15 meters. The distance between the camera's occupied space corresponding to the 58th intermediate camera pose and the furniture was detected to be 0.06 meters. Therefore, the 58th intermediate camera pose was moved 0.12 meters away from the furniture to ensure the adjusted camera's occupied space met the passage constraint. After completing the detection and processing of all intermediate camera poses, a target camera pose sequence containing 81 target camera poses was obtained.

[0093] This embodiment uses access constraints when determining traversable areas to pre-exclude areas with collision risks or insufficient clearance from the spatial range represented by scene geometry, narrowing the search range for camera motion paths and improving the efficiency of camera motion path generation. By reusing access constraints after determining the target camera pose sequence based on the camera motion path, a fine review can be performed on the specific camera occupancy space formed after path sampling, pose interpolation, continuity processing, and camera pose changes, identifying camera poses that do not meet access requirements due to path smoothing, position adjustment, or pose changes. By using consistent access constraints in the region-level planning stage and the pose-level review stage, the consistency between the camera motion path planning standard and the target camera pose verification standard can be maintained, reducing the mismatch between the planning results and the actual pose caused by using different judgment standards at different stages. This embodiment addresses the issue that when traversal determination is only performed during the path planning stage, the specific camera pose obtained after path processing may still have spatial conflicts with the scene structure. It adopts a hierarchical constraint concept that reuses traversal constraints in both the traversable area determination stage and the target camera pose determination stage, thereby balancing the efficiency of camera motion path generation and the spatial executability of the target camera pose sequence.

[0094] In some embodiments, the candidate camera pose sequence or the target camera pose sequence can be transformed to a normalized coordinate system. The normalized coordinate system can be based on the first target camera pose in the target camera pose sequence, with the camera position corresponding to the first target camera pose as the origin and the camera orientation corresponding to the first target camera pose as the reference orientation. By transforming the camera poses corresponding to different camera motion paths to a unified normalized coordinate system, the coordinate consistency of camera pose data during reprojection processing and video generation model processing can be improved.

[0095] In some embodiments, the step of reprojecting pixels in the input panoramic image to the image space corresponding to the respective target camera pose based on the scene geometry information for each target camera pose, to obtain reprojection condition information corresponding to the target camera pose, may include: mapping pixels in the input panoramic image to a three-dimensional space based on the scene geometry information; projecting pixels mapped to the three-dimensional space to the image space corresponding to the respective target camera pose for each target camera pose; determining the reprojected image and target generation region corresponding to the target camera pose based on the projection coverage of the pixels in the corresponding image space; and obtaining the reprojection condition information corresponding to the target camera pose based on the reprojected image and the target generation region.

[0096] Mapping pixels from an input panoramic image to 3D space can be a process of determining the 3D spatial position of each pixel based on scene geometry. For example, the spatial orientation of a pixel can be determined based on its position in the input panoramic image, and then the 3D position of the pixel along that spatial orientation can be determined based on its depth, thus establishing a correspondence between the color and texture content in the input panoramic image and its 3D spatial position.

[0097] Projecting pixels mapped to 3D space into image space can be achieved by transforming pixels with 3D spatial positions to 2D positions under the target's viewing angle, based on the target camera's pose. During projection, the same input pixel may correspond to different image positions under different target camera poses; some 3D scene content may also enter or move out of the visible range corresponding to the target camera pose due to changes in the viewing angle.

[0098] Projection coverage can be used to characterize the coverage of pixels in an input panoramic image on the image space corresponding to the target camera pose after projection. Projection coverage can include whether a pixel position in the image space has a projected pixel, whether the projected pixel has valid image content, and the projection coverage range corresponding to different regions.

[0099] Determining the reprojected image and target generation region based on projection coverage can be achieved by distinguishing regions that can obtain effective projection image content from regions that cannot, based on the pixel coverage status in the target image space. Regions with effective projection image content are used to form the reprojected image, while regions without effective projection image content are determined as the target generation region.

[0100] In this embodiment, firstly, based on scene geometric information, pixels in the input panoramic image are mapped to three-dimensional space, establishing a correspondence between the image content in the input panoramic image and the position in three-dimensional space. Then, for each target camera pose, the pixels mapped to three-dimensional space are projected onto the image space corresponding to that target camera pose. Based on the projection coverage, regions that can obtain effective projected image content and regions that cannot are distinguished, thereby determining the reprojected image and the target generation region. Finally, reprojection condition information corresponding to each target camera pose is formed based on the reprojected image and the target generation region.

[0101] This embodiment establishes a correspondence between the input image content and the scene's spatial location by mapping pixels in the input panoramic image to 3D space based on scene geometry information. By projecting pixels in 3D space onto the corresponding image space for each target camera pose, the target camera pose can be converted into pixel-level image content from the target's viewing perspective. By determining the reprojected image and the target generation region based on projection coverage, it's possible to distinguish between image content directly provided by the input panoramic image and image content requiring subsequent generation. By forming reprojection condition information based on the reprojected image and the target generation region, both geometric appearance priors and image generation range from the target's perspective can be provided simultaneously. This embodiment addresses the lack of a pixel-level correspondence between abstract camera motion information and target image content, and the difficulty in directly providing image content for newly visible areas arising from viewpoint changes, by employing a concept of converting the input panoramic image into reprojection condition information through 3D spatial mapping and target perspective projection. This improves the clarity of target camera pose control and provides accurate regional basis for generating image content in the target generation region.

[0102] For example, the input panoramic image has a resolution of 4096×2048, and the target camera pose is translated by 0.8 meters and rotated by 15 degrees relative to the acquisition position of the input panoramic image. Based on the scene geometry information, the pixels in the input panoramic image are mapped to three-dimensional space, and then the pixels mapped to three-dimensional space are projected onto the 2048×1024 image space corresponding to the target camera pose.

[0103] Based on the projection coverage, approximately 82% of the area in the image space was determined to have valid projected image content, and a reprojected image was formed based on this content. The remaining approximately 18% of the area did not have valid projected image content due to changes in the observation position and was identified as the target generation area. The reprojected image and the position of the target generation area in the image space were combined to form the reprojection condition information corresponding to the target camera pose.

[0104] In some embodiments, color information from the input panoramic image can be mapped to corresponding triangular faces or vertices of the scene mesh model to form a textured 3D mesh; alternatively, color information from the input panoramic image can be associated with corresponding 3D points in a 3D point cloud to form a colored 3D point cloud. Textured 3D meshes or colored 3D point clouds can simultaneously represent the 3D position of the scene surface and the color content provided by the input panoramic image.

[0105] For a given target camera pose, a textured 3D mesh or a colored 3D point cloud can be rendered into an equidistant cylindrical projection image space to obtain a reprojected panoramic image. For locations in the equidistant cylindrical projection image space where no valid projected pixels are obtained, corresponding region markers can be generated, and these region markers can be organized into an occlusion mask. The first marker value in the occlusion mask can represent the region with valid reprojected image content, and the second marker value can represent the target generation region where image content needs to be generated by the video generation model.

[0106] For example, the color information of the input panoramic image is mapped to a scene mesh model to form a textured 3D mesh. For a target camera pose that has been translated 1.20 meters relative to the acquisition position of the input panoramic image, the textured 3D mesh is rendered into an equidistant cylindrical projection image space with a resolution of 2048×1024 to obtain a reprojected panoramic image, and an occlusion mask of the same resolution is generated. In the occlusion mask, pixels with a value of 0 represent valid reprojected image content, and pixels with a value of 1 represent the target generation area.

[0107] In some embodiments, the video generation model includes a latent video generation network. The step of generating multiple video frames corresponding to each of the target camera poses based on the reprojection condition information corresponding to each target camera pose may include: encoding the reprojection condition information corresponding to each target camera pose according to the order of the target camera poses in the target camera pose sequence to obtain conditional feature information; obtaining a noise latent representation; performing denoising processing based on the noise latent representation and the conditional feature information through the latent video generation network to obtain a video latent representation; and decoding the video latent representation to obtain the multiple video frames.

[0108] A latent video generation network can be a network in a video generation model that performs video content generation in a latent space. Latent video generation networks can be latent video diffusion networks, stream matching video generation networks, or rectified stream video generation networks. Instead of directly processing full-resolution video frames, latent video generation networks process lower-dimensional latent representations, thereby reducing the computational and storage requirements of the video generation process.

[0109] The order of the target camera pose sequence can be the sequence of multiple target camera poses used to control the camera motion in the scene video. The reprojection condition information corresponding to each target camera pose is arranged in this order to form a condition sequence, so that the condition feature information corresponds to the multiple video frames to be generated in the time dimension. For example, the first target camera pose corresponds to the first reprojection condition information and the first video frame, and the second target camera pose corresponds to the second reprojection condition information and the second video frame.

[0110] Encoding reprojection conditional information can transform it from image space into a feature form suitable for processing by potential video generation networks. The encoding process can extract color, texture, spatial structure, and location features of the target generation region from the reprojected image, and can form temporal features while preserving the sequential relationships between multiple reprojection conditional information pieces. The encoding process can be implemented using 3D variational autoencoders, convolutional coding networks, or conditional projection networks.

[0111] Conditional feature information can be obtained by encoding multiple reprojection conditional information and is used to control the content generated by the latent video generation network. Conditional feature information can include appearance features corresponding to the reprojected image, region features corresponding to the target generation region, and temporal correspondence features between different target camera poses. Conditional feature information can be provided to the latent video generation network through methods such as channel concatenation, cross-attention, adaptive normalization, or additive conditional embedding.

[0112] A noisy latent representation can serve as the initial latent data for denoising processing by a latent video generation network. This representation can consist of random noise sampled according to a defined probability distribution, and includes latent data corresponding to multiple video frames in the temporal dimension. The spatial resolution and feature dimension of the noisy latent representation can be lower than the number of video frames, thus supporting video generation within the latent space.

[0113] Obtaining a noise latent representation can be the process of generating or reading a noise latent representation based on the number of video frames to be generated and the data specifications of the latent space. Acquisition methods can include randomly sampling noise data, reading pre-generated noise data, or generating noise data according to a set random seed.

[0114] Denoising based on noise latent representation and conditional feature information can be seen as a process in which a latent video generation network, under the constraint of conditional feature information, progressively reduces the noise component in the noise latent representation and generates video content features. Conditional feature information enables the denoising process to retain the known image content provided by the reprojection conditional information and generate image content at the location corresponding to the target generation region that matches the semantics and texture of the surrounding scene.

[0115] Video latent representation can be obtained by denoising the noisy latent representation and can be decoded into multiple video frames. The video latent representation can include appearance features, spatial structure features, and temporal variation features corresponding to multiple video frames, and is organized in the order of the target camera pose sequence.

[0116] Decoding a video latent representation can be the process of converting the video latent representation in the latent space into multiple video frames in the image space. This decoding process can be implemented using the decoding portion of a 3D variational autoencoder or other video decoding networks, recovering the low-dimensional features in the video latent representation into image content with the target spatial resolution.

[0117] In some embodiments, a 3D variational autoencoder can be used to jointly encode multiple reprojection condition information arranged in the target camera pose order. The 3D variational autoencoder can extract image content features, region location features, and inter-frame correlation features from the multiple reprojection condition information in the spatial and temporal dimensions to obtain condition feature information.

[0118] During the training of the video generation model, the parameters of the 3D variational autoencoder can be kept constant to ensure that the training video segments, training conditional segments, and video latent representations adopt a relatively stable latent space representation. Conditional feature information can be concatenated with the noisy latent representation along the feature channel dimension, and the concatenation result can be input into the latent video generation network; conditional feature information can also be provided to the latent video generation network through cross-attention, feature modulation, or feature addition.

[0119] For example, the noisy latent representation includes 16 time locations, each corresponding to 32 feature channels; the conditional feature information also includes 16 time locations, each corresponding to 16 feature channels. After concatenation along the feature channel dimension, a network input with 48 feature channels is obtained for each time location. After denoising the network input, the latent video generation network decodes the video latent representation into 16 video frames through the decoding part of the 3D variational autoencoder.

[0120] Multiple video frames can be a sequence of images generated by a video generation model based on reprojection conditions corresponding to the poses of multiple target cameras. Each video frame corresponds to a target camera pose and is arranged in the order of the target camera pose sequence. Multiple video frames can be panoramic video frames, perspective video frames, or video frames with other projection methods.

[0121] In this embodiment, the reprojection condition information corresponding to each target camera pose is encoded according to the order of the multiple target camera poses in the target camera pose sequence, forming conditional feature information that corresponds to the video frames to be generated in time sequence. Then, a noisy latent representation is obtained, and through a latent video generation network, denoising is performed on the noisy latent representation under the constraints of the conditional feature information, gradually converting the noisy latent representation into a video latent representation that includes scene appearance, target generation region content, and inter-frame variation relationships. Finally, the video latent representation is decoded to obtain multiple video frames corresponding to the multiple target camera poses.

[0122] This embodiment encodes multiple reprojection condition information in the order of the target camera pose sequence, thus maintaining the temporal correspondence between the conditional feature information and the poses of multiple target cameras and multiple video frames. By encoding the reprojection condition information into conditional feature information, the known image content and target generation region from the target viewpoint can be converted into generation conditions that the latent video generation network can utilize. By denoising the noisy latent representation under the constraints of the conditional feature information, the latent video generation network can generate image content corresponding to the target generation region while maintaining the scene appearance and pose control information provided by the reprojected image. By performing denoising in the latent space, the computational overhead of directly generating video in the full-resolution image space can be reduced. By decoding the video latent representation, multiple video frames corresponding to the poses of multiple target cameras in temporal order can be obtained. This embodiment addresses the problem that reprojection condition information is difficult to be directly and effectively utilized by video generation networks, and that the target generation region in pure reprojection results lacks image content. It adopts the concept of encoding ordered reprojection condition information into conditional feature information and constraining latent spatial denoising generation, thereby improving the correspondence between video frames and target camera poses, and improving the image integrity of the target generation region and the temporal consistency between multiple video frames.

[0123] For example, the target camera pose sequence includes 81 target camera poses, each corresponding to a reprojection condition information with a resolution of 2048×1024. The 81 reprojection condition information are encoded according to the order of the 81 target camera poses to obtain conditional feature information including 81 time positions.

[0124] Noise latent representations corresponding to 81 video frames are obtained. A latent video generation network is then used to perform denoising based on these noise latent representations and conditional feature information to obtain the video latent representations. These video latent representations are then decoded to obtain 81 video frames with a resolution of 2048×1024, each corresponding to a target camera pose in the target camera pose sequence.

[0125] In some embodiments, the step of denoising the video using the latent video generation network based on the noise latent representation and the conditional feature information to obtain a video latent representation may include: determining multiple denoising steps and denoising step size information corresponding to each denoising step; determining the noise latent representation as the input latent representation corresponding to the first denoising step; according to the execution order of the multiple denoising steps, for each denoising step, obtaining denoising prediction information corresponding to the corresponding denoising step through the latent video generation network based on the input latent representation, the conditional feature information, noise time information, and denoising step size information corresponding to the corresponding denoising step; updating the input latent representation corresponding to the corresponding denoising step based on the denoising prediction information and denoising step size information corresponding to the corresponding denoising step to obtain the output latent representation corresponding to the corresponding denoising step, and determining the output latent representation as the input latent representation corresponding to the next denoising step; determining the output latent representation corresponding to the last denoising step as the video latent representation; wherein, in each denoising step, only denoising prediction based on the conditional feature information is performed, and denoising prediction not based on the conditional feature information is not performed.

[0126] A denoising step can be a processing unit in which the latent video generation network performs a denoising prediction and a latent representation update on the latent representation. Multiple denoising steps are executed in a predetermined order, with the output latent representation of the previous denoising step serving as the input latent representation of the next denoising step. The number of denoising steps can be determined based on video generation efficiency and quality, and can be set to two, four, eight, or other numbers.

[0127] The denoising step size can represent the range of noise state changes corresponding to a denoising step or information that potentially represents the update scale. The denoising step size can be determined based on the difference between adjacent noise times or based on a pre-defined denoising time series. A larger denoising step size can reduce the number of denoising steps required to complete the denoising process, and the potential video generation network can adjust the denoising prediction results based on the denoising step size information.

[0128] The first denoising step can be the first denoising step to be executed among multiple denoising steps. The first denoising step receives the noise latent representation and uses it as the input latent representation for the first denoising step to initiate the stepwise denoising process.

[0129] The input latent representation can be the latent data received and updated by a denoising step. The input latent representation for the first denoising step is the noisy latent representation, and the input latent representation for subsequent denoising steps is the output latent representation obtained from the previous denoising step.

[0130] Noise temporal information can be used to characterize the current noise state of the input latent representation. Noise temporal information can be represented as continuous time values, discrete time steps, noise levels, or signal-to-noise ratio parameters. The latent video generation network can determine the noise level contained in the current input latent representation based on the noise temporal information and output denoising prediction information that matches the current noise state.

[0131] Denoising prediction based on input latent representation, conditional feature information, noise temporal information, and denoising step size information can be achieved by a latent video generation network (LDN) comprehensively considering current latent data, video generation conditions, current noise state, and the current update scale to predict the direction of change in the input latent representation or the noise removal process. By simultaneously inputting noise temporal information and denoising step size information, the LDN can generate corresponding denoising prediction information for different noise states and update scales.

[0132] Denoising prediction information can be the output of a latent video generation network for a denoising step, used to update the input latent representation. Denoising prediction information can include velocity field information, noise prediction information, latent representation change amount, or target latent representation prediction information. In stream-matching video generation, denoising prediction information can be velocity field information representing the direction and rate of change of the latent representation.

[0133] Updating the input latent representation based on denoising prediction information and denoising step size information can be a process of calculating the output latent representation based on the direction of change of the latent representation predicted by the latent video generation network and the corresponding update scale. The update method can include adding the product of denoising prediction information and denoising step size information to the input latent representation, or using other update methods that can advance the state of the latent representation based on denoising prediction information.

[0134] The output latent representation can be the latent data obtained after updating the input latent representation of a denoising step. Except for the last denoising step, the output latent representation corresponding to a denoising step is determined as the input latent representation corresponding to the next denoising step, thus forming a continuous iterative process of latent representation.

[0135] Denoising prediction based on conditional feature information can be a denoising prediction performed by a potential video generation network upon receiving conditional feature information. This denoising prediction leverages features corresponding to the reprojected image and the target generation region to ensure that the video content maintains a correspondence with the target camera pose and scene appearance.

[0136] Denoising prediction not based on conditional feature information can be a denoising prediction performed by the latent video generation network when it does not receive conditional feature information or when the conditional feature information is set to zero. The relevance classifier-guided approach can perform denoising prediction based on conditional feature information and denoising prediction not based on conditional feature information separately, and then combine the two denoising prediction results.

[0137] Performing denoising prediction based solely on conditional feature information means directly using the conditional feature information to perform denoising prediction once in each denoising step, without additional denoising prediction not based on conditional feature information. This approach reduces the number of calls to the potential video generation network corresponding to each denoising step and makes the denoising prediction results directly constrained by the reprojection conditions.

[0138] In this embodiment, multiple denoising steps and their corresponding denoising step sizes are first determined, and the noise latent representation is used as the input latent representation for the first denoising step. Subsequently, following the execution order of the multiple denoising steps, in each denoising step, a latent video generation network is used to obtain denoising prediction information based on the input latent representation, conditional feature information, noise temporal information, and denoising step size information. The input latent representation is then updated based on the denoising prediction information and the denoising step size information, and the resulting output latent representation is passed to the next denoising step. Only denoising prediction based on conditional feature information is performed in each denoising step, and finally, the output latent representation of the last denoising step is determined as the video latent representation.

[0139] This embodiment determines denoising step size information for each of the multiple denoising steps, enabling the latent video generation network to perform corresponding denoising predictions based on the update scale of each denoising step. By using the output latent representation of the previous denoising step as the input latent representation of the next denoising step, the noise latent representation can be progressively transformed into a video latent representation that can characterize the video content. By simultaneously generating denoising prediction information based on the input latent representation, conditional feature information, noise temporal information, and denoising step size information, the matching degree between the denoising prediction and the current noise state, update scale, and target camera pose conditions can be improved. By performing only denoising prediction based on conditional feature information and not denoising prediction without conditional feature information, the number of network predictions in each denoising step can be reduced, lowering the computational cost of the video generation process. This embodiment addresses the problem that related video generation methods require many denoising steps, and each denoising step requires separate conditional and unconditional predictions, leading to a large inference computation. It adopts the concept of conditionalizing the denoising step size and retaining only conditional denoising prediction, thereby improving the inference efficiency of the latent video generation network while maintaining the reprojection conditional constraint effect.

[0140] For example, the latent noise representations for 81 video frames are obtained, and four denoising steps are set. The noise time information corresponding to the four denoising steps are 1.00, 0.75, 0.50, and 0.25, respectively, and the denoising step size information corresponding to each denoising step is 0.25.

[0141] In the first denoising step, the noise latent representation is used as the input latent representation. The latent video generation network obtains the first denoising prediction information based on the input latent representation, conditional feature information, noise temporal information with a value of 1.00, and denoising step size information with a value of 0.25, and accordingly obtains the first output latent representation. The remaining three denoising steps are performed sequentially in the same manner, with each step performing only denoising prediction based on conditional feature information. The output latent representation obtained in the fourth denoising step is determined as the video latent representation.

[0142] In some embodiments, the Euler update method can be used to update the input latent representation corresponding to a denoising step. The Euler update method can determine the change in the input latent representation in the corresponding denoising step based on the product of the denoising prediction information and the denoising step size information, and obtain the output latent representation based on the change.

[0143] In each denoising step, the guiding weight for the unguided classifier can be set to 1, performing only denoising prediction based on conditional feature information, without additional denoising prediction not based on conditional feature information. Therefore, each denoising step only needs to call the latent video generation network once.

[0144] For example, a four-step denoising process is used to generate the latent representation of a video. Each denoising step performs only one forward pass of the latent video generation network based on conditional feature information, resulting in a total of four forward passes for the latent video generation network. In contrast, performing two denoising predictions—one based on conditional feature information and one without—in ten denoising steps requires a total of 20 forward passes for the latent video generation network. Therefore, the four-step single-branch denoising method reduces the number of forward passes for the latent video generation network from 20 to 4, and also reduces the computational and storage requirements of the unconditional prediction branch.

[0145] In some embodiments, the video generation model can be trained by: receiving training data, the training data including a training camera pose sequence, a training video sequence corresponding to the training camera pose sequence, and a training reprojection condition sequence, the training reprojection condition sequence including multiple training reprojection condition information corresponding to multiple training camera poses in the training camera pose sequence; determining multiple training pose segments with different trajectory spans from the training camera pose sequence; determining training video segments and training condition segments corresponding to each of the training pose segments from the training video sequence and the training reprojection condition sequence; and training the video generation model based on each of the training video segments and the corresponding training condition segments, so that training pose segments with different trajectory spans and different target generation region sizes participate in the training of the video generation model.

[0146] Training data can be a dataset used to train a video generation model. Training data includes training camera pose sequences, training video sequences, and training reprojection condition sequences. These three sequences have a temporal and positional correspondence, enabling a training camera pose, a training video frame, and a training reprojection condition to correspond to the same observation time or the same camera viewpoint.

[0147] The training camera pose sequence can be multiple training camera poses arranged in the order of camera movement. The training camera pose can be information representing the camera position and orientation corresponding to a training video frame. The training video sequence can be a set of video frames corresponding to the training camera pose sequence. The training reprojection condition sequence can be multiple training reprojection condition information arranged in the pose order of the training camera pose sequence. The training reprojection condition information can be generation condition information formed for a single training camera pose.

[0148] The trajectory span can be used to characterize the range of camera motion corresponding to a training pose segment. The trajectory span can be represented by the number of camera poses included in the training pose segment, the cumulative camera movement distance, the distance between the starting and ending camera poses, the corresponding video duration, or the cumulative change in camera pose. Training pose segments with smaller trajectory spans correspond to shorter distances of camera motion, while training pose segments with larger trajectory spans correspond to longer distances of camera motion.

[0149] Training pose segments can be continuous pose subsequences extracted from training camera pose sequences. Multiple training pose segments can have different trajectory spans to represent short-range, medium-range, and long-range camera motions, respectively. By extracting training segments of different spans from the complete trajectory, both short-range and long-range segments can be used together in the training of the video generation model.

[0150] Training video segments can be subsequences of video frames determined from a training video sequence, corresponding to a training pose segment. Training video segments and training pose segments can use the same start and end numbers, so that each training video frame in a training video segment corresponds to a training camera pose in a training pose segment.

[0151] Training conditional segments can be subsequences of conditional information determined from the training reprojection conditional sequence, corresponding to a training pose segment. The training conditional segments and training video segments correspond to each other in terms of frame count and arrangement, enabling the video generation model to learn and generate corresponding training video frames based on the training reprojection conditional information corresponding to each training camera pose.

[0152] The target generation region size can be used to characterize the size of the region whose image content needs to be generated by the video generation model in the training reprojection condition information. The target generation region size can be represented by the number of pixels in the target generation region, its area, the proportion of its area to the image space, the number of connected components, or the area of ​​the largest connected component. When the trajectory span is small, the target generation region caused by the viewpoint change can be small; when the trajectory span is large, the newly exposed target generation region due to the viewpoint change can be large.

[0153] Training pose segments with different trajectory spans and target generation region sizes participate in the training together. This means that the training samples of the video generation model simultaneously include training samples corresponding to multiple camera motion ranges and multiple target generation region sizes. Different training pose segments can participate in model parameter updates separately in different training iterations, or they can be organized into the same training batch to participate in model parameter updates together, so that the video generation model can receive training supervision corresponding to both short-distance and long-distance motion conditions.

[0154] In some embodiments, for a single training iteration, a training scene can be randomly selected from multiple scenes included in the training data, and the target trajectory span can be uniformly and randomly determined within a preset range of trajectory span. Subsequently, the starting position of a segment is randomly determined from the training camera pose sequence corresponding to the training scene, and continuous training pose segments are extracted according to the target trajectory span.

[0155] Uniformly and randomly determining the target trajectory span can be achieved by ensuring that different trajectory spans within a preset range have the same or approximately the same probability of being selected. The trajectory span can be represented by the number of training camera poses, the number of video frames, the cumulative movement distance, or the video duration. Therefore, training pose segments of different scales can participate in the video generation model training at a relatively balanced frequency across multiple training iterations.

[0156] In some embodiments, the training data may further include a training input panorama. The training reprojection condition sequence may be pre-generated based on the training input panorama, the training scene geometry corresponding to the training input panorama, and the training camera pose sequence, so that the training reprojection condition information corresponds to the training camera pose and the training video frames.

[0157] For example, the frame range of the training pose segment is set to 8 to 48 frames. In one training iteration, 32 frames are uniformly and randomly sampled from 8 to 48, and the 20th training camera pose is randomly selected as the segment start position from the training camera pose sequence containing 81 training camera poses. The training pose segment consists of the 20th to the 51st training camera poses. Training video segments and training conditional segments are determined from the training video sequence and the training reprojection conditional sequence according to the same sequence number range.

[0158] In this embodiment, corresponding training camera pose sequences, training video sequences, and training reprojection condition sequences are received. Multiple training pose segments with different trajectory spans are determined from the training camera pose sequences. Based on the position range of each training pose segment within the training camera pose sequences, corresponding training video segments and training condition segments are extracted from the training video sequences and training reprojection condition sequences, respectively. Subsequently, a video generation model is trained based on the corresponding training video segments and training condition segments, allowing training samples corresponding to different camera motion ranges and different target generation region sizes to jointly influence the parameter learning of the video generation model.

[0159] This embodiment improves the temporal alignment accuracy between camera pose, generation conditions, and reference video content by maintaining correspondence between the training camera pose sequence, training video sequence, and training reprojection condition sequence. By identifying training pose segments with different trajectory spans from the training camera pose sequence, the camera motion range covered by the video generation model training can be expanded. By determining corresponding training video segments and training condition segments for each training pose segment, matching generation conditions and reference video content can be provided for different camera motion conditions. By having training pose segments of different target generation region sizes participate in training together, the video generation model can simultaneously learn the image content preservation ability when the target generation region is small and the image content generation ability when the target generation region is large. This embodiment addresses the problem that training with a single trajectory span is difficult to simultaneously adapt to different camera motion ranges and different target generation region sizes. It adopts the concept of training pose segments with multiple trajectory spans and their corresponding training video segments and training condition segments together, thereby improving the video generation model's adaptability to different camera motion scales and balancing appearance preservation under smaller target generation regions and content generation under larger target generation regions.

[0160] For example, the training camera pose sequence includes 81 training camera poses, the training video sequence includes 81 training video frames corresponding to the 81 training camera poses, and the training reprojection condition sequence includes 81 training reprojection condition information corresponding to the 81 training camera poses.

[0161] Three training pose segments, each consisting of 9, 25, and 49 consecutive training camera poses, were identified from the training camera pose sequence. The cumulative camera movement distances corresponding to these three training pose segments were 0.4 meters, 1.2 meters, and 2.4 meters, respectively. Based on the sequence number range corresponding to the three training pose segments, corresponding training video segments and training condition segments were determined from the training video sequence and the training reprojection condition sequence, respectively.

[0162] The target generation region area ratios for the three training pose segments are approximately 5%, 18%, and 36%, respectively. A video generation model is trained based on these three training video segments and corresponding training condition segments, allowing training samples with different trajectory spans and target generation region sizes to participate in the parameter updates of the video generation model.

[0163] In some embodiments, the video generation model includes a latent video generation network. Training the video generation model based on each training video segment and a corresponding training condition segment may include: encoding the training video segments to obtain a training video latent representation; generating a training input latent representation and supervised denoising information based on the training video latent representation, noise information, and training noise temporal information; encoding the training condition segments to obtain training conditional feature information; for each training iteration, selecting a target training method from supervised denoising training and consistency training methods, and determining the training denoising step size information corresponding to the target training method; when the supervised denoising training method is selected, performing a training denoising step through the latent video generation network based on the training input latent representation, the training conditional feature information, the training noise temporal information, and the training denoising step size information to obtain training denoising prediction information, and based on the training denoising prediction information and the supervised denoising information... The difference between the training and reference denoising predictions is used to determine the supervised denoising training loss. When the consistency training method is selected, multiple reference denoising steps are performed using a reference latent video generation network based on the training input latent representation, the training conditional feature information, the reference noise time information, and the reference denoising step size information to obtain reference denoising prediction information. Then, using the latent video generation network, training denoising steps are performed based on the training input latent representation, the training conditional feature information, the training noise time information, and the training denoising step size information to obtain training denoising prediction information. The consistency training loss is determined based on the difference between the training denoising prediction information and the reference denoising prediction information. The number of training denoising steps is less than the number of reference denoising steps. The reference latent video generation network is either a parameter-frozen teacher network or the latent video generation network in a gradient-free mode. The video generation model is updated based on the supervised denoising training loss or the consistency training loss corresponding to the target training method.

[0164] Encoding training video segments can be a process of transforming them from image space to latent space. The encoding result can preserve the scene appearance, spatial structure, and temporal variation features of the training video segments, while reducing the data dimensionality processed by subsequent noise addition and denoising prediction.

[0165] The training video latent representation can be the latent data obtained by encoding training video segments. The training video latent representation characterizes the real video content in the training video segments and serves as the basis for generating training input latent representations and supervised denoising information.

[0166] Noise information can be used to add perturbations to the latent representation of the training video. The noise information can be random noise sampled according to a predetermined probability distribution, such as standard Gaussian noise. Different training iterations can use different noise information so that the latent video generation network learns to handle different noise samples.

[0167] Training noise temporal information can be used to characterize the noise state corresponding to the training input latent representation. The training input latent representation can be generated based on the training video latent representation, noise information, and training noise temporal information, and then input into the noisy latent data of the latent video generation network.

[0168] Supervised denoising information can be target information determined based on the latent representation and noise information of the training video, used to supervise the output of the latent video generation network. The supervised denoising information and the trained denoising prediction information use a matching data representation format. The supervised denoising information can be target velocity field information, target noise information, or target latent representation information; in the stream matching training method, the supervised denoising information can be the direction of change between the noise information and the latent representation of the training video.

[0169] Encoding the training condition segments can transform them into a feature format that the potential video generation network can utilize. The encoding process can extract appearance features corresponding to the reprojected image, region features corresponding to the target generation region, and temporal sequence features among multiple training reprojection condition information.

[0170] Training conditional features can be model input information obtained by encoding training condition segments. Training conditional features have the same data meaning and model input function as conditional features in the inference phase, used to constrain the output of the potential video generation network to match the reprojection conditions with denoising prediction information.

[0171] A training iteration can be a training process in which the video generation model performs training data processing, loss calculation, and model parameter updates once. Different training iterations can choose different training objectives, allowing supervised denoising training and consistency training to participate in the video generation model training respectively.

[0172] Supervised denoising training involves comparing the trained denoised predictions from the latent video generation network with supervised denoising information, and training the video generation model based on the differences between the two. Supervised denoising training is used to maintain the latent video generation network's ability to learn the basic denoising objective.

[0173] Consistency training can utilize reference denoising prediction information obtained from multiple reference denoising steps to supervise the latent video generation network in obtaining similar prediction results through fewer training denoising steps. Consistency training enables the latent video generation network to learn to complete the prediction process, which originally required multiple smaller denoising steps, with a larger denoising step size.

[0174] The target training method can be selected from supervised denoising training and consistency training methods in a single training iteration. The target training method can be randomly selected according to a set probability, or it can be selected according to a set training plan, so that the two training methods alternately participate in model training in different training iterations.

[0175] The training denoising step size information can correspond to the target training method and is used to characterize the state progression scale when the potential video generation network performs a training denoising step. The training denoising step size information has the same data meaning as the denoising step size information during the inference phase. In the consistency training method, the training denoising step size can be larger than the reference denoising step size, enabling the potential video generation network to learn denoising prediction capabilities with larger step sizes.

[0176] The training denoising step can be a processing unit in which the latent video generation network obtains training denoising prediction information based on a single input. A training denoising step takes the training input latent representation, training conditional feature information, training noise temporal information, and training denoising step size information as inputs, and outputs training denoising prediction information.

[0177] The training denoising prediction information can be the information output by the latent video generation network based on the training input latent representation, training conditional feature information, training noise temporal information, and training denoising step size information. The training denoising prediction information can be predicted velocity field information, predicted noise information, or predicted latent representation information, and it adopts a matching data representation form with the supervised denoising information or reference denoising prediction information.

[0178] Supervised denoising training loss can be a measure of the difference between the training denoised prediction information and the supervised denoised information. Supervised denoising training loss can take the form of squared difference, absolute difference, or other loss methods that can measure the difference between the prediction result and the supervised target, and is used to update the video generation model.

[0179] A reference latent video generation network can be a network used to generate reference denoising prediction information in a consistency training manner. The reference latent video generation network can be a teacher network with frozen parameters or a latent video generation network in a gradient-free mode. The reference latent video generation network does not update its parameters based on the consistency training loss when generating reference denoising prediction information, thus providing relatively stable training supervision for the latent video generation network.

[0180] A parameter-frozen teacher network can be a reference latent video generation network whose model parameters remain unchanged during the reference denoising prediction process. The parameter-frozen teacher network can have the same or corresponding network structure as the latent video generation network and provide reference prediction results through multiple reference denoising steps.

[0181] A latent video generation network in gradient-free mode refers to a latent video generation network that does not record gradient information for backpropagation when performing a reference denoising step. A latent video generation network in gradient-free mode can be used as a reference latent video generation network, while restricting the reference prediction branch from participating in parameter updates.

[0182] Reference noise timing information can be used to characterize the noise state corresponding to the input potential in the reference denoising step. As multiple reference denoising steps are executed sequentially, the reference noise timing information can be updated according to the reference denoising step size of each reference denoising step.

[0183] Reference denoising step size information can be used to characterize the state progression scale of the reference latent video generation network in a reference denoising step. Multiple reference denoising steps can employ a reference denoising step size smaller than the training denoising step size to achieve finer-grained latent representation updates.

[0184] A reference denoising step can be a processing unit in which a reference latent video generation network performs a reference denoising prediction and a latent representation update. Multiple reference denoising steps are executed sequentially with the trained input latent representation as the initial input latent representation, and the output of the previous reference denoising step is used as the input of the next reference denoising step.

[0185] Reference denoising prediction information can be the reference result obtained by the latent video generation network through multiple reference denoising steps, used for supervised training of denoising prediction information. Reference denoising prediction information can be determined based on the prediction information obtained from multiple reference denoising steps, such as combining or averaging multiple predicted velocity field information.

[0186] In the consistency training method, the number of training denoising steps is less than the number of reference denoising steps, and a training denoising step size larger than the reference denoising step size can be used to learn to approximate the prediction results of multiple reference denoising steps with fewer steps.

[0187] Consistency training loss is a measure of the difference between the trained denoised prediction and the reference denoised prediction. It is used to constrain the potential video generation network to obtain predictions similar to those obtained from multiple reference denoising steps with fewer training denoising steps, thus providing a training basis for fewer-step inference.

[0188] Updating the video generation model can be a process of adjusting the trainable parameters in the model based on the training loss corresponding to the target training method. When choosing supervised denoising training, the video generation model is updated based on the supervised denoising training loss; when choosing consistency training, the video generation model is updated based on the consistency training loss.

[0189] In some embodiments, the video generation model may include a 3D variational autoencoder and a latent video generation network. The 3D variational autoencoder can be used to encode video data and reprojection conditional information, and to decode the latent video representation. The latent video generation network may include a video generation backbone network, and low-rank adapters, conditional projection layers, and stride embedding heads associated with the video generation backbone network.

[0190] A low-rank adapter can be placed in one or more layers of the video generation backbone network to adjust the processing results of the corresponding network layer through a low-rank parameter matrix. A conditional projection layer can be used to transform conditional feature information to a feature space that matches the input or intermediate features of the video generation backbone network. A stride embedding head can be used to encode denoising stride information to obtain stride conditional information characterizing the scale of denoising state progression.

[0191] When training the video generation model, the parameters of at least one of the low-rank adapter, conditional projection layer, and stride embedding head can be updated based on supervised denoising training loss or consistency training loss, while keeping the parameters of the video generation backbone network and the 3D variational autoencoder unchanged. By keeping the parameters of the video generation backbone network unchanged, the pre-learned video content generation capability of the video generation backbone network can be preserved; by keeping the parameters of the 3D variational autoencoder unchanged, the representational stability of the training video segments, training conditional segments, and generated videos in the latent space can be maintained.

[0192] For example, during a backpropagation process, the parameter gradients of the low-rank adapter, conditional projection layer, and stride embedding head can be calculated, and the parameters of the low-rank adapter, conditional projection layer, and stride embedding head can be updated based on the calculated parameter gradients. The video generation backbone network and the 3D variational autoencoder can participate in forward processing, but the parameter gradients used for parameter updates are not calculated or retained, so that the parameters of the video generation backbone network and the 3D variational autoencoder remain unchanged during training.

[0193] In other embodiments, the parameters of at least one of the video generation backbone network and the 3D variational autoencoder may be kept unchanged, and at least some of the parameters of the other may be updated according to training requirements.

[0194] In this embodiment, training video segments are first encoded, and training input latent representations and supervised denoising information are generated based on the training video latent representation, noise information, and training noise temporal information. Simultaneously, training conditional segments are encoded to obtain training conditional feature information. For each training iteration, a target training method is selected from supervised denoising training and consistency training, and the corresponding training denoising step size is determined. Under supervised denoising training, the latent video generation network outputs training denoising prediction information based on the training input latent representation, training conditional feature information, training noise temporal information, and training denoising step size, and calculates the supervised denoising training loss with the supervised denoising information. Under consistency training, the reference latent video generation network obtains reference denoising prediction information through multiple reference denoising steps, and the latent video generation network obtains training denoising prediction information through fewer training denoising steps, calculating the consistency training loss based on the difference between the two predictions. Finally, the video generation model is updated according to the training loss corresponding to the target training method.

[0195] This embodiment encodes training video segments and training condition segments separately, ensuring that the latent representation of the training input and the training condition feature information maintain data type correspondence with the model input during the inference stage. By generating latent representations of the training input and supervised denoising information based on the latent representation of the training video, noise information, and training noise temporal information, the latent video generation network can be provided with noisy inputs and supervised targets that match the noisy inputs. By selecting a target training method between supervised denoising training and consistency training, the ability to denoise in fewer steps can be trained while maintaining basic denoising prediction capabilities. By performing multiple reference denoising steps with reference to the latent video generation network, finer-grained reference denoising prediction information can be obtained. By performing fewer training denoising steps with the latent video generation network than reference denoising steps and using consistency training loss to constrain the training of denoising prediction information, the latent video generation network can learn to approximate the prediction results of multiple reference steps in fewer steps. By using a parameter-frozen teacher network or a latent video generation network in a gradient-free mode to generate reference denoising prediction information, the stability of reference supervision can be improved, and the participation of the reference prediction branch in model parameter updates can be limited. This embodiment addresses the problem that prediction accuracy tends to decrease when video generation models use fewer denoising steps, and that it is difficult to obtain large-step prediction capabilities using only basic supervised training. It adopts a hybrid training concept that combines supervised denoising training with multi-reference step supervision of fewer training steps, thereby balancing the basic denoising capability and the ability of video generation models to perform fewer inference steps.

[0196] For example, in one training iteration, the training video segment includes 16 training video frames. The training video segment is encoded to obtain the training video latent representation; noise information and training noise temporal information with a value of 0.6 are sampled to generate the training input latent representation and supervised denoising information. The corresponding training condition segments are encoded to obtain training condition feature information.

[0197] Supervised denoising training is selected with a probability of 0.75, and consistency training is selected with a probability of 0.25. When selecting the supervised denoising training method, the latent video generation network outputs training denoising prediction information based on the latent representation of the training input, training conditional feature information, training noise temporal information, and training denoising step size information. It then calculates the supervised denoising training loss based on the difference between the training denoising prediction information and the supervised denoising information, and updates the video generation model based on the supervised denoising training loss.

[0198] When selecting the consistency training method, the training denoising step size of 0.20 is divided into two reference denoising step sizes of 0.10. The reference latent video generation network performs two reference denoising steps and obtains reference denoising prediction information, while the latent video generation network performs one training denoising step and obtains training denoising prediction information. The consistency training loss is then calculated based on the difference between the training denoising prediction information and the reference denoising prediction information, and the video generation model is updated based on the consistency training loss.

[0199] In some embodiments, the step of performing a training denoising step through the latent video generation network based on the training input latent representation, the training conditional feature information, the training noise temporal information, and the training denoising step size information to obtain training denoising prediction information may include: encoding the training denoising step size information to obtain step size conditional information; injecting the step size conditional information into the temporal conditional path of the latent video generation network; and obtaining the training denoising prediction information through the latent video generation network based on the training input latent representation, the training conditional feature information, the training noise temporal information, and the step size conditional information.

[0200] Encoding the training denoising step size information can involve converting the numerical or discrete form of the training denoising step size information into a feature form that the potential video generation network can process. Encoding methods can include positional encoding, frequency encoding, neural network mapping, or lookup table mapping, so that different training denoising steps correspond to different feature representations.

[0201] Step size conditional information can be a conditional feature obtained by encoding the training denoising step size information. Step size conditional information is used to explicitly indicate to the potential video generation network the state advancement scale corresponding to the current training denoising step, enabling the potential video generation network to adjust the training denoising prediction information for smaller or larger training denoising step sizes.

[0202] Temporal conditional pathways can be feature processing pathways in a latent video generation network used to convey conditions related to denoising time and the denoising process. Temporal conditional pathways may include temporal coding layers, feature projection layers, and connection structures that provide temporal conditions to the network processing layers. Training noise temporal information and stride conditional information can be encoded separately, fused, and then provided to multiple processing layers in the latent video generation network.

[0203] Injecting step-size conditional information into the temporal conditional pathway can be achieved by fusing the step-size conditional information with the temporal features corresponding to the training noise temporal information. This allows the latent video generation network to simultaneously perceive the current noise state and the denoising span of the current training denoising step. Fusion methods can include feature addition, feature concatenation, linear projection followed by addition, or feature fusion based on conditional modulation. Through this process, the latent video generation network can learn to adjust its denoising predictions according to a given denoising step size.

[0204] In this embodiment, the training denoising step size information is first encoded to obtain step size conditional information that characterizes the denoising span corresponding to the current training denoising step. Then, the step size conditional information is injected into the temporal conditional path of the latent video generation network, enabling the latent video generation network to simultaneously obtain the current noise state represented by the training noise temporal information and the state progression scale represented by the step size conditional information. Subsequently, the latent video generation network performs training denoising prediction based on the training input latent representation, training conditional feature information, training noise temporal information, and step size conditional information, obtaining training denoising prediction information that matches the corresponding training denoising step size.

[0205] This embodiment encodes the training denoising step size information, converting the denoising span into step size conditional information usable by the latent video generation network. By injecting the step size conditional information into the temporal conditional path, the latent video generation network can simultaneously distinguish the current noise state and the update scale of the current training denoising step. By obtaining training denoising prediction information based on the training input latent representation, training conditional feature information, training noise temporal information, and step size conditional information, the matching degree between the training denoising prediction and the video generation conditions, noise state, and denoising step size can be improved. By explicitly inputting the denoising step size during the training phase, the latent video generation network can learn to adapt to different denoising spans, providing a training basis for using larger denoising step sizes and fewer denoising steps during the inference phase. This embodiment addresses the problem that the latent video generation network has difficulty distinguishing different denoising spans when predicting based solely on noise time, and that prediction accuracy easily decreases with larger denoising step sizes. It adopts the concept of encoding the training denoising step size into step size conditional information and injecting it into the temporal conditional path, thereby improving the latent video generation network's adaptability to different denoising step sizes and supporting efficient, low-step video generation.

[0206] For example, in a single training denoising step, the training noise time information is 0.60, and the training denoising step size information is 0.20. The training denoising step size information with a value of 0.20 is encoded to obtain step size condition information. This step size condition information is then fused with the time features corresponding to the training noise time information with a value of 0.60 and injected into the temporal conditional path of the potential video generation network.

[0207] The latent video generation network outputs training denoising prediction information based on the training input latent representation, training conditional features, training noise time information with a value of 0.60, and step size condition information with a value of 0.20. In another training denoising step, the training denoising step size can be set to 0.10, enabling the latent video generation network to learn to generate corresponding training denoising prediction information for different training denoising spans.

[0208] In other embodiments, a video generation backbone network or a low-rank adapter can be initialized based on a pre-trained video generation model, so that the video generation model inherits the scene appearance, object structure and video temporal features already learned by the pre-trained video generation model, and is further trained on the training data corresponding to the reprojection condition information.

[0209] Reprojection conditional information can also be provided to the potential video generation network along with at least one conditional information from text description, semantic segmentation map, and depth map to control video content generation from different aspects such as scene semantics, object category, or spatial depth.

[0210] The number of denoising steps can be set to two, three, four, eight, or other numbers. During training, the range of values ​​for the training denoising step size can be changed, allowing the latent video generation network to learn denoising prediction capabilities that match different numbers of denoising steps.

[0211] Video generation models can be combined with model quantization or model pruning to further reduce the parameter storage or computational cost of the video generation model. After decoding the latent representation of the video to obtain multiple video frames, the spatial resolution of these multiple video frames can be improved through super-resolution networks.

[0212] In some embodiments, the training input latent representation can be generated based on the training video latent representation, noise information, and training noise temporal information using a stream matching method. The training input latent representation can be determined by the following formula.

[0213] in, This represents the latent representation of the training input. This represents the latent representation of the training video. Represents noise information. This indicates the noise level determined by the training noise time information.

[0214] The noise information can be random noise sampled according to a standard Gaussian distribution, as shown below.

[0215] in, Let I represent a Gaussian distribution with zero mean and covariance matrix equal to the identity matrix, where I represents the identity matrix.

[0216] In the implementation using discrete noise time, the noise level can be determined by the following formula.

[0217] Where t represents the noise time step corresponding to the training noise time information. Indicates the total number of noise time steps. Noise level. It can be between 0 and 1, and is used to control the proportion of training video latent representation and noise information in the training input latent representation.

[0218] The supervised denoising information can be the supervised velocity field information. The supervised velocity field information can be determined by the following formula.

[0219] Where v represents the monitoring velocity field information, Represents noise information. The training video latent representation is represented. Supervised velocity field information characterizes the direction of change of the training input latent representation from the training video latent representation towards noise information.

[0220] When using supervised denoising training, the training denoising step size can be set to 0, causing the latent video generation network to perform standard stream matching prediction. The training denoising prediction information output by the latent video generation network can be represented as follows.

[0221] in, This represents the training denoising prediction information under supervised denoising training. The parameter is Potential video generation networks, denoted by , where t represents the training input latent representation, t represents the training noise time information, the third input parameter zero indicates that the training denoising step size is 0, and c represents the training conditional feature information.

[0222] The supervised denoising training loss can be determined based on the difference between the training denoising prediction information and the supervised velocity field information, as shown below.

[0223] Alternatively, after representing the training denoising prediction information separately, the supervised denoising training loss can be represented as follows.

[0224] in, This represents the supervised denoising training loss. This represents the squared L2 norm. By reducing the supervised denoising training loss, the trained denoising prediction information output by the potential video generation network can be made closer to the supervised velocity field information.

[0225] In some embodiments, when a consistent training method is selected, the training denoising step size \Deltat can be divided into two reference denoising steps, each of which is half of the training denoising step size, as shown below.

[0226] in, This indicates the step size information for training denoising. This indicates the first reference denoising step size information. This indicates the second reference denoising step size information.

[0227] The reference latent video generation network can perform a first reference denoising step based on the training input latent representation, training conditional feature information, first reference noise temporal information, and first reference denoising step size information to obtain first reference denoising prediction information.

[0228] in, This represents the first reference denoising prediction information. The parameter is The reference latent video generation network can be a teacher network with frozen parameters, or a latent video generation network in a gradient-free mode.

[0229] Based on the first reference denoising prediction information and the first reference denoising step size information, the training input latent representation can be updated to obtain the intermediate reference latent representation.

[0230] in, Let U denote the intermediate reference latent representation, and let U denote the latent representation update function that updates the latent representation based on the reference denoising prediction information and the reference denoising step size information. This expression can encompass different numerical update methods without limiting the latent representation update direction or specific integration method.

[0231] The reference latent video generation network can perform a second reference denoising step based on intermediate reference latent representations, training conditional feature information, second reference noise temporal information, and second reference denoising step size information to obtain second reference denoising prediction information.

[0232] in, This represents the second reference denoising prediction information. This indicates the reference noise time information corresponding to the intermediate reference potential representation.

[0233] The first reference denoising prediction information and the second reference denoising prediction information can be averaged to obtain the reference denoising prediction information.

[0234] in, This represents the reference denoising prediction information. The reference denoising prediction information combines the prediction results from two smaller reference denoising steps to form a relatively refined reference supervision.

[0235] The latent video generation network can perform a training denoising step based on the latent representation of the training input, training conditional feature information, training noise temporal information, and complete training denoising step size information to obtain training denoising prediction information.

[0236] in, This represents the training denoising prediction information under the consistency training method. The latent video generation network uses the full training denoising step size. To perform a training denoising prediction, the reference to the latent video generation network uses two values. The reference denoising step size is used to perform two reference denoising steps.

[0237] The consistency training loss can be determined based on the difference between the training denoised prediction information and the reference denoised prediction information, as shown below.

[0238] Alternatively, if the training denoising prediction information is represented separately, the consistency training loss can be expressed as follows.

[0239] in, This represents the consistency training loss. By reducing the consistency training loss, the training denoising prediction information obtained by the latent video generation network using a larger training denoising step size can be made closer to the reference denoising prediction information obtained by the reference latent video generation network using two smaller reference denoising steps.

[0240] To clarify that the reference denoising prediction branch does not participate in gradient backpropagation, the consistency training loss can also be expressed as follows.

[0241] in, This indicates stopping gradient computation. When the teacher network with frozen parameters does not participate in parameter updates, the explicit indication to stop gradient computation can be omitted.

[0242] For example, if the training denoising step size is set to 0.20, then the first reference denoising step size and the second reference denoising step size can both be as follows.

[0243] The latent video generation network sequentially performs two reference denoising steps with a reference denoising step size of 0.10 to obtain the first reference denoising prediction information and the second reference denoising prediction information; the latent video generation network performs a training denoising step with a training denoising step size of 0.20, and determines the consistency training loss based on the difference between the training denoising prediction information and the average of the two reference denoising prediction information.

[0244] In some embodiments, for a single training iteration, a target training method can be selected from supervised denoising training and consistency training methods using a random variable. The target training loss can be determined using the following formula.

[0245] in, Indicates the training loss of the target. This represents the supervised denoising training loss. Let represent the consistency training loss, and b represent the random variable used to select the target training method.

[0246] Random variable b can follow the parameter as The Bernoulli distribution of is represented as follows.

[0247] in, This represents the probability of choosing the supervised denoising training method. This represents the probability of choosing the consistency training method.

[0248] The probability of random variable b taking a value can be expressed as follows.

[0249] When the random variable b is 1, the target training loss is the supervised denoising training loss.

[0250] When the random variable b is 0, the target training loss is the consistency training loss.

[0251] Therefore, each training iteration uses only one of the supervised denoising training loss or the consistency training loss, rather than updating parameters based on both training losses simultaneously. By randomly switching between supervised denoising training and consistency training in different training iterations, the video generation model can simultaneously learn standard denoising prediction capabilities and short-step prediction capabilities with larger denoising step sizes.

[0252] For example, the probability of selecting the supervised denoising training method can be set as follows.

[0253] Accordingly, the probability of choosing the consistency training method is as follows.

[0254] In multiple training iterations, approximately 75% of the training iterations used supervised denoising training loss, and approximately 25% used consistency training loss. In each training iteration, based on the value of the random variable b, a training loss was selected from the supervised denoising training loss and the consistency training loss, and the video generation model was updated based on the selected training loss.

[0255] See Figure 2 , Figure 2This is a schematic diagram of scene geometry restoration, camera motion path planning, and reprojection condition generation provided in an embodiment of this application.

[0256] like Figure 2 As shown, the system first receives an input panoramic image and determines the depth information corresponding to the pixels in the image. Based on the pixels and depth information in the input panoramic image, the scene content represented by the image can be mapped to 3D space to obtain a 3D point cloud. Further triangulation of the 3D point cloud yields a scene mesh model. At least one of the depth information, the 3D point cloud, and the scene mesh model can serve as scene geometric information.

[0257] Based on scene geometry information, passable areas can be determined according to access constraints. For example, candidate ground regions can be identified based on the scene mesh model, and obstacle regions and regions with insufficient clearance can be eliminated from the candidate ground regions according to collision constraints and clearance constraints to obtain passable regions. Within the passable regions, camera motion paths are generated, and multiple target camera poses are determined along the camera motion paths. The multiple target camera poses are then combined into a target camera pose sequence according to the camera motion order.

[0258] For each target camera pose in the target camera pose sequence, pixels in the input panoramic image can be mapped to 3D space based on scene geometry information, and the mapped pixels in 3D space can be projected onto the image space corresponding to the target camera pose. Based on the projection coverage of pixels in the image space, the reprojected image and target generation region corresponding to the target camera pose can be determined, and the reprojection condition information corresponding to the target camera pose can be obtained based on the reprojected image and target generation region. The target generation region is the area in the image space where no valid reprojected image content provided by pixels in the input panoramic image has been obtained.

[0259] Figure 2 By sequentially showing the input panoramic image, depth information, 3D point cloud, scene mesh model, passable area, camera motion path, target camera pose sequence, reprojected image, and target generation area, the process of recovering scene geometric information from the input panoramic image, planning camera motion, and generating pose-by-pose reprojection condition information is illustrated.

[0260] See Figure 3 , Figure 3 This is a schematic diagram illustrating the collaborative processing of reprojection condition information and potential video generation network provided in an embodiment of this application.

[0261] like Figure 3As shown, the reprojection condition information corresponding to multiple target camera poses can be arranged in the order of the target camera pose sequence. Each reprojection condition information can include the reprojected image and target generation region corresponding to the corresponding target camera pose. By conditionally encoding multiple reprojection condition information, conditional feature information corresponding to the order of the target camera pose sequence can be obtained.

[0262] The video generation model can include a latent video generation network. After obtaining the noisy latent representation, the noisy latent representation and conditional feature information can be provided to the latent video generation network. Under the constraints of the conditional feature information, the latent video generation network can denoise the noisy latent representation, gradually converting it into a video latent representation. The conditional feature information can be used to provide the latent video generation network with the reprojected image content corresponding to each target camera pose, the target generation region, and the sequential relationship between different target camera poses.

[0263] After the latent video generation network outputs a latent video representation, this representation can be decoded to obtain multiple video frames. These multiple video frames correspond to different target camera poses in the target camera pose sequence and can be arranged in the order of the target camera pose sequence. The video generation model can preserve the image content provided by the input panoramic image based on the reprojected image and generate image content corresponding to the corresponding target generation region in each video frame.

[0264] Figure 3 By illustrating the conditional processing pathway of "reprojection image and target generation region - conditional encoding - conditional feature information" and the video generation pathway of "noise latent representation and conditional feature information - latent video generation network - video latent representation - decoding - multiple video frames", the constraint effect of reprojection conditional information on the latent spatial video generation process is demonstrated.

[0265] See Figure 4 , Figure 4 This is a schematic diagram of a long-short mixed training and consistency training provided in an embodiment of this application.

[0266] like Figure 4 As shown on the left, multiple training pose segments with different trajectory spans can be determined from the training camera pose sequence. For each training iteration, a target trajectory span can be uniformly and randomly determined within a preset trajectory span range, and corresponding training pose segments can be extracted from the training camera pose sequence according to the target trajectory span. Different training pose segments can include short-range training pose segments with smaller trajectory spans and long-range training pose segments with larger trajectory spans.

[0267] Based on the positional range of each training pose segment within the training camera pose sequence, the corresponding training video segments and training condition segments can be determined from the training video sequence and the training reprojection condition sequence, respectively. Short-range training pose segments correspond to smaller camera pose changes and smaller target generation region sizes, while long-range training pose segments correspond to larger camera pose changes and larger target generation region sizes. Therefore, training samples with different trajectory spans and different target generation region sizes can jointly participate in the training of the video generation model.

[0268] like Figure 4 As shown on the right, for a single training iteration, the target training method can be selected from supervised denoising training method (A) and consistency training method (B).

[0269] When choosing a supervised denoising training method, a latent video generation network can be used to perform training denoising steps based on the latent representation of the training input, training conditional features, training noise temporal information, and training denoising step size information to obtain trained denoising prediction information. By comparing the trained denoising prediction information with the supervised denoising information, the supervised denoising training loss can be determined based on the difference between the two.

[0270] When choosing a consistency training method, multiple reference denoising steps can be performed using a latent video generation network to obtain reference denoising prediction information; and fewer training denoising steps can be performed using the latent video generation network to obtain training denoising prediction information. Comparing the training denoising prediction information with the reference denoising prediction information allows the consistency training loss to be determined based on their differences.

[0271] Depending on the target training method selected in a training iteration, the video generation model can be updated based on either supervised denoising training loss or consistency training loss. For example, when supervising denoising training is selected, the video generation model is updated based on the supervised denoising training loss; when consistency training is selected, the video generation model is updated based on the consistency training loss.

[0272] Figure 4 The process of constructing training segments with different trajectory spans on the left illustrates that short-range training pose segments and long-range training pose segments jointly participate in the training of the video generation model. The supervised denoising training branch and consistency training branch on the right illustrate that while maintaining the basic denoising prediction ability, the video generation model learns to approximate the processing results of multiple reference denoising steps with fewer training denoising steps, thus providing a training basis for low-step video generation.

[0273] See Figure 5 , Figure 5 This is a structural block diagram of a video generation device based on a three-dimensional scene provided in an embodiment of this application.

[0274] This application also provides a video generation device based on a three-dimensional scene, the device including a data receiving module, an information determining module, a reprojection module and a video generation module.

[0275] The data receiving module is used to receive the input panoramic image.

[0276] The information determination module is used to determine the scene geometry information and target camera pose sequence corresponding to the input panoramic image, wherein the target camera pose sequence includes multiple target camera poses.

[0277] The reprojection module is used to reproject the pixels in the input panoramic image to the image space corresponding to the respective target camera pose based on the scene geometry information for each target camera pose, thereby obtaining reprojection condition information corresponding to the target camera pose. The reprojection condition information is used to characterize the reprojected image obtained by reprojecting the pixels in the input panoramic image and the target generation region. The target generation region is the region in the image space that has not obtained effective reprojected image content provided by the pixels in the input panoramic image.

[0278] The video generation module is used to generate multiple video frames corresponding to the poses of each target camera based on the reprojection condition information corresponding to the poses of each target camera, and to arrange the multiple video frames according to the order of the poses of each target camera in the target camera pose sequence to obtain a scene video. The video generation module generates image content corresponding to the corresponding target generation region in each video frame.

[0279] In some embodiments, the information determination module may include a scene geometry restoration submodule and a path planning submodule. The scene geometry restoration submodule is used to determine depth information based on the input panoramic image, and generate a 3D point cloud or scene mesh model based on the pixel and depth information in the input panoramic image; the path planning submodule is used to determine a passable area based on the scene geometry information and according to the passable constraints, generate a camera motion path within the passable area, and determine the target camera pose sequence based on the camera motion path.

[0280] The path planning submodule may include a passable area extraction unit, an obstacle area processing unit, a path generation unit, and a path verification unit. The passable area extraction unit is used to detect ground areas; the obstacle area processing unit is used to determine obstacle areas based on non-ground structures, expand the obstacle areas, and filter areas with insufficient clearance; the path generation unit is used to generate camera motion paths within passable areas; and the path verification unit is used to perform camera occupancy space detection and clearance detection on the camera pose.

[0281] The reprojection module can include a 3D mapping submodule, a target view projection submodule, and a region determination submodule. The 3D mapping submodule is used to map pixels in the input panoramic image to 3D space; the target view projection submodule is used to project the pixels mapped to 3D space to the image space corresponding to the pose of each target camera; and the region determination submodule is used to determine the reprojected image and the target generation area based on the projection coverage.

[0282] The video generation module may include a conditional coding submodule, a latent video generation submodule, and a video decoding submodule. The conditional coding submodule is used to encode the reprojection condition information; the latent video generation submodule is used to perform denoising based on the noise latent representation and conditional feature information; and the video decoding submodule is used to decode the video latent representation to obtain multiple video frames.

[0283] In some embodiments, the video generation apparatus based on a 3D scene may further include a model training module. The model training module may include a long-range sampling submodule, a training method selection submodule, a supervised denoising loss calculation submodule, a consistency loss calculation submodule, and a parameter update submodule. The long-range sampling submodule is used to determine training pose segments with different trajectory spans; the training method selection submodule is used to select a target training method from supervised denoising training methods and consistency training methods; and the parameter update submodule is used to update the video generation model according to the training loss corresponding to the target training method.

[0284] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above methods.

[0285] This application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of any of the above methods.

[0286] See Figure 6 , Figure 6 This is a structural block diagram of a computer device provided in an embodiment of this application.

[0287] This application also provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of any of the above methods.

[0288] The embodiments of this application do not limit the computer device, such as a local computer device, a cloud computer device, a distributed computer device, etc.

[0289] The computer device may include: a memory 110, a processor 120, and a communication interface 130. The memory 110, the processor 120, and the communication interface 130 are connected through internal connection paths.

[0290] The memory 110 is used to store computer programs, which in some implementations may include code for implementing the methods of the embodiments of this application.

[0291] The processor 120 is used to execute the computer program stored in the memory 110 to control the communication interface 130 to receive input data and information, and output operation results and other data.

[0292] It is understood that in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application.

[0293] It is understood that the various implementation methods described in this application can be implemented individually or in combination, and this application does not limit them.

[0294] Unless otherwise stated, all technical and scientific terms used in this application have the same meaning as would be understood by one of ordinary skill in the art. The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The singular forms “a,” “an,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0295] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution.

[0296] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the embodiments described above can be referred to the corresponding processes in other embodiments, and will not be repeated here.

[0297] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and other division methods may exist in actual implementation. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections can be through some interfaces; indirect couplings or communication connections between apparatuses or units can be electrical, mechanical, or other forms.

[0298] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A video generation method based on a 3D scene, characterized in that, The method includes: Receive input panoramic image; Determine the scene geometry information and target camera pose sequence corresponding to the input panoramic image, wherein the target camera pose sequence includes multiple target camera poses; For each target camera pose, based on the scene geometry information, the pixels in the input panoramic image are reprojected to the image space corresponding to the target camera pose to obtain reprojection condition information corresponding to the target camera pose. The reprojection condition information is used to characterize the reprojected image obtained by reprojecting the pixels in the input panoramic image and the target generation region. The target generation region is the region in the image space that has not obtained effective reprojected image content provided by the pixels in the input panoramic image. Based on the reprojection condition information corresponding to the poses of each target camera, a plurality of video frames corresponding to the poses of each target camera are generated by a video generation model. The plurality of video frames are arranged in the order of the poses of each target camera in the target camera pose sequence to obtain a scene video. The image content corresponding to the corresponding target generation region in each video frame is generated by the video generation model.

2. The video generation method based on a three-dimensional scene according to claim 1, characterized in that, Determining the scene geometry information corresponding to the input panoramic image includes: Based on the input panoramic image, determine the depth information corresponding to the pixels in the input panoramic image; Based on the pixels in the input panoramic image and the depth information, the scene content represented by the input panoramic image is mapped to three-dimensional space to obtain the scene geometric information.

3. The video generation method based on a three-dimensional scene according to claim 1, characterized in that, The process of determining the target camera pose sequence includes: Based on the scene geometry information, the passable area is determined according to the traffic constraints; Generate a camera motion path within the passable area; The poses of the multiple target cameras are determined based on the camera motion path to obtain the target camera pose sequence. The passage constraints include at least one of collision constraints and clearance constraints.

4. The video generation method based on a three-dimensional scene according to claim 3, characterized in that, Determining the poses of the multiple target cameras based on the camera motion path includes: A candidate camera pose sequence is generated based on the camera motion path, wherein the candidate camera pose in the candidate camera pose sequence includes camera position information and camera attitude information; Based on the pose continuity condition, the candidate camera pose sequence is subjected to continuity processing to obtain an intermediate camera pose sequence. The continuity processing includes adjusting the candidate camera pose sequence when the position change and / or attitude change between adjacent candidate camera poses does not meet the pose continuity condition. Based on scene geometric information, it is detected whether the camera occupancy space corresponding to each intermediate camera pose in the intermediate camera pose sequence satisfies the passage constraint, and the detection result is obtained. Based on the detection results, the poses of intermediate cameras that do not meet the passage constraints are adjusted, deleted, or replaced to obtain the target camera pose sequence.

5. The video generation method based on a three-dimensional scene according to claim 1, characterized in that, For each target camera pose, based on the scene geometry information, the pixels in the input panoramic image are reprojected onto the image space corresponding to the respective target camera pose to obtain reprojection condition information corresponding to the target camera pose, including: Based on the scene geometry information, the pixels in the input panoramic image are mapped to three-dimensional space; For each of the target camera poses, the pixels mapped to the three-dimensional space are projected to the image space corresponding to the target camera pose. Based on the projection coverage of the pixels in the corresponding image space, the reprojected image and target generation region corresponding to the target camera pose are determined; Based on the reprojected image and the target generation region, the reprojection condition information corresponding to the target camera pose is obtained.

6. The video generation method based on a three-dimensional scene according to claim 1, characterized in that, The video generation model includes a latent video generation network. The step of generating multiple video frames corresponding to the poses of each target camera, based on reprojection condition information corresponding to the poses of each target camera, includes: According to the order of each target camera pose in the target camera pose sequence, the reprojection condition information corresponding to each target camera pose is encoded to obtain condition feature information; Obtain the latent representation of noise; The latent video generation network performs denoising based on the noise latent representation and the conditional feature information to obtain the video latent representation. The latent representation of the video is decoded to obtain the plurality of video frames.

7. The video generation method based on a three-dimensional scene according to claim 6, characterized in that, The step of performing denoising processing based on the noise latent representation and the conditional feature information through the latent video generation network to obtain the video latent representation includes: Determine multiple denoising steps and the denoising step size information corresponding to each denoising step; The noise latent representation is determined as the input latent representation corresponding to the first denoising step; According to the execution order of the plurality of denoising steps, for each of the denoising steps, the latent video generation network obtains the denoising prediction information corresponding to the corresponding denoising step based on the input latent representation, the conditional feature information, the noise time information and the denoising step size information. Based on the denoising prediction information and denoising step size information corresponding to the corresponding denoising step, the input latent representation corresponding to the corresponding denoising step is updated to obtain the output latent representation corresponding to the corresponding denoising step, and the output latent representation is determined as the input latent representation corresponding to the next denoising step. The output latent representation corresponding to the last denoising step is determined as the video latent representation; In each of the denoising steps, only denoising prediction based on the conditional feature information is performed, and denoising prediction not based on the conditional feature information is not performed.

8. The video generation method based on a three-dimensional scene according to claim 1, characterized in that, The video generation model was trained in the following way: Receive training data, the training data including a training camera pose sequence, a training video sequence corresponding to the training camera pose sequence, and a training reprojection condition sequence, the training reprojection condition sequence including multiple training reprojection condition information corresponding to multiple training camera poses in the training camera pose sequence respectively; Multiple training pose segments with different trajectory spans are determined from the training camera pose sequence; From the training video sequence and the training reprojection condition sequence, determine the training video segment and training condition segment corresponding to each training pose segment, respectively. The video generation model is trained based on the training video segments and corresponding training condition segments, so that training pose segments with different trajectory spans and different target generation region sizes participate in the training of the video generation model.

9. The video generation method based on a three-dimensional scene according to claim 8, characterized in that, The video generation model includes a latent video generation network, and training the video generation model based on each of the training video segments and corresponding training condition segments includes: The training video segments are encoded to obtain the latent representation of the training video; Based on the latent representation of the training video, noise information, and training noise temporal information, a latent representation of the training input and supervised denoising information are generated. The training condition segments are encoded to obtain training condition feature information; For a single training iteration, a target training method is selected from supervised denoising training and consistency training, and the training denoising step size information corresponding to the target training method is determined. When the supervised denoising training method is selected, the latent video generation network performs a training denoising step based on the training input latent representation, the training condition feature information, the training noise time information and the training denoising step size information to obtain training denoising prediction information. Based on the difference between the training denoising prediction information and the supervised denoising information, the supervised denoising training loss is determined. When the consistency training method is selected, multiple reference denoising steps are performed using a reference latent video generation network based on the training input latent representation, the training conditional feature information, the reference noise time information, and the reference denoising step size information to obtain reference denoising prediction information; training denoising steps are performed using the latent video generation network based on the training input latent representation, the training conditional feature information, the training noise time information, and the training denoising step size information to obtain training denoising prediction information; consistency training loss is determined based on the difference between the training denoising prediction information and the reference denoising prediction information; wherein the number of training denoising steps is less than the number of reference denoising steps, and the reference latent video generation network is a parameter-frozen teacher network or the latent video generation network in a gradient-free mode; The video generation model is updated based on the supervised denoising training loss or consistency training loss corresponding to the target training method.

10. The video generation method based on a three-dimensional scene according to claim 9, characterized in that, The step of performing a training denoising step through the latent video generation network, based on the training input latent representation, the training conditional feature information, the training noise temporal information, and the training denoising step size information, to obtain training denoising prediction information includes: The training denoising step size information is encoded to obtain step size condition information; The step size condition information is injected into the temporal conditional path of the potential video generation network; The training denoising prediction information is obtained through the latent video generation network based on the training input latent representation, the training condition feature information, the training noise time information, and the step size condition information.

11. A video generation device based on a three-dimensional scene, characterized in that, The device includes: The data receiving module is used to receive the input panoramic image; An information determination module is used to determine the scene geometric information and target camera pose sequence corresponding to the input panoramic image, wherein the target camera pose sequence includes multiple target camera poses; The reprojection module is used to reproject the pixels in the input panoramic image to the image space corresponding to the respective target camera pose based on the scene geometry information for each target camera pose, thereby obtaining reprojection condition information corresponding to the target camera pose. The reprojection condition information is used to characterize the reprojected image obtained by reprojecting the pixels in the input panoramic image and the target generation region. The target generation region is the region in the image space that has not obtained effective reprojected image content provided by the pixels in the input panoramic image. The video generation module is used to generate multiple video frames corresponding to the poses of each target camera based on the reprojection condition information corresponding to the poses of each target camera, and to arrange the multiple video frames according to the order of the poses of each target camera in the target camera pose sequence to obtain a scene video. The video generation model generates image content in each video frame that corresponds to the corresponding target generation region.

12. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 10.

14. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 10.

Images