Interactive video resource generation method, device and product for four-dimensional dynamic scene modeling

Abstract

The application provides a four-dimensional dynamic scene modeling interactive video resource generation method, device and product, which is used for processing multiple videos of a four-dimensional dynamic scene into an interactive video resource capable of expressing time change and supporting subsequent interactive display, not directly combining the multiple videos into a common two-dimensional video, nor generating a complete three-dimensional model frame by frame, but expressing the dynamic scene as a structured resource in which a static base, a dynamic base and a dynamic deformation flow cooperate with each other, thereby providing a data basis for frame-by-frame recovery and free perspective interactive display of the four-dimensional dynamic scene.

Description

Technical Field

[0001] This application relates to the field of interactive graphics immersive technology, specifically to an interactive video resource generation method, computer device, and program product for four-dimensional dynamic scene modeling. Background Technology

[0002] In scenarios such as handicraft making, product display, figure operation, teaching demonstration, live streaming and live stream replay, it is often necessary to record the movement, posture changes, deformation process or operation process of dynamic subjects in the time dimension. Existing methods usually use single-channel or multi-channel video to shoot dynamic scenes, thereby realizing four-dimensional dynamic scene reconstruction, and providing the captured video files as display content for users to watch.

[0003] While multi-camera shooting can record dynamic scenes from multiple real-world shooting perspectives, the final displayed content is still a two-dimensional video frame sequence. Users can only view the dynamic scene according to preset shooting angles, editing order, or camera switching methods, making it difficult to freely change the viewing angle within a single video frame. For scenarios such as handicraft production, product detail display, or complex operation tutorials, users often want to pause the dynamic process at a certain point and observe the state of the dynamic subject from different angles such as the side, top view, or zoomed-in section. Relying solely on ordinary video playback cannot meet these interactive viewing needs.

[0004] Meanwhile, multiple high-resolution videos result in large data volumes. Directly transmitting multiple video files or generating a complete 3D model for each video frame would impose a significant burden on network transmission, storage, loading, decoding, and rendering. Especially in scenarios requiring rapid loading, timeline playback, pause, jump, and interactive viewing via terminal devices, traditional video files or frame-by-frame complete 3D models struggle to balance dynamic expression capabilities, resource size, and interactive display efficiency.

[0005] Therefore, how to process multiple video streams of a four-dimensional dynamic scene into an interactive video resource that can both express the changes of the dynamic scene over time and is suitable for transmission, loading, and interactive display has become a technical problem that needs to be solved. Summary of the Invention

[0006] This application provides a method, computer device, and program product for generating interactive video resources for four-dimensional dynamic scene modeling, in order to solve the technical problem of how to process multiple video streams of a four-dimensional dynamic scene into an interactive video resource that can both express the changes of the dynamic scene over time and is suitable for transmission, loading, and interactive display.

[0007] According to one aspect of the embodiments of this application, an interactive video resource generation method for four-dimensional dynamic scene modeling is disclosed, the method comprising:

[0008] For multiple videos of a four-dimensional dynamic scene, scene initialization training of the four-dimensional dynamic scene is performed using multi-view video frames and camera parameters from multiple videos to obtain an initial Gaussian representation of the complete scene. Based on the initial Gaussian representation, the parameters of each camera, and the region mask corresponding to the multi-view video frame and the camera viewpoint, the Gaussian elements in the initial Gaussian representation are trained for static-dynamic separation to obtain a set of static Gaussian elements and a set of dynamic Gaussian elements. The region mask is used to distinguish between dynamic subject regions and static background regions and serves as the supervision information for the static-dynamic separation training. The dynamic Gaussian set is trained with a deformation field to obtain the deformation field shared by the dynamic Gaussian set, and the dynamic deformation flow of the dynamic Gaussian set corresponding to each video frame is generated through the deformation field. Separate quantization compression is performed on the static basis obtained by transforming the static Gaussian set, the dynamic basis obtained by transforming the dynamic Gaussian set, and the dynamic deformation flow to generate interactive video resources of the four-dimensional dynamic scene.

[0009] According to one aspect of the embodiments of this application, a computer device is disclosed, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method as described in any of the preceding claims.

[0010] According to one aspect of the embodiments of this application, a computer program product is disclosed, including a computer program that, when executed by a processor, implements the steps of the method as described in any of the preceding claims.

[0011] This application embodiment is used to process multi-channel video of a four-dimensional dynamic scene into interactive video resources that can express time changes and support subsequent interactive displays. First, scene initialization training is performed based on multi-view video frames and camera parameters from the multi-channel video to obtain an initial Gaussian representation of the complete scene. Then, using region masks corresponding to the video frames and camera viewpoints, static and dynamic Gaussian elements in the initial Gaussian representation are trained to separate static and dynamic elements, obtaining a set of static Gaussian elements and a set of dynamic Gaussian elements. Further, deformation field training is performed on the set of dynamic Gaussian elements to obtain a deformation field shared by the set of dynamic Gaussian elements, and a dynamic deformation flow corresponding to each video frame is generated through this deformation field. Finally, separate quantization compression is performed on the static basis obtained by transforming the set of static Gaussian elements, the dynamic basis obtained by transforming the set of dynamic Gaussian elements, and the dynamic deformation flow to generate interactive video resources of the dynamic scene. Therefore, the embodiments of this application do not directly synthesize multiple video streams into ordinary two-dimensional video, nor do they generate complete three-dimensional models frame by frame. Instead, they express dynamic scenes as structured resources in which static base, dynamic base, and dynamic deformation flow cooperate with each other, providing a data foundation for frame-by-frame restoration of four-dimensional dynamic scenes and free-viewpoint interactive display.

[0012] Based on the above technical solution, the following specific explanation is provided: First, this embodiment obtains an initial Gaussian representation of the complete scene through scene initialization training, unifying the multi-view image information from multiple videos into a spatially consistent Gaussian expression, providing a stable foundation for subsequent motion-static separation and deformation field training. Compared to directly processing dynamic subjects on two-dimensional video frames, this embodiment can perform four-dimensional dynamic scene modeling at the three-dimensional Gaussian primitive level, which is beneficial to improving the stability of the separation between dynamic subjects and static backgrounds.

[0013] Second, this embodiment of the application uses region mask-supervised Gaussian unit (GSU) training to separate static and dynamic elements, thus splitting the four-dimensional dynamic scene into a set of static GSUs and a set of dynamic GSUs, allowing the static background and dynamic subject to be expressed separately in data organization. Therefore, when generating interactive video resources, the static background does not need to be repeatedly stored with each video frame, and the dynamic subject does not need to be mixed with the static background to form a complete scene resource frame by frame, thereby reducing the amount of repetitive data in the four-dimensional dynamic scene resource.

[0014] Third, the embodiments of this application train a shared deformation field on a dynamic Gaussian primitive set and generate dynamic deformation flows corresponding to each video frame, so that the temporal changes of the dynamic subject can be jointly expressed by the dynamic basis and the dynamic deformation flow. Compared with the method of generating a complete Gaussian model frame by frame, this method only needs to record the attribute changes of the dynamic Gaussian primitives in each video frame, which can reduce the size of dynamic scene interactive video resources and improve the feasibility of restoring the dynamic state frame by frame.

[0015] Fourth, this application embodiment performs separate quantization compression on the static base, dynamic base, and dynamic deformation stream, allowing different types of data to be processed separately according to their temporal variation characteristics. The static base corresponds to the long-term unchanging scene content, the dynamic base corresponds to the baseline state of the dynamic subject, and the dynamic deformation stream corresponds to the frame-by-frame deformation data that changes over time. This separate resource organization method can reduce the repetitive encoding of static content and enable dynamic content to be compressed and expressed in a deformation stream manner, thereby improving the compression efficiency and loading efficiency of interactive video resources.

[0016] Fifth, the interactive video resources generated in this application embodiment can provide a data foundation for subsequent terminal devices to restore the dynamic scene state according to the target video frame. The terminal device can generate a Gaussian representation of the target video frame based on the static basis, the dynamic basis, and the dynamic deformation data corresponding to the target video frame, and further perform free-viewpoint interactive display. Thus, the resources generated in this application can not only express the temporal changes of the dynamic scene, but also break through the limitation that ordinary videos can only be viewed from a preset shooting angle, providing a foundation for interactive observation of the dynamic scene under the target video frame.

[0017] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0018] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this application. Attached Figure Description

[0019] The above and other objectives, features and advantages of this application will become more apparent from a detailed description of exemplary embodiments thereof with reference to the accompanying drawings.

[0020] Figure 1 This is a flowchart illustrating an interactive video resource generation method for four-dimensional dynamic scene modeling according to an exemplary embodiment.

[0021] Figure 2 This is a flowchart illustrating a method for acquiring and preprocessing multiple video streams of a four-dimensional dynamic scene, obtaining camera parameters for each stream and region masks corresponding to each video frame, with the region masks corresponding to the video frames and camera viewpoints, according to an exemplary embodiment.

[0022] Figure 3 It is based on Figure 1 The flowchart of the corresponding embodiment describes the process of initializing the four-dimensional dynamic scene by using multi-view video frames and camera parameters from multiple video streams to obtain an initial Gaussian representation of the complete scene.

[0023] Figure 4 It is based on Figure 1The corresponding embodiment shows a flowchart describing the steps of performing static and dynamic separation training on Gaussian primitives in the initial Gaussian representation based on the initial Gaussian representation, camera parameters, and region masks corresponding to multi-view video frames and camera viewpoints to obtain a set of static Gaussian primitives and a set of dynamic Gaussian primitives.

[0024] Figure 5 It is based on Figure 1 The corresponding embodiment shows a flowchart describing the steps of training a deformation field on a dynamic Gaussian set, obtaining a shared deformation field for the dynamic Gaussian set, and generating a dynamic deformation flow for each video frame based on the deformation field.

[0025] Figure 6 It is based on Figure 1 The flowchart described in the corresponding embodiment describes the steps of performing separate quantization compression on the static basis obtained by transforming the static Gaussian set, the dynamic basis obtained by transforming the dynamic Gaussian set, and the dynamic deformation flow to generate interactive video resources for a four-dimensional dynamic scene.

[0026] Figure 7 This is a schematic diagram of the display interface showing attribute changes from different viewing angles.

[0027] Figure 8 yes Figure 7 A schematic diagram of the display interface showing attribute changes from another viewing angle, as shown in the corresponding embodiment. Detailed Implementation

[0028] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided to make the description of this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art. The drawings are merely illustrative of this application and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0029] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more exemplary embodiments. Numerous specific details are provided in the following description to give a full understanding of exemplary embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced with one or more of the specific details omitted, or other methods, components, steps, etc., can be employed. In other instances, well-known structures, methods, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0030] Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0031] This application embodiment is used to reconstruct four-dimensional dynamic scenes from multiple videos collected in scenarios such as handicraft recording, product display, live streaming and live streaming replay, so as to obtain interactive video resources of the reconstructed four-dimensional dynamic scenes.

[0032] See Figure 1 , Figure 1 This is a flowchart illustrating an interactive video resource generation method for four-dimensional dynamic scene modeling according to an exemplary embodiment.

[0033] The interactive video resource generation method for four-dimensional dynamic scene modeling provided in this application includes: Step S110: For the multi-channel video of the four-dimensional dynamic scene, the scene initialization training of the four-dimensional dynamic scene is carried out by using the multi-view video frames and camera parameters in the multi-channel video to obtain the initial Gaussian representation of the complete scene. Step S120: Based on the initial Gaussian representation, the parameters of each camera, and the region mask corresponding to the multi-view video frame and the camera viewpoint, the Gaussian primitives in the initial Gaussian representation are trained for static-dynamic separation to obtain a set of static Gaussian primitives and a set of dynamic Gaussian primitives. The region mask is used to distinguish between the dynamic subject region and the static background region and serves as the supervision information for the static-dynamic separation training. Step S130: Perform deformation field training on the dynamic Gaussian set to obtain the deformation field shared by the dynamic Gaussian set, and generate the dynamic deformation flow of the dynamic Gaussian set corresponding to each video frame through the deformation field. Step S140: Perform separate quantization compression on the static basis obtained by transforming the static Gaussian set, the dynamic basis obtained by transforming the dynamic Gaussian set, and the dynamic deformation flow to generate interactive video resources of a four-dimensional dynamic scene.

[0034] These steps are explained in detail below.

[0035] In this embodiment, a four-dimensional dynamic scene refers to a dynamic scene with a three-dimensional spatial structure that undergoes movement, posture changes, deformation changes, or appearance changes over time. For example, a four-dimensional dynamic scene may include a dynamic handicraft production scene, a character operation scene, a product display scene, a teaching demonstration scene, a live streaming scene, a live streaming replay scene, or a small dynamic object display scene. The dynamic subjects in a four-dimensional dynamic scene include characters, hands, tools, materials, semi-finished products, goods, moving parts, flexible objects, or other objects that change over time; the static background includes workbenches, display stands, backgrounds, fixed props, fixed supports, or other areas that remain stationary for extended periods.

[0036] The interactive video resources generated in this application differ from ordinary two-dimensional video files. Ordinary two-dimensional video files typically use a pre-shot sequence of video frames as playback content, and the user's viewing experience is limited by the actual shooting perspective. In contrast, the interactive video resources in this application include a static basis obtained from a static Gaussian set, a dynamic basis obtained from a dynamic Gaussian set, and the dynamic deformation flow of the dynamic Gaussian set corresponding to each video frame. Therefore, the interactive video resources can restore the dynamic state according to the target video frame in subsequent terminal devices, providing a data foundation for free-viewpoint interactive display.

[0037] In step S110, the multiple video streams are acquired by multiple cameras arranged around a four-dimensional dynamic scene. Each video stream corresponds to a camera viewpoint. In other words, multiple video streams refer to video data acquired by multiple cameras from different camera viewpoints of the same four-dimensional dynamic scene. The multi-view video frames in the multiple video streams refer to image frames extracted from different camera viewpoints and used for training under a unified time axis. Camera parameters describe the mapping relationship between the corresponding camera viewpoint and a unified three-dimensional world coordinate system. Camera parameters include intrinsic and extrinsic parameters. Intrinsic parameters include focal length, principal point, distortion parameters, etc., while extrinsic parameters include the camera's rotation and translation in the unified three-dimensional world coordinate system.

[0038] It should be noted that the multi-channel video used for scene initialization training in step S110 is the multi-channel video data obtained after preprocessing. That is to say, in this embodiment of the application, the acquired multi-channel video will be preprocessed before performing step S110 to provide input for step S110.

[0039] To further clarify, this preprocessing step includes at least: multi-channel video synchronization, frame extraction, camera parameter estimation, sparse 3D point cloud generation, image distortion correction, and region mask generation. After preprocessing, synchronized multi-view video frames, camera parameters for each channel, sparse 3D point clouds, and region masks corresponding to the video frames and camera viewpoints are obtained. Therefore, the scene initialization training in step S110 is performed on training data that has already undergone camera parameter estimation and video frame alignment.

[0040] Further explanation of the preprocessing steps involves multi-channel video synchronization, which maps videos captured by different cameras to the same reference timeline. For the c-th video stream, the mapping from its local time t to the reference timeline can be expressed as:

[0041] in, It is the time t of the local time t of the c-th video channel mapped to the reference time axis; It is a time-scale correction factor; It's a time offset.

[0042] The start and end times of the peak high-five were detected as follows: and The start and end times of the clapping peak detected on the reference time axis corresponding to the reference camera are respectively... and ,but: ;

[0043] If the camera clock is stable and the acquisition time is short, then let A value of 1 indicates that only the time offset is estimated. If the acquisition period is long, or if the synchronization signal still deviates after the synchronization signal is aligned at the beginning, the time scale correction coefficient and time offset are calculated based on the start and end times corresponding to the peak of the clapping, in order to perform linear drift correction. This allows the video streams to be mapped to the same reference time axis even if there is a slight time drift between different cameras.

[0044] Through the above time mapping, each video stream is extracted into multi-view video frames that share a unified frame number and correspond to a unified timestamp.

[0045] In an exemplary embodiment, the execution process of multi-channel video synchronization specifically includes: acquiring multiple video streams of the target object captured by multiple cameras, wherein the multiple video streams are used to provide multi-camera shooting content of the target object, and each video stream contains an audio clapping synchronization signal and / or clapping action footage; obtaining the clapping action footage by detecting the clapping peak value in each audio track, and performing synchronous mapping of the multiple video streams to a reference time axis, so that the video frames of the multiple video streams are synchronized to the reference time axis.

[0046] Based on this, synchronization of multiple video streams in the temporal dimension is achieved, enabling video frames captured by different cameras to be mapped to the same reference timeline. Time synchronization affects the accuracy of subsequent camera parameter estimation, region mask generation, and three-stage Gaussian training. If there is a time offset or clock drift between the multiple video streams, different camera images at the same frame number will actually correspond to different action moments. This can easily lead to cross-viewpoint inconsistencies during subsequent Gaussian training based on multi-view video frames, thus affecting the stability of the initial Gaussian representation, motion-static separation training, and deformation field training.

[0047] Multi-camera shooting refers to capturing a target object from different perspectives using multiple cameras. Each camera outputs a corresponding video stream, and each video stream has its own audio track, video frame sequence, frame rate, resolution, encoding format, start recording time, and end recording time. Multiple video streams collectively record the target object's appearance, posture, actions, or operational processes during the presentation.

[0048] The audio clapping synchronization signal refers to the audio peak signal formed during video acquisition by clapping, tapping a synchronization board, tapping a table, or other actions that produce a noticeable instantaneous sound. This audio clapping synchronization signal should be able to be recorded by the audio tracks of multiple video streams. The clapping action frame refers to the visual image of clapping, tapping a synchronization board, or other synchronized actions that can be observed in the video frame. The audio clapping synchronization signal is used for time alignment based on audio peaks, while the clapping action frame is used for auxiliary calibration in situations where audio peaks are unclear, a video stream lacks an audio track, or there is significant environmental noise.

[0049] In practice, before the target object display begins, all cameras are put into recording mode. Then, one or more clear clapping or sync-pad tapping actions are performed to generate a detectable instantaneous energy peak in the audio track of each video stream, while ensuring that the clapping action is visible in at least some camera frames. Furthermore, after the target object display ends, the clapping or sync-pad tapping actions are performed again to determine whether clock drift exists between different cameras based on the start and end synchronization signals.

[0050] An audio track refers to the sound data recorded synchronously with the image frames in a video. A clapping peak refers to the instantaneous energy peak in the audio track generated by clapping, tapping a sync board, or other synchronized actions. A reference timeline is a time base used to uniformly represent the time positions of multiple video streams. For example, the timeline of one video stream is selected as the reference timeline; alternatively, a reference timeline can be established based on an external timestamp, global timecoding, or an independent synchronization clock. Synchronization mapping refers to the mapping relationship between the local time of each video stream and the reference timeline. Through synchronization mapping, video frames from different cameras can be aligned according to the same timestamp or the same unified frame number.

[0051] For example, the audio track is decoded from each video stream, and all audio tracks are standardized to the same sampling rate. Then, short-time energy, peak envelope, or other audio features characterizing instantaneous sound intensity are calculated for each audio track. To avoid interference from ambient noise, device noise floor, or false triggering, the energy envelope is smoothed, and the audio peaks corresponding to the clapping synchronization signal are selected by combining peak height, peak interval, peak duration, or peak confidence.

[0052] After detecting the clapping peaks in each audio track, a video with clear audio, stable frame rate, and visible synchronized motion is selected as the reference video, and its timeline is used as the reference timeline. For the c-th video, based on the clapping peak time in the c-th video and the corresponding clapping peak time in the reference video, the synchronization mapping relationship from the local time of the c-th video to the reference timeline is calculated, as shown in the aforementioned mapping relationship from the local time t of the c-th video to the reference timeline.

[0053] After establishing a synchronization mapping from each video stream to the reference timeline, in an exemplary embodiment, synchronized video frames are extracted from each video stream according to a unified timestamp. For the same target timestamp on the reference timeline, the corresponding video frames are found and extracted according to the synchronization mapping relationship of each video stream, thereby forming a synchronized frame sequence with a unified frame number for each video stream. The image frames corresponding to the same frame number constitute a multi-view video frame.

[0054] After synchronization mapping is completed, in an exemplary embodiment, a synchronization quality check will also be performed. The synchronization quality check covers three time points: the beginning, middle, and end of the acquisition segment. If alignment is achieved at the beginning but there is a significant offset at the end, it indicates possible clock drift between different cameras, and linear drift correction based on the start and end synchronization signals should be enabled. If a video stream has frame skipping, file segmentation, abnormal recording stops, or missing segments, this is recorded in the frame extraction configuration or synchronization quality report, and the abnormal frames are excluded in subsequent camera parameter estimation, region mask generation, or Gaussian training.

[0055] Therefore, this embodiment of the application can establish a synchronous mapping from multiple video streams to a reference time axis using audio clapping synchronization signals and / or clapping action images in the absence of hardware synchronization equipment, aligning the video frames according to a unified time axis. This synchronization result ensures that multi-view images under the same frame number correspond to the same action moment during the display of the target object, thereby guaranteeing the temporal consistency of the multi-view data used in subsequent camera parameter estimation, region mask generation, and three-stage Gaussian training. This reduces cross-view supervision conflicts caused by time offsets in multiple video streams, improving the stability of initial Gaussian representation training, motion-static separation training, and deformation field training.

[0056] Camera parameter estimation also needs to be completed before scene initialization training. Cross-camera feature extraction and feature matching are performed on synchronized multi-view video frames. The relative camera pose is estimated based on the cross-camera matching relationship, and sparse 3D point clouds are obtained through triangulation. Subsequently, the camera parameters and sparse 3D point clouds are jointly optimized based on the reprojection error. The camera projection relationship can be expressed as:

[0057] in, Used to characterize 3D points in a sparse 3D point cloud, it can be the homogeneous coordinates of the 3D point in the world coordinate system. It is the first c Road camera internal reference; and It is the first c External parameters of road cameras; These are homogeneous coordinates of the image plane.

[0058] Based on this, joint optimization is performed to obtain the camera parameters of each camera and the sparse 3D point cloud. The obtained camera parameters will be used for the time-differentiable rendering in step S110, the dynamic-static separation training in step S120, and the deformation field training in step S130. The sparse 3D point cloud will be used for Gaussian primitive initialization in step S110.

[0059] In an exemplary embodiment, the specific execution process of estimating camera parameters and obtaining sparse 3D point clouds includes: for each camera capturing multiple video streams, performing multi-view feature comparison to obtain feature point matching relationships between multi-view video frames based on the cross-camera correspondence between feature points in each video stream; estimating the relative poses between each camera based on the feature point matching relationships, and triangulating the cross-view matching points mapped by the estimated relative poses and feature point matching relationships to obtain sparse 3D point clouds; finally, jointly optimizing the sparse 3D point clouds, intrinsic parameters, and extrinsic parameters of each camera through the projection relationship of the sparse 3D point clouds onto the cross-view matching points distributed in the multi-view video frames, and outputting the intrinsic and extrinsic parameters of each camera as camera parameters, wherein the extrinsic parameters include the rotation and translation of the camera in a unified 3D world coordinate system.

[0060] In summary, camera parameters are estimated based on the content of multi-view video frames from multiple video streams, and a sparse 3D point cloud is obtained for subsequent scene initialization training. In this exemplary embodiment, video frames from different camera perspectives can be unified into the same 3D world coordinate system, providing a consistent spatial basis for subsequent Gaussian primitive initialization, differentiable rendering supervision, static / dynamic separation training, and deformation field training.

[0061] Each camera refers to multiple cameras used to capture multiple video feeds of a target object. Each camera corresponds to one video feed and one camera viewpoint. Cross-camera correspondence refers to the correspondence between the same spatial point, texture point, corner point, or local image structure captured from different camera viewpoints. Feature points refer to local image points that can be stably detected in an image and used for cross-viewpoint matching, such as corner points, texture points, stable local structure points near edges, or local descriptor points obtained through feature extraction algorithms. Feature point matching relationships refer to pairs or sets of feature points considered to correspond to the same spatial location between different multi-view video frames.

[0062] In one exemplary embodiment, multi-view video frames for camera parameter estimation are selected from synchronized multi-video streams. These multi-view video frames are preferably from the same time frame on the same reference time axis, or from a time period where the target object and scene are relatively stable, with minimal motion blur and occlusion, to reduce interference from target object motion on feature matching and improve the stability of camera parameter estimation.

[0063] Feature extraction is performed on multi-view video frames to obtain local feature points and their descriptors in each image frame. Then, feature comparison is performed between image frames corresponding to different camera viewpoints to determine cross-camera feature point matching relationships. Feature comparison can be performed based on descriptor distance, nearest neighbor ratio, geometric consistency constraints, epipolar constraints, or other matching filtering methods. For surround or semi-surround camera deployment scenarios, feature matching relationships are preferentially established between adjacent camera positions, while global matching relationships can also be established between non-adjacent camera positions with significant field-of-view overlap.

[0064] In another exemplary embodiment, to reduce the impact of dynamic objects on camera parameter estimation, the weight of feature points within dynamic regions is reduced during feature extraction or feature matching. Alternatively, feature points from static backgrounds, fixed props, workbenches, calibration boards, textured backgrounds, or other long-term static regions are preferentially used for matching. Since a region mask has not yet been obtained, matching reliability is improved by selecting images from static phases, adding background textures, setting removable calibration objects, or manually removing obviously dynamic regions.

[0065] The feature point matching relationship between multi-view video frames is obtained. This feature point matching relationship can include two-view matching point pairs, multi-view trajectory points, feature descriptor matching confidence, matching image number, camera number, and the two-dimensional coordinates of the matching point in the corresponding image, which serve as the basis for camera relative pose estimation and sparse 3D point cloud generation.

[0066] For example, in a product display scenario, multiple cameras are arranged around the product and the display stand. The system can extract stable feature points such as the edge of the display stand, background texture, and fixed texture of the product in time frames before the product begins to move or when the display action is paused, and match them across different camera viewpoints. Through these cross-camera feature point matching relationships, the spatial relationships between different cameras can be inferred.

[0067] Relative pose is used to represent the spatial position and orientation relationship between cameras, usually including rotation and translation; triangulation is the process of recovering the corresponding three-dimensional spatial points based on the two-dimensional matching points and camera poses from multiple camera views; sparse three-dimensional point cloud is an initial scene geometric representation composed of multiple three-dimensional spatial points.

[0068] The geometric constraints between cameras are estimated based on the obtained feature point matching relationships, and the relative poses between the cameras are recovered from these geometric constraints. After obtaining the relative poses, for cross-view matching points that are visible from two or more camera perspectives, triangulation is performed according to the projection relationships of the corresponding cameras to obtain the corresponding 3D spatial points. Furthermore, mismatched points and unstable 3D points are eliminated based on reprojection error, parallax angle, depth range, or the number of visible cameras to obtain a sparse 3D point cloud.

[0069] Two-dimensional feature matching relationships in multi-view images are transformed into sparse geometric structures in three-dimensional space. This sparse 3D point cloud is not only used to assist in camera parameter optimization, but also serves as the initial spatial location source for Gaussian units in subsequent scene initialization training.

[0070] Projection relationships refer to the mapping relationship between 3D spatial points projected onto a 2D image plane under given camera intrinsic and extrinsic parameters. Camera intrinsic parameters describe the camera's own imaging model, while camera extrinsic parameters describe the camera's position and orientation in a unified 3D world coordinate system. Joint optimization refers to simultaneously optimizing sparse 3D point clouds, camera intrinsic parameters, and camera extrinsic parameters to ensure that the projected positions of the sparse 3D point clouds onto each multi-view video frame are as close as possible to the corresponding cross-view matching points.

[0071] Based on the obtained sparse 3D point cloud and initial camera pose, each 3D point is projected onto its visible camera view, and the reprojection error between the projected position and the corresponding 2D feature point is calculated. Then, with the goal of reducing the reprojection error, the sparse 3D point cloud, camera intrinsic parameters, and camera extrinsic parameters are jointly optimized to obtain the camera parameters for each camera path. The optimized camera parameters can be used to project Gaussian elements in subsequent training onto the corresponding camera view, and the optimized sparse 3D point cloud can be used to initialize Gaussian elements.

[0072] This embodiment can recover camera parameters and sparse 3D point clouds of each camera in a unified 3D world coordinate system from multiple video streams, enabling the multiple video streams to have a unified spatial geometric relationship. This spatial geometric relationship directly coordinates with the subsequent three-stage Gaussian training: scene initialization training relies on sparse 3D point clouds to initialize Gaussian primitives and relies on camera parameters for differentiable rendering supervision; motion-static separation training relies on camera parameters to render the dynamic response values ​​of Gaussian primitives into dynamic prediction masks; deformation field training relies on camera parameters to perform multi-view supervision of the current Gaussian scene in the target video frame. Therefore, this embodiment improves the spatial consistency and training stability when converting multiple video streams into Gaussian interactive content.

[0073] In step S110, scene initialization training of the four-dimensional dynamic scene is performed based on multi-view video frames and parameters of each camera to obtain an initial Gaussian representation of the complete scene.

[0074] Corresponding to a sparse 3D point cloud, the initial Gaussian representation includes several Gaussian elements. A Gaussian element is the basic unit used to represent a 3D scene. For example, a Gaussian element includes appearance attributes such as spatial position, scale, rotation, transparency, and color, with spatial position represented by the center position of the Gaussian element. Gaussian elements are described by their attributes and exist in the form of attribute data. For example, a Gaussian element is represented as:

[0075] in, It is the central location; It is log-scale, i.e., the scale value; It is a unit quaternion; This is the opacity logit value; and These are the low-frequency and high-order spherical harmonic color coefficients, respectively. It is either a static / dynamic label or a dynamic probability.

[0076] Before step S110, the method provided in this application embodiment further includes: acquiring and preprocessing multiple videos of a four-dimensional dynamic scene to obtain camera parameters for each video and region masks corresponding to each video frame, wherein the region masks correspond to the video frames and camera viewpoints.

[0077] The preprocessing process includes, as previously described, multi-channel video synchronization, frame extraction, camera parameter estimation, sparse 3D point cloud generation, and image distortion correction. Region masks refer to image region labeling data corresponding to video frames and camera viewpoints, which will serve as dynamic region supervision data corresponding to video frames and camera viewpoints in subsequent processes. It is important to clarify that region masks are not used to distinguish the semantic categories of target objects; for example, they are not used to label hands, tools, materials, or finished products separately. Rather, they are used to distinguish moving objects from static backgrounds within the camera viewpoint of the corresponding video frame.

[0078] In short, a region mask is used to separate a moving target object (i.e., a moving object) from a static background. That is, a region mask corresponds to a video frame and a camera viewpoint. The video frame is identified by its frame number, and the camera viewpoint corresponds to a specific camera channel, thus it can be identified using the camera code of that channel. Based on this, region masks are stored using camera numbers and video frame numbers as indexes. For example, a region mask... Indicates the first v Road camera in the t The region mask corresponding to the nth video frame, for the nth... t Pixels in a video frame u Through area mask Determine the pixel u Is it projected onto a dynamic subject, or does it belong to a large static background? The projection area of ​​the same dynamic object differs under different camera perspectives, and its position and shape may also differ under different video frames. Therefore, the region mask needs to correspond to both the video frame and the camera perspective.

[0079] By making the region mask correspond to both video frames and camera viewpoints simultaneously, the subsequent training process can utilize region information from different times and viewpoints to constrain the spatial and dynamic properties of Gaussian units.

[0080] See Figure 2 , Figure 2 This is a flowchart illustrating a method for acquiring and preprocessing multiple video streams of a four-dimensional dynamic scene, obtaining camera parameters for each stream and region masks corresponding to each video frame, with the region masks corresponding to the video frames and camera viewpoints, according to an exemplary embodiment.

[0081] The embodiments of this application provide a step for acquiring and preprocessing multiple video streams of a four-dimensional dynamic scene to obtain camera parameters for each stream and region masks corresponding to each video frame. The region masks correspond to the video frames and camera viewpoints. This step includes: Step S201: For the synchronized and distortion-removed multi-view video frames, perform region mask candidate segmentation based on the dynamic region cue words configured for the dynamic subject to obtain candidate dynamic regions. Step S202: Based on the candidate dynamic regions and adjacent video frames, a frame-by-frame temporal mask is obtained by performing propagation, tracking and keying processing of the candidate dynamic regions in the video. Step S203: Perform morphological post-processing on the frame-by-frame temporal mask to obtain a region mask, and store the region mask with the camera number and the frame number of the corresponding video frame as indexes, so that the stored region mask corresponds one-to-one with the video frame through the camera number and the frame number.

[0082] These steps are explained in detail below.

[0083] Steps S201 to S203 are used to generate region masks corresponding to video frames and camera viewpoints. These region masks are used to distinguish between dynamic object regions and static background regions, and serve as supervision information for subsequent motion-static separation training. It should be noted that the region mask does not require semantic-level subdivision of hands, tools, materials, or product parts, but rather is used to identify regions that move, deform, or are significantly displaced relative to a static background.

[0084] In step S201, the synchronized and distortion-corrected multi-view image frames refer to multiple video frames that have been aligned to the same reference time axis and have undergone distortion correction according to camera parameters. Dynamic region cue words are prompts used to guide the segmentation model in locating dynamic object regions. They are configured based on the target object's display scene, and may include descriptions of people, hands, tools, manipulated objects, materials, semi-finished products, sporting goods, or other objects that may move or deform. Candidate dynamic regions are the initial segmentation results obtained based on the dynamic region cue words. Their purpose is to cover as many object regions as possible that require subsequent dynamic modeling, avoiding the omission of key dynamic subjects.

[0085] In a specific implementation of an exemplary embodiment, keyframes or sampled frames are selected from the synchronized and distortion-free multi-view image frames. Single-frame candidate segmentation is performed using dynamic region cues to obtain candidate dynamic regions. In cases where the hand and tool overlap, the tool and material are in contact, or the subject occludes the target object, overlapping objects are merged into the same dynamic region, without necessarily distinguishing the semantic boundaries of different objects. Whether the manipulated material is included in the dynamic region can be determined based on whether it undergoes deformation, movement, or posture change.

[0086] In step S202, the propagation of candidate dynamic regions in the video refers to expanding the candidate dynamic regions from keyframes or sampled frames to adjacent video frames; tracking refers to maintaining the position and range of the dynamic regions in consecutive video frames; keying refers to refining the boundaries of the dynamic regions to improve the separation effect between the dynamic regions and the static background. Frame-by-frame temporal mask refers to the sequence of dynamic region masks corresponding to each video frame.

[0087] In the specific implementation, the candidate dynamic regions obtained in step S201 are used as the initial mask. Combined with the image changes between adjacent video frames, the candidate dynamic regions are propagated and tracked through video. For regions with unstable boundaries, occlusion changes, or motion blur, the boundaries are further corrected through keying. As a result, the candidate dynamic regions on a small number of keyframes or sampled frames are expanded into a frame-by-frame temporal mask covering the video timeline, enabling subsequent motion-static separation training to obtain dynamic region supervision across multiple time frames.

[0088] In step S203, morphological post-processing includes at least one of opening, closing, hole filling, small connected region removal, edge smoothing, and edge feathering. Morphological post-processing reduces the impact of isolated noise, boundary jitter, and local holes in the mask on subsequent training. Uniform camera and temporal sampling refers to uniformly selecting region masks from different camera viewpoints and different time frames for storage, inspection, or training, to avoid excessive concentration of region masks in a few viewpoints or time periods.

[0089] In the specific implementation, the processed region masks are indexed and stored according to their respective camera numbers and video frame numbers. For example, the region mask corresponding to the t-th frame of the c-th camera is denoted as M(t,c), and it corresponds one-to-one with the t-th video frame of the c-th camera. In this way, during subsequent motion-static separation training, the region mask corresponding to the current training image is accurately read according to the camera number and frame number.

[0090] Through steps S201 to S203, this embodiment of the application obtains dynamic region supervision information corresponding to the camera viewpoint and video frame. The technical role of this region mask is not to provide semantic classification results, but to provide two-dimensional supervision for subsequent Gaussian-level dynamic-static separation training. During subsequent training, the dynamic response values ​​of each Gaussian can be rendered as a dynamic prediction mask, and the dynamic response values ​​of each Gaussian can be optimized by analyzing the difference between the dynamic prediction mask and the region mask obtained in this embodiment. Thus, the dynamic region information in the two-dimensional video frame can be converted into a three-dimensional Gaussian dynamic-static attribute partition, providing a foundation for determining a stable set of dynamic Gaussian elements, training a shared deformation field, and generating dynamic deformation flow.

[0091] The aforementioned camera parameters determine the projection and differentiable rendering relationship of Gaussian elements under different camera perspectives; the region mask provides supervision constraints between dynamic regions and static backgrounds for subsequent dynamic-static separation training. The original multi-channel videos are organized into training data containing temporal synchronization relationships, spatial geometric relationships, and dynamic-static region supervision relationships, thus providing the foundation for the three-stage Gaussian training starting from step S110.

[0092] Thus, in step S110, scene initialization training involves performing a complete Gaussian initialization of the scene based on multi-view video frames from multiple video streams and camera parameters. Specifically, based on the previously obtained sparse 3D point cloud, the 3D points in the sparse 3D point cloud are initialized as Gaussian elements; then, the Gaussian elements are differentiable based on the camera parameters to obtain the predicted image from the corresponding camera viewpoint; finally, based on the image differences between the predicted image and the actual captured video frames, the attributes of the Gaussian elements are optimized to obtain an initial Gaussian representation. The attributes of the Gaussian elements include their center position, scale, rotation, transparency, and color. The scene initialization training phase does not distinguish between the target object and the static background, but rather models the complete scene containing the target object, thereby providing a stable initial spatial representation for subsequent motion-static separation and deformation field training.

[0093] See Figure 3 , Figure 3 It is based on Figure 1 The flowchart of the corresponding embodiment describes the process of initializing the four-dimensional dynamic scene by using multi-view video frames and camera parameters from multiple video streams to obtain an initial Gaussian representation of the complete scene.

[0094] The embodiment of this application provides a step S110 for initializing the scene of a four-dimensional dynamic scene using multi-view video frames and camera parameters from multiple video streams to obtain an initial Gaussian representation of the complete scene. This step includes: Step S111: Based on the sparse 3D cloud corresponding to the multi-view video frame, initialize each 3D cloud in the sparse 3D cloud as a Gaussian element. Step S112: Perform differentiable rendering of Gaussian units based on the camera parameters corresponding to each camera viewpoint to obtain the predicted image under the corresponding camera viewpoint. Step S113: Based on the image difference between the predicted image and the actual video frame corresponding to the camera viewpoint, optimize the properties of the Gaussian unit to obtain an initial Gaussian representation representing the complete scene. The actual video frame is the video frame corresponding to the camera viewpoint among the multi-view video frames.

[0095] Step S110 is the first stage of the three-stage Gaussian training implemented in this application. Its purpose is to first establish a complete Gaussian representation of the scene where the target object is located, that is, to obtain a set of Gaussian primitives with initial geometry, appearance and visibility based on multi-view image frames, camera parameters and sparse three-dimensional point clouds, so as to be used for subsequent dynamic and static separation training and deformation field training.

[0096] With the sparse 3D point cloud obtained during the aforementioned camera parameter acquisition process, as described in step S202, triangulation of the cross-view matching points yields the sparse 3D point cloud, which is then initialized as Gaussian elements based on each sparse 3D point cloud. Because random initialization of Gaussian elements is difficult, sparse 3D point clouds are used to provide the foundation for Gaussian element construction.

[0097] For example, the process of initializing Gaussian elements from a sparse 3D point cloud includes the initialization of the corresponding attributes. The attributes of a Gaussian element include spatial position, scale, and rotation. Color and transparency The initialization process for various attributes includes: First, the positions of each sparse 3D point cloud are used as the initial center positions of the corresponding Gaussian elements. These initial center positions are configured as spatial positions, i.e., let the sparse 3D point cloud be... ,in N It is the number of point clouds. Characterizing the first i The spatial location of a sparse 3D point cloud; for each sparse 3D point cloud This sparse three-dimensional point cloud The position is configured as the spatial position of the initialized Gaussian element. ,Right now: In sparse 3D point clouds i Includes color Then the color Convert to low-frequency spherical harmonics, such as the initial value of low-frequency spherical harmonic color. Initial values ​​of higher-order spherical harmonic colors , , This is done to provide initial appearance information for the initialized Gaussian unit.

[0098] Furthermore, the initial scale of the Gaussian unit is determined based on the K-nearest neighbor distance of the sparse 3D point cloud, assuming... It is a sparse 3D point cloud i The initial scale is calculated using the following formula for the K nearest neighbors. : And set the initial log-scale to: .

[0099] in, Indicates the first i The neighborhood scale of a sparse 3D point cloud, It is the scaling factor. It is a constant used to avoid taking the logarithm of zero. As a result, Gaussian units in sparse regions of the point cloud will have a larger initial coverage area to avoid blurring dense details, while Gaussian units in dense regions of the point cloud will have a smaller initial coverage area to avoid missing details.

[0100] The rotations of Gaussian elements are initialized to unit quaternions, i.e. Opacity is initialized to a small constant. The corresponding logit value, i.e. .

[0101] Through the above initialization process, the scene initialization training does not start from a random spatial distribution, but from the sparse geometric structure obtained by multi-view reconstruction, so that the subsequent differentiable rendering optimization has a relatively stable initial geometric foundation.

[0102] With the completion of step S111, Gaussian element initialization, the Gaussian element set is obtained. Gausky metaset Represents the current Gaussian scene.

[0103] In step S112, the obtained Gaussian element set is projected onto the image plane of each camera viewpoint according to the corresponding camera parameters, and the corresponding predicted image is obtained through differentiable rendering. As mentioned above, the camera viewpoint is the shooting viewpoint corresponding to each video stream, and the camera parameters describe the mapping relationship of Gaussian elements projected from a unified three-dimensional world coordinate system to the corresponding camera image plane. Therefore, in the specific execution process, for any camera viewpoint, according to the camera parameters corresponding to that camera viewpoint, the currently obtained Gaussian element set is projected onto the image plane of that camera viewpoint, and the image is differentiablely rendered according to the attributes of each Gaussian element to obtain the predicted image under that camera viewpoint.

[0104] Since the rendering process of the predicted image is differentiable, the difference between the predicted image and the actual video frame can be backpropagated to the properties of the Gaussian units, thus enabling the Gaussian unit set to gradually fit the real image from multiple perspectives.

[0105] In step S113, by predicting the image difference between the image and the actual video frame corresponding to the camera viewpoint, Gaussian attribute optimization is performed to obtain an initial Gaussian representation representing the complete scene. The actual image frame is the video frame corresponding to the camera viewpoint among the multi-view image frames.

[0106] In this context, the actual captured video frames are image frames from multiple video streams that correspond to the current camera's viewpoint. Image differences can include pixel color differences, brightness differences, structural similarity differences, or weighted combinations thereof. Gaussian byte attribute optimization refers to adjusting the center position, scale, rotation, transparency, and color attributes of Gaussian bytes based on the differences between the predicted image and the actual captured video frames, so that the rendering result of the Gaussian byte set under multiple camera viewpoints gradually approximates the actual captured content.

[0107] In the specific implementation, training images are selected from multi-view image frames, and the corresponding camera parameters are read according to the camera view of the training image. Then, the current Gaussian set is differentiable and rendered to obtain the predicted image. The image difference between the predicted image and the corresponding real-world video frame is calculated, and the Gaussian attributes are optimized in reverse based on the image difference. During training, Gaussian proliferation, splitting, and pruning can also be performed based on the gradient magnitude, projection radius, opacity, visibility, or reconstruction error of the Gaussian to supplement under-represented areas in the scene and remove Gaussian with low contribution or instability.

[0108] For example, the obtained predicted image With the v Real-time video frames from a camera's perspective The comparison is performed, and the Gaussian meta-attributes are optimized based on the image differences. The image differences include structural similarity loss, as shown in the loss function below:

[0109] in, This represents the initial training loss for the scene; This represents the pixel difference between the predicted image and the actual captured video frame. Indicates structural similarity; This represents the weighting coefficient between the two.

[0110] The loss function described above constrains the predicted image to approximate the real image in terms of pixel color and also in terms of local structure.

[0111] Therefore, through algorithmic execution, the properties of Gaussian elements are optimized in reverse, and Gaussian element proliferation, splitting, and pruning are performed based on the gradient, projection radius, and opacity of Gaussian elements to supplement underexpressed regions and remove Gaussian elements with low contribution or instability.

[0112] This exemplary embodiment provides a stable Gaussian primitive foundation for subsequent static / dynamic separation training, enabling the learning of dynamic response values ​​to be built upon Gaussian primitives that already possess good geometric and aesthetic representations. Simultaneously, this stage also provides a reference spatial representation for subsequent deformation field training, avoiding the direct learning of temporal deformation on an unstable set of Gaussian primitives. Therefore, scene initialization training, subsequent static / dynamic separation training, and deformation field training form a progressive relationship, which is beneficial for improving the training stability and final display quality of the target object display video's transformation into Gaussian interactive content.

[0113] Step S110 will work in conjunction with step S120 to perform three-stage Gaussian training on the scene where the target object is located, based on multiple video streams, camera parameters, and region masks, to obtain a static Gaussian set, a dynamic Gaussian set, and a deformation field shared by the dynamic Gaussian set. The three-stage Gaussian training includes scene initialization training, static-dynamic separation training, and deformation field training for the dynamic Gaussian set.

[0114] Gaussian elements are the basic units of representation used to characterize 3D scenes, such as static backgrounds or target objects. Gaussian elements can have attributes such as center position, scale, rotation, transparency, color, or spherical harmonics. Multiple Gaussian elements together constitute a Gaussian representation of the scene containing the target object.

[0115] Specifically, the three-stage Gaussian training includes three sequentially related training stages. The first stage is the scene initialization training executed in step S110, which is used to obtain an initial Gaussian representation of the complete scene. The second stage is the static-dynamic separation training executed in the current step S120, which is used to divide the Gaussian primitives in the initial Gaussian representation into a set of static Gaussian primitives and a set of dynamic Gaussian primitives based on a region mask. The third stage is the deformation field training executed in the subsequent step S130, which is used to establish a shared deformation field for the set of dynamic Gaussian primitives, enabling the dynamic Gaussian primitives to recover attribute changes under different video frames based on spatial location and timestamp.

[0116] In the static-dynamic separation training performed in step S120, the geometric and appearance attributes of the initial Gaussian representation are first frozen. Based on this, a dynamic response value is introduced for each Gaussian primitive. The dynamic response value represents the degree to which the Gaussian primitive belongs to a dynamic object region. It should be further noted that the dynamic response value is a time-independent Gaussian primitive attribute; that is, for the same Gaussian primitive, its dynamic response value does not change with the video frame timestamp. At the level of expressed physical quantities, the dynamic response value of a Gaussian primitive is essentially a dynamic probability, characterizing the probability that the Gaussian primitive belongs to a dynamic object region.

[0117] During static / dynamic separation training, the original colors of Gaussian elements are not used as the mask training output. Instead, the dynamic probabilities of each Gaussian element are used as renderable scalar attributes. For any video frame and any camera viewpoint, the dynamic probabilities of the Gaussian elements are rendered onto the image plane based on the corresponding camera parameters, resulting in a dynamic prediction mask. Since the projection positions and visibility of Gaussian elements onto image pixels differ across different video frames and camera viewpoints, even though the dynamic response values ​​of the Gaussian elements themselves are time-independent, the dynamic prediction mask can still generate different 2D mask prediction results for different video frames and camera viewpoints. Subsequently, a region mask corresponding to the video frame and camera viewpoint is used for supervision. By analyzing the differences between the dynamic prediction mask and the region mask, the dynamic response values ​​of each Gaussian element are optimized.

[0118] After completing the static-dynamic separation training, based on the comparison results between the optimized dynamic response value and the preset threshold, the Gaussian elements in the initial Gaussian representation are divided into a static Gaussian element set and a dynamic Gaussian element set.

[0119] Because the dynamic response value is independent of time, once a Gaussian primitive is identified as a dynamic Gaussian primitive, it remains part of the dynamic Gaussian primitive set throughout the entire video timeline; conversely, once a Gaussian primitive is identified as a static Gaussian primitive, it remains part of the static Gaussian primitive set throughout the entire video timeline. This setting differs from the method of temporarily determining the dynamic or static state of Gaussian primitives for each video frame, thus avoiding frequent switching of Gaussian primitive identities between adjacent frames.

[0120] Under the influence of time-independent dynamic response values, the dynamic Gaussian primitive set maintains stable membership and index relationships during training, export, compression, and playback, providing a foundation for the one-to-one correspondence between the subsequent dynamic basis and dynamic deformation flow. Since the dynamic response values ​​do not change over time, the dynamic Gaussian primitives can form a fixed baseline representation. The dynamic Gaussian primitives can learn their own continuous changes over time from the shared deformation field, rather than reselecting different dynamic Gaussian primitives in each frame. The existence of dynamic response values ​​effectively reduces temporal instability, deformation field learning difficulties, and playback-end index confusion caused by dynamic identity perturbations, thereby improving the continuity of dynamic Gaussian primitive modeling and the stability of terminal playback.

[0121] For further details, please refer to [link / reference]. Figure 4 , Figure 4 It is based on Figure 1 The corresponding embodiment shows a flowchart describing the steps of performing static and dynamic separation training on Gaussian primitives in the initial Gaussian representation based on the initial Gaussian representation, camera parameters, and region masks corresponding to multi-view video frames and camera viewpoints to obtain a set of static Gaussian primitives and a set of dynamic Gaussian primitives.

[0122] The step S120 provided in this application embodiment, which involves performing static and dynamic separation training on Gaussian primitives in the initial Gaussian representation based on the initial Gaussian representation, camera parameters, and region masks corresponding to multi-view video frames and camera viewpoints, to obtain a set of static and dynamic Gaussian primitives, includes: Step S121: Set dynamic response values ​​for each Gaussian element in the initial Gaussian representation. The dynamic response values ​​are Gaussian element-level parameters that are independent of time. Step S122: Based on the parameters of each camera, render the dynamic response values ​​of each Gaussian primitive to the corresponding video frame and camera viewpoint to obtain the dynamic prediction mask. Step S123: Use the region mask corresponding to the video frame and the camera viewpoint for supervision, and optimize the dynamic response value of each Gaussian unit by dynamically predicting the difference between the mask and the corresponding region mask. Step S124: Based on the optimized dynamic response value, the Gaussian elements in the initial Gaussian representation are divided into a set of static Gaussian elements and a set of dynamic Gaussian elements.

[0123] Dynamic response value These are trainable parameters at the Gaussian level, used to represent the th... i The tendency of a dynamic Gaussian property. For example, the dynamic response value is expressed through the Sigmoid function. The mapping is to dynamic probabilities, that is: , .

[0124] in, Let represent the dynamic probability of the i-th Gaussian element. The closer the value is to 1, the more likely the Gaussian element is to be a dynamic Gaussian element; The closer it is to 0, the more likely the Gaussian element is to belong to the static background, that is, the higher the probability of it being included in the set of static Gaussian elements.

[0125] It should be noted that the dynamic response value and the dynamic probability derived from it All are time-independent Gaussian primitive attributes. That is, for the same Gaussian primitive, its dynamic response value will not change with the video frame timestamp. This setting ensures that once a Gaussian primitive is identified as dynamic, it remains part of the dynamic Gaussian primitive set throughout the entire video timeline; once identified as static, it remains part of the static Gaussian primitive set throughout the entire video timeline. This avoids frequent switching between dynamic and static identities for Gaussian primitives between adjacent video frames, providing a foundation for maintaining consistency in the dynamic base, dynamic deformation flow, and dynamic Gaussian primitive indexes during terminal playback.

[0126] During training, the original color attributes of Gaussian elements are not used as the rendering output; instead, the dynamic probabilities of each Gaussian element are used. As a renderable scalar property. For time t The video frame (i.e. the first frame in the video) t (video frames) and camera perspective v Adapting the camera parameters to the camera's viewpoint, each Gaussian primitive in the initial Gaussian representation is rendered to the corresponding image plane to obtain the time step. t Camera perspective v Dynamic prediction mask :

[0127] in, It is a differentiable rendering process that uses dynamic probability as a rendering attribute. It is the initial Gaussian representation. It is the first v Camera intrinsics from a single camera perspective. It indicates time. t Next v The camera extrinsic parameters or equivalent projection parameters corresponding to each camera viewpoint.

[0128] For pixels in a dynamic prediction mask u The corresponding value is obtained by accumulating the dynamic probabilities of multiple Gaussian elements at that pixel location according to their rendering contributions:

[0129] in, It is the first i At any given moment, a high-level element... t Pixels under camera perspective v u The rendering contribution weights. Although the dynamic probability of each Gaussian primitive. While inherently independent of time, the dynamic prediction mask ultimately results from variations in the visibility, projection position, and rendering contribution of Gaussian elements across different video frames and camera perspectives. It can still generate different two-dimensional dynamic region prediction results for different video frames and camera perspectives.

[0130] Subsequently, a region mask corresponding to the video frame and camera viewpoint is used. For dynamic prediction masks Supervision is performed. In this process, the loss function used can be the binary cross-entropy loss function, i.e.:

[0131] in, Represents the set of image pixels of a video frame. This is a stabilizing term used to avoid taking the logarithm of zero. Under the influence of the loss function, if the region mask... Indicator Pixel u If it belongs to a dynamic Gaussian unit, then training will enable dynamic prediction of the mask. Approaching 1; if the region mask Indicator Pixel u If it belongs to a static Gaussian unit, then training will prompt dynamic prediction of the mask. Approaching 0. Through joint supervision using region masks from multiple video frames and multiple camera perspectives, the dynamic region information in the 2D image is inversely constrained to the dynamic response value of the 3D Gaussian primitive.

[0132] Building upon this, for example, sparse regularization or class balancing weights can be added to the loss function. Sparse regularization is used to suppress excessive Highskirches from being classified as dynamic Highskirches, avoiding excessive expansion of dynamic regions. Class balancing weights are used to alleviate the training bias caused by the imbalance between the areas of dynamic image regions and static background regions. The above constraints can be selected based on the quality of the region mask, the area of ​​the dynamic region, and the scene complexity.

[0133] During this training process, the optimization only includes the dynamic response values ​​of each Gaussian primitive, without updating attributes such as the center position, scale, rotation, opacity, and color of the Gaussian primitives. The advantage of this approach is that the static-dynamic separation training does not destroy the complete scene geometry and appearance representation already obtained in the first stage. Instead, it specifically learns the static-dynamic attributes of the Gaussian primitives, making the static-dynamic separation results more stable.

[0134] After training is complete, based on the optimized dynamic probabilities With preset dynamic threshold Based on the comparison results, dynamic and static labels are determined for each Gaussian element. The Gaussian elements are then incorporated into either a dynamic Gaussian element set or a static Gaussian element set based on these determined labels.

[0135] in, Indicates the first i The Gaussian element belongs to the dynamic Gaussian element. Indicates the first i Each Gaussian element belongs to a static Gaussian element.

[0136] Finally, a dynamic Gaussian set is obtained. and static Gaussian set .

[0137] Through the aforementioned dynamic-static separation training, the two-dimensional dynamic-static supervision in the region mask is transformed into a three-dimensional Gaussian-level dynamic-static partition. Since the dynamic response value is time-independent, the resulting dynamic Gaussian set remains stable throughout the entire video timeline. Subsequent deformation field training only establishes a time-dependent deformation field based on this stable dynamic Gaussian set. The subsequent compression stage also generates a dynamic basis and frame-by-frame dynamic deformation flow based on this stable set. During the terminal playback stage, the deformation data corresponding to the target video frame is written back to its attributes based on the stable dynamic Gaussian index. Therefore, this embodiment not only achieves dynamic-static separation but also establishes a unified data foundation for deformation modeling, resource compression, and frame-by-frame updates at the playback end.

[0138] In the deformation field training of step S130, a time-dependent deformation field is established only for the dynamic Gaussian elements obtained by separating static and dynamic elements. The static Gaussian element set retains the attributes learned during scene initialization training and is not processed by the deformation field; the dynamic Gaussian element set saves its basic attributes in the baseline state and predicts attribute changes in the corresponding video frames through the deformation field. In an exemplary embodiment, the deformation field structurally includes a HashGrid Encoder, a Multilayer Perceptron (MLP Decoder), and a function structure of three independent output heads connected to the MLP. These three independent output heads are used to output the attribute changes of the Gaussian elements, such as spatial position, scale, and rotation.

[0139] Specifically, the spatial position of the dynamic Gaussian primitive and the timestamp of the corresponding video frame are input into the deformation field to obtain the attribute changes of the dynamic Gaussian primitive under the corresponding video frame. The input spatial position of the dynamic Gaussian primitive is the reference spatial position of this Gaussian primitive. The attribute changes include spatial position changes, scale changes, and / or rotation changes. In addition, transparency changes and color changes may also be included.

[0140] After obtaining the attribute changes of the deformation field output, the attributes of the dynamic Gaussian primitives in the corresponding video frame are updated by these attribute changes, and the dynamic Gaussian representation of the corresponding video frame can be obtained.

[0141] The dynamic Gaussian representation of the corresponding video frame is synthesized with the static Gaussian set to obtain the current Gaussian scene of the corresponding video frame; the current Gaussian scene is differentiable based on the corresponding camera parameters to obtain the predicted image from the corresponding camera view; and the deformation field is optimized based on the image difference between the predicted image and the multi-view image frames from the corresponding camera view.

[0142] In one exemplary embodiment, before performing deformation field training, an initial deformation field is first constructed, and the initial deformation field is initialized to output zero deformation or near-zero deformation. In this way, based on the initial deformation field, the dynamic Gaussian units are in a property close to their baseline state at the beginning of training. Subsequently, the temporal change relationship is gradually learned through supervision by real video frames, thereby reducing the risk of large drift or deformation explosion of dynamic Gaussian units in the early stage of training.

[0143] In one exemplary embodiment, deformation field training will also incorporate temporal smoothing constraints, deformation amplitude regularization, scale stability constraints, rotational continuity constraints, mask constraints, or cross-view consistency constraints. Temporal smoothing constraints are used to suppress abrupt deformation changes between adjacent frames; deformation amplitude regularization is used to prevent dynamic Gaussian elements from generating excessive or unreasonable offsets; scale stability constraints and rotational continuity constraints are used to improve the geometric continuity of dynamic Gaussian elements between different video frames.

[0144] For example, based on the corresponding camera parameters, differentiable rendering of the current Gaussian scene yields the predicted image from the corresponding camera's viewpoint. This can be expressed by the following formula:

[0145] in, It is the rendering function. It is the first t The current Gaussian scene in the nth video frame. This Gaussian scene is the nth... t Under one video frame (i.e.) t The current Gaussian scene for the corresponding video frame is obtained by synthesizing the dynamic Gaussian representation (at any given moment) with the static Gaussian element set. It is the first v Road camera internal reference, It is the first t The first video frame v External parameters of the road camera.

[0146] As mentioned above, based on the parameters of each camera and the region mask, step S120 can transform the dynamic and static region supervision in the two-dimensional video frame into a three-dimensional Gaussian level dynamic and static partitioning; step S120 further uses time-independent dynamic response values ​​to determine a stable dynamic Gaussian set, and based on this set, it can generate a static basis, a dynamic basis, and a dynamic deformation flow by performing training on the shared deformation field in step S130.

[0147] See Figure 5 , Figure 5 It is based on Figure 1The corresponding embodiment shows a flowchart describing the steps of training a deformation field on a dynamic Gaussian set, obtaining a shared deformation field for the dynamic Gaussian set, and generating a dynamic deformation flow for each video frame based on the deformation field.

[0148] The step S130, which involves training the dynamic Gaussian set with a deformation field to obtain the shared deformation field of the dynamic Gaussian set, and generating the dynamic deformation stream of the dynamic Gaussian set corresponding to each video frame through the deformation field, as shown in the embodiments of this application, includes: Step S131: Input the spatial position of the dynamic Gaussian element in the dynamic Gaussian element set and the timestamp of the corresponding video frame into the deformation field to obtain the attribute changes of the dynamic Gaussian element under the video frame. The attribute changes include spatial position changes, scale changes and / or rotation changes. Step S132: Based on the attribute changes of the dynamic Gaussian primitives under the video frame, update the Gaussian attributes of the dynamic Gaussian primitives under the video frame to obtain the dynamic Gaussian representation under the corresponding video frame. Step S133: Combine the dynamic Gaussian representation of the corresponding video frame with the static Gaussian set to obtain the current Gaussian scene of the corresponding video frame; Step S134: Perform differentiable rendering of the current Gaussian scene based on the corresponding camera parameters to obtain the predicted image from the corresponding camera view. Step S135: Optimize the deformation field based on the image differences between the predicted image and the multi-view video frames from the corresponding camera viewpoint.

[0149] As mentioned earlier, deformation field training no longer establishes temporal deformation relationships for all Gaussian primitives, but only establishes a shared deformation field for the dynamic Gaussian primitive set; the static Gaussian primitive set retains the Gaussian properties learned in the first stage and directly participates in subsequent rendering. This avoids the incorrect introduction of temporal deformation into static backgrounds or fixed props, while allowing the deformation field to focus on learning the position, scale, and rotation changes of dynamic Gaussian primitives over time.

[0150] For the first in the dynamic Gaussian set i A dynamic Gaussian element, whose properties in the baseline state are denoted as: ,in, Indicates the first i The spatial position of a dynamic Gaussian element in the baseline state. Indicates its reference scale, Indicates its base rotation quaternion, This represents the logit value, indicating the opacity. This represents the color or spherical harmonic color attribute. Since the dynamic Gaussian set is determined by time-independent dynamic response values ​​in the previous stage, it has stable membership and index order throughout the time axis, which is beneficial for subsequent deformation field training and frame-by-frame deformation data generation.

[0151] During deformation field training, the spatial location of the dynamic Gaussian unit and the timestamp of the corresponding video frame are used together as the deformation field input. For the th i A dynamic Gaussian element at time... t The input can be represented as In expanded coordinate form, it is represented as , t This is the timestamp of the corresponding video frame. After receiving this input, the deformation field outputs the attribute changes of the dynamic Gaussian elements in the corresponding video frame, i.e.:

[0152] in, Indicates a change in spatial location. Indicates scale change. This indicates a rotational change. In other exemplary embodiments, the aforementioned attribute changes may be extended to changes in transparency, color, or other Gaussian attributes, depending on the actual implementation, but at least include changes in spatial position, scale, and rotation to express the geometric state of a dynamic object within the target video frame.

[0153] After obtaining attribute changes, update the dynamic Gaussian meta-element. i In video frames t By examining the attributes below, you can obtain the corresponding video frame. t Dynamic Gaussian representation The update process is as follows:

[0154]

[0155]

[0156]

[0157] in, Indicates the first i The spatial position of a dynamic Gaussian element in video frame t. Indicates scale. Indicates rotation, This indicates that the quaternion is normalized to ensure that the rotation representation satisfies the unity quaternion constraint.

[0158] For a static Gaussian element set, which remains invariant at video frame t without passing through a deformation field, it is represented as: Video t The dynamic Gaussian representation is synthesized with the static Gaussian tuple to obtain the current Gaussian scene for that video frame: .

[0159] Differentiable rendering of the current Gaussian scene is performed based on the camera parameters corresponding to the camera's viewpoint, thus obtaining the corresponding camera viewpoint. v Predicted image below Predict the image Real multi-view image frames from the corresponding camera perspective The images are compared, and the deformation field is optimized based on the differences between them.

[0160] In step S130, a dynamic deformation stream corresponding to each video frame is generated using the deformation field obtained from training. The dynamic deformation stream is a data stream generated based on the dynamic Gaussian element set and the deformation field, corresponding to the attribute changes of each video frame. For example, the dynamic deformation stream includes the position changes, scale changes, and rotation changes of the dynamic Gaussian elements in each video frame.

[0161] A dynamic deformation stream is generated from a dynamic Gaussian set using a deformation field, corresponding to each video frame. Since the dynamic Gaussian set is determined by time-independent dynamic response values, its membership and order remain stable throughout the reference timeline. Therefore, the dynamic deformation data for each video frame can be organized according to the same dynamic Gaussian index order. This stable indexing relationship allows the dynamic deformation stream to be compressed and stored as frame-by-frame attribute label maps, frame-by-frame binary data, video attribute streams, or other forms without the index misalignment problem caused by different video frames corresponding to different Gaussian sets.

[0162] As the static and dynamic Gaussian set is obtained through the static-dynamic separation training in step S120, and the dynamic deformation flow generated by the deformation field obtained in step S130 on the dynamic Gaussian set, the interactive video resources of the four-dimensional dynamic scene are generated through the execution of split quantization compression in step S140.

[0163] In step S140, the static basis obtained by transforming the static Gaussian set, the dynamic basis obtained by transforming the dynamic Gaussian set, and the dynamic deformation flow generated by the deformation field of the dynamic Gaussian set are subjected to separate quantization compression to generate interactive video resources of a four-dimensional dynamic scene.

[0164] A static basis is a resource-based representation obtained by transforming a set of static Gaussian primitives, used to store the geometry, transparency, and appearance attributes of the static Gaussian primitives. A dynamic basis is a resource-based representation obtained by transforming a set of dynamic Gaussian primitives, used to store the basic attributes of the dynamic Gaussian primitives in the baseline state or canonical state.

[0165] In the specific implementation process, the static Gaussian element set is first converted into a static basis, and the dynamic Gaussian element set is converted into a dynamic basis. After attribute quantization encoding of both the static and dynamic basis, they are mapped to a two-dimensional attribute map. More specifically, the attributes of the Gaussian elements include spatial position, scale, and rotation. Normalization, quantization, and two-dimensional image representation are performed on each attribute to obtain the corresponding two-dimensional attribute map.

[0166] The attribute changes in the dynamic deformation stream are quantized and encoded frame by frame to obtain the Gaussian deformation attribute label map corresponding to the video frame. The Gaussian deformation attribute label map maps the deformation variables corresponding to various attributes such as spatial position, scale, and rotation of the dynamic Gaussian elements, i.e., attribute changes, to label values ​​in image pixels. Further, the positional changes corresponding to spatial position are decomposed into high-bit labels and low-bit labels, and the attribute changes corresponding to scale (scale change) and rotation (rotation change) are mapped to bit labels, which are then stored in separate channels or regions. Finally, they are organized into a two-dimensional atlas arrangement to obtain a dynamic attribute atlas sequence. After encoding by an encoder, dynamic attribute compression is completed to obtain the Gaussian deformation attribute label map stored in H.265 / HEVC encoding mode.

[0167] As step S140 is executed, the obtained two-dimensional attribute map corresponding to the static base, the two-dimensional map corresponding to the dynamic base, the Gaussian deformation attribute label map corresponding to each video frame, and the corresponding generated resource metadata are packaged to obtain interactive video resources of the four-dimensional dynamic scene.

[0168] In this regard, it should be further explained that resource metadata is a specification document generated for the quantization compression implementation, providing the necessary information for subsequent decoding and recovery. For example, resource metadata includes the number of static Gaussian bits, the number of dynamic Gaussian bits, the attribute graph size, quantization parameters, attribute arrangement order, frame rate, frame count, timestamp index, dynamic deformation stream index, dynamic representation method, resource version number, or other information required for parsing.

[0169] In step S140, the training results obtained in steps S120 and S130 are converted from the training-state Gaussian model into resource-state data suitable for loading and playback on the terminal device. Through the separation, quantization, and compression of the static basis, dynamic basis, and dynamic deformation flow, the interactive video resource is no longer a collection of multiple original video files, nor does each video frame correspond to a complete 3D model. Instead, it uses a stable set of static Gaussian primitives, a stable set of dynamic Gaussian primitives, and frame-by-frame deformation data to jointly express the four-dimensional dynamic scene display process in which the dynamic subject resides. This resource organization method, in conjunction with the aforementioned dynamic response values, ensures that the dynamic basis and dynamic deformation flow remain consistent at the index level, providing a foundation for subsequent frame-by-frame recovery by the terminal.

[0170] See Figure 6 , Figure 6 It is based on Figure 1 The flowchart described in the corresponding embodiment describes the steps of performing separate quantization compression on the static basis obtained by transforming the static Gaussian set, the dynamic basis obtained by transforming the dynamic Gaussian set, and the dynamic deformation flow to generate interactive video resources for a four-dimensional dynamic scene.

[0171] The step S140 provided in this application embodiment, which involves performing separate quantization compression on the static basis obtained by transforming the static Gaussian set, the dynamic basis obtained by transforming the dynamic Gaussian set, and the dynamic deformation flow to generate interactive video resources for a four-dimensional dynamic scene, includes: Step S141: Perform attribute quantization encoding on the static and dynamic bases, and map the quantized Gaussian attributes into a two-dimensional attribute graph. Step S142: Quantize and encode the attribute changes in the dynamic deformation stream frame by frame to obtain the Gaussian deformation attribute label map corresponding to the video frame. Step S143: Package the two-dimensional attribute map and the Gaussian deformation attribute label map corresponding to each video frame to generate interactive video resources for a four-dimensional dynamic scene. In this context, the dynamic Gaussian elements in the dynamic basis maintain an index correspondence with the deformation attributes in the Gaussian element deformation attribute label map corresponding to each video frame.

[0172] Static basis is used to store the properties of static Gaussian primitives, while dynamic basis is used to store the properties of dynamic Gaussian primitives in the baseline state. To facilitate the recovery of Gaussian primitive properties by terminal devices through image decoding, texture uploading, or binary parsing, the one-dimensional Gaussian properties are quantized and mapped to a two-dimensional property map.

[0173] When mapping the quantized static and dynamic bases to a 2D attribute map, each Gaussian primitive corresponds to a pixel position or a fixed tile position in the 2D attribute map. The width and height of the 2D attribute map are determined based on the number of Gaussian primitives, and unused positions can be filled with default values. Furthermore, to improve compression efficiency and access locality, Gaussian primitives will be arranged according to spatial order, Morton order, or other preset sorting methods. It is important to note that the sorting of dynamic Gaussian primitives in the dynamic base should be consistent with the sorting in the subsequent frame-by-frame dynamic deformation data; otherwise, the deformation data will not be accurately applied to the corresponding dynamic Gaussian primitives during terminal playback.

[0174] After completing the deformation field training through the aforementioned steps, the timestamps of each video frame are input sequentially according to the exported timeline. The attribute changes of the dynamic Gaussian set under each video frame are obtained through the deformation field. In the execution of step S142, the dynamic deformation data corresponding to each video frame is quantized, and the quantized attribute changes are encoded into a Gaussian deformation attribute label map. The Gaussian deformation attribute label map is not the final display screen, but a data map used to store the quantized labels of the Gaussian deformation attributes. After the terminal device decodes the label map, it interprets the pixel values ​​in it as quantization labels, and then dequantizes and recovers the deformation data according to the quantization parameters in the resource package.

[0175] In one exemplary embodiment, positional changes can be quantized using 16-bit quantization and split into low 8 bits and high 8 bits for storage; scale changes and rotation vectors can be quantized using 8-bit quantization. Correspondingly, the attribute changes of each video frame are organized into an attribute label map including low position bits, high position bits, scale changes, and rotation changes. For example, the low position bits, high position bits, scale changes, and rotation vectors can be arranged in different regions of a two-dimensional map, or stored as different attribute map files.

[0176] Similarly, Gaussian meta-deformation attribute label maps from multiple consecutive video frames are combined into an image sequence or video attribute stream, and compressed using WebP, bin sequences, H.265 / HEVC, or other encoding methods. Unlike ordinary video encoding, the encoding object here is the Gaussian meta-attribute label map, not the image screen viewed by the user. Therefore, when using video encoding, priority should be given to encoding configurations that can maintain the accuracy of label values ​​as much as possible to reduce the impact of color space conversion, chroma sampling, or compression errors on deformation data recovery.

[0177] In step S134, the two-dimensional attribute map corresponding to the static basis, the two-dimensional attribute map corresponding to the dynamic basis, the Gaussian deformation attribute label map corresponding to each video frame, and the metadata are packaged together to generate an interactive video resource for a four-dimensional dynamic scene. The metadata records the resource version, frame rate, number of frames, number of static Gaussian elements, number of dynamic Gaussian elements, attribute map size, sorting method, quantization range, dynamic representation method, frame-by-frame file index, dynamic basis reference relationship, and verification information, etc.

[0178] Therefore, the exemplary embodiment of this application converts the floating-point Gaussian attributes obtained during the training phase into a resource packet structure suitable for loading and decoding by the terminal device. The resulting interactive video is neither multiple original video files nor a complete set of Gaussian models frame by frame, but is composed of a static basis, a dynamic basis, and a frame-by-frame dynamic deformation flow. The static basis is used to recover a stable scene representation in one go, the dynamic basis is used to provide the baseline state of the dynamic Gaussian primitives, and the dynamic deformation flow is used to recover the attribute changes of the dynamic Gaussian primitives frame by frame. Thus, while maintaining the recoverability of the dynamic state of the target object, it avoids repeatedly storing the complete scene Gaussian attributes for each frame, and provides a data foundation for frame-by-frame decoding, local attribute write-back, and real-time interactive rendering during the terminal playback phase.

[0179] Through steps S110 to S140, this embodiment of the application converts preprocessed multi-channel video of a four-dimensional dynamic scene into interactive video resources. These interactive video resources consist of a static base, a dynamic base, and a dynamic deformation stream. Static content only needs to be expressed once in the resource, while dynamic content is expressed through a baseline state and frame-by-frame deformation data. Therefore, the generated resources reduce the redundant storage of complete scene data frame by frame and provide a data foundation for dynamic state recovery and free-viewpoint interactive display in subsequent target video frames.

[0180] In one exemplary embodiment, the generated interactive video resources are loaded, decoded, and rendered in the terminal device. The terminal device parses the static and dynamic bases and decodes the dynamic deformation stream to obtain deformation data corresponding to the video frame. In response to a display request for the target video frame, the terminal device generates a Gaussian representation of the target video frame based on the deformation data, static base, and dynamic base, and renders the Gaussian representation of the target video frame in real time to obtain the interactive display screen under the target video frame.

[0181] Specifically, the terminal device merges the static Gaussian primitives recovered from the static basis and the dynamic Gaussian primitives recovered from the dynamic basis into a unified Gaussian rendering object, and records the index range of the dynamic Gaussian primitives within this Gaussian rendering object. When the target video frame changes, the terminal device only performs attribute write-back on the corresponding region of the dynamic Gaussian primitives, while the static Gaussian primitives remain unchanged. Attribute write-back includes updating the position, scale, and / or rotation attributes of the dynamic Gaussian primitives. For transparent Gaussian rendering, after the position of the dynamic Gaussian primitives changes, the sorting center, bounding box, or rendering auxiliary data are updated synchronously. Therefore, the terminal device does not need to reconstruct the complete Gaussian scene frame by frame, and supports users to rotate, scale, translate, or change the virtual camera's viewpoint within the target video frame to achieve free-viewpoint interactive display.

[0182] In one implementation, the scene initialization training in step S110 is implemented in parallel using multiple GPUs. Specifically, after obtaining multiple Gaussian primitives based on sparse 3D point clouds, the multiple Gaussian primitives are divided into multiple Gaussian primitive slices according to their index or the spatial region where they are located, and different Gaussian primitive slices are assigned to different graphics processors. Each graphics processor holds the center position, scale, rotation, transparency, and appearance attributes of the corresponding Gaussian primitive slice, and optimizes the Gaussian primitive slices it holds.

[0183] When performing differentiable rendering for a specific camera viewpoint, each graphics processor (GPU) calculates the rendering contribution of its Gaussian primitive (GMT) piece to the predicted image based on its held GMT pieces, the camera parameters corresponding to that viewpoint, and the current GMT attributes. The rendering contributions calculated by different GPUs are then aggregated to obtain the predicted image for that camera viewpoint. During backpropagation based on the image differences between the predicted image and the actual captured video frames, the gradients corresponding to each GMT piece are returned to the GPU holding that GMT piece, which then updates its local GMT attributes. Therefore, step S110 can complete scene initialization training without requiring a single GPU to hold all GMT attributes, reducing the memory usage of a single GPU.

[0184] In one implementation, the static / dynamic separation training in step S120 follows the Gaussian element fragmentation relationship formed in step S110. Since the geometric and appearance attributes of the initial Gaussian representation remain frozen in step S120, each graphics processor maintains dynamic response values ​​for each Gaussian element in its held Gaussian element fragments and maps these dynamic response values ​​to dynamic probabilities. For any video frame and camera viewpoint, each graphics processor calculates the corresponding local dynamic mask contribution based on the dynamic probability of its local Gaussian element fragment, and aggregates the local dynamic mask contributions of each graphics processor to obtain the complete dynamic prediction mask.

[0185] After calculating the static / dynamic separation loss based on the difference between the complete dynamic prediction mask and the corresponding region mask, the gradient of the dynamic response value is returned to the graphics processor (GPU) where the corresponding Gaussian element resides. This GPU then updates the dynamic response value of its local Gaussian element. After training, each GPU determines whether its local Gaussian element is a static or dynamic Gaussian element based on its dynamic probability, and summarizes the static / dynamic separation results according to the global Gaussian element index, obtaining a set of static Gaussian elements and a set of dynamic Gaussian elements. Therefore, step S120 can maintain consistency between the static / dynamic separation results and the global Gaussian element index under piecewise parallel conditions.

[0186] In one implementation, the deformation field training in step S130 is performed based on the dynamic Gaussian unit set obtained in step S120. For dynamic Gaussian units assigned to different graphics processors, each graphics processor maintains the properties of its local dynamic Gaussian units in the baseline state. The deformation field network is stored in a replicated manner on multiple graphics processors, using the same network structure, the same temporal input method, and the same spatial normalization boundary. The spatial normalization boundary is determined by the global bounding box of the dynamic Gaussian unit set and is synchronized to each graphics processor before the deformation field training begins.

[0187] During deformation field training, each graphics processor (GPU) inputs the reference spatial position of its local dynamic Gaussian primitives (LGPs) and the timestamp of the corresponding video frame into the deformation field network to obtain the position, scale, and / or rotation changes of the LLPs in the corresponding video frame. The attributes of the LLPs in the corresponding video frame are then updated based on these attribute changes. Each GPU calculates the rendering contribution of its local static Gaussian primitives and the updated local dynamic Gaussian primitives to the predicted image, and the predicted image for the current video frame and camera viewpoint is obtained through distributed differentiable rendering aggregation.

[0188] When backpropagating based on the image differences between the predicted image and the corresponding multi-view video frames, the gradients of the local dynamic Gaussian primitive attributes are returned to the corresponding graphics processor (GPU). The gradients of the deformation field network parameters are synchronized across multiple GPUs to ensure consistency of the deformation field network on each GPU. During the warmup phase of deformation field training, each GPU freezes the baseline attributes of the local dynamic Gaussian primitives and only updates the deformation field network parameters synchronously. After the warmup, each GPU jointly optimizes the baseline attributes of the local dynamic Gaussian primitives and the deformation field network.

[0189] During the interactive video resource export phase, each graphics processor (GPU) aggregates its local static Gaussian primitives, dynamic Gaussian primitives, and dynamic deformation data for the corresponding video frames into the export process. The export process generates static bases, dynamic bases, and dynamic deformation streams according to the global Gaussian primitive index order, maintaining consistency between the dynamic Gaussian primitive order in the dynamic base and the deformation attribute order in the dynamic deformation stream. Therefore, multi-GPU parallel training does not disrupt the stable indexing relationship of the dynamic Gaussian primitive set during the export, compression, and terminal playback phases.

[0190] Thus, based on the method implementation provided by the exemplary embodiments of this application, an end-to-end dynamic playback implementation is provided for the generated interactive video resources, namely, the terminal playback embodiment. Under the action of the interactive video resources generated in the embodiments of this application, excessive limitations on browsers, memory, and players in dynamic scene video playback can be avoided.

[0191] In one exemplary embodiment, interactive video resources are loaded, decoded, and rendered in a terminal device to enable interaction of a four-dimensional dynamic scene within a target video frame. The method implementation provided in this application further includes: The interactive video resources are loaded into the terminal device, the static and dynamic bases are parsed, and the dynamic deformation stream is decoded to obtain the deformation data corresponding to the video frame. The static base is used to recover the static Gaussian elements, the dynamic base is used to recover the attributes of the dynamic Gaussian elements in the baseline state, and the deformation data is used to represent the attribute changes of the dynamic Gaussian elements in the corresponding video frame. In response to the target video frame pointed to by the player's timeline during dynamic playback, a Gaussian representation of the target video frame is generated for the static and dynamic bases based on the deformation data corresponding to the target video frame. Real-time rendering of the target video frame using Gaussian representation yields the interactive display screen under the target video frame.

[0192] After loading the interactive video resources, the terminal device first parses the resource metadata within the interactive video resources. The resource metadata describes the correspondence between the static basis, dynamic basis, and dynamic deformation stream. The resource metadata includes resource version, frame rate, frame count, number of static Gaussian primitives, number of dynamic Gaussian primitives, dynamic representation method, attribute map size, quantization range, rotation encoding method, sorting method, frame-by-frame file index, and dynamic basis reference relationships. Based on the resource metadata, the terminal device determines the file locations, attribute arrangement methods, and inverse quantization parameters of the static basis, dynamic basis, and dynamic deformation stream.

[0193] Furthermore, interactive video resources are loaded into the terminal device, static and dynamic bases are parsed to obtain them, and the dynamic deformation stream is decoded to obtain deformation data corresponding to the video frames, including: The interactive video resources are loaded and parsed in the terminal device to obtain the two-dimensional attribute map corresponding to the static basis, the two-dimensional attribute map corresponding to the dynamic basis, and the Gaussian deformation attribute label map corresponding to each video frame. The two-dimensional attribute map is dequantized to recover the static and dynamic basis. The Gaussian deformation attribute label map corresponding to each video frame is decoded and dequantized to obtain the deformation data corresponding to each video frame. The Gaussian deformation attribute label map is a data map used to carry the deformation attributes of dynamic Gaussian. The dynamic Gaussian in the dynamic basis maintains an index correspondence with the deformation attributes in the Gaussian deformation attribute label map corresponding to each video frame. The terminal device can accurately apply the deformation data of the target video frame to the corresponding dynamic Gaussian.

[0194] When loading static and dynamic bases, the terminal device decodes and inverse-quantizes the two-dimensional attribute maps corresponding to the static and dynamic bases to recover the attributes of the static and dynamic Gaussian primitives in the baseline state. These attributes include center position, scale, rotation, transparency, and appearance. For attributes stored using high and low position tags, the terminal device combines the low and high tags into a position quantization value and performs inverse quantization according to the quantization range in the resource metadata. For attributes stored using log-scale, quaternion compression, rotation vector, or codebook methods, the terminal device performs the corresponding inverse transform according to the encoding method declared in the resource metadata.

[0195] For dynamic deformation streams, the terminal device determines the storage method of the current interactive video resource—whether it uses a frame-by-frame attribute file, a binary packet, or a video-based attribute stream—based on the resource metadata. If the dynamic deformation stream uses a frame-by-frame attribute file or a binary packet, the terminal device reads the Gaussian deformation attribute label map corresponding to the target video frame and interprets the pixel values ​​in the label map as deformation attribute labels, rather than as display pixels. If the dynamic deformation stream uses a video-based attribute stream, the terminal device decodes the video-based attribute stream to obtain the original attribute label data of the corresponding video frame, and performs inverse quantization based on the resource metadata to recover the deformation data under the corresponding video frame.

[0196] The deformation data includes the position, scale, and / or rotation changes of dynamic Gaussian elements in the corresponding video frame. If the dynamic deformation stream uses a differential representation, the deformation data represents the attribute changes of the dynamic Gaussian elements relative to the dynamic basis. The terminal device synthesizes the deformation data with the reference attributes in the dynamic basis to recover the dynamic Gaussian attributes of the target video frame.

[0197] Furthermore, in response to the target video frame pointed to by the player's timeline during dynamic playback, a Gaussian representation of the target video frame is generated based on the deformation data corresponding to the target video frame, using both the static and dynamic bases, including: Create Gaussian rendering objects from the static Gaussian primitives recovered from the static base and the dynamic Gaussian primitives recovered from the dynamic base; In response to the target video frame currently pointed to by the player's timeline, obtain the deformation data corresponding to the target video frame from the deformation data corresponding to each video frame; Based on the attribute changes of the dynamic Gaussian primitives carried by the deformation data under the target video frame, the attributes of the corresponding regions of the dynamic Gaussian primitives in the Gaussian rendering object are written back to obtain the rendering data of the target video frame in the browser memory. The rendering data is the Gaussian representation of the target video frame and is used to draw the interactive display screen of the target video frame. The data corresponding to the static Gaussian primitives remain unchanged when the target video frame changes.

[0198] The terminal device merges the static Gaussian primitives recovered from the static base and the dynamic Gaussian primitives recovered from the dynamic base into a unified Gaussian rendering object. After merging, the static Gaussian primitives are located at the beginning of the data in the Gaussian rendering object, and the dynamic Gaussian primitives are located at the end of the data in the Gaussian rendering object. The terminal device records the dynamic starting index and the dynamic quantity of the dynamic Gaussian primitives in the Gaussian rendering object. The dynamic starting index indicates the starting position of the dynamic Gaussian primitive in the unified Gaussian rendering object, and the dynamic quantity indicates the number of dynamic Gaussian primitives that need to be updated frame by frame. Therefore, during subsequent dynamic playback, the terminal device does not need to reconstruct the complete Gaussian rendering object, but only performs partial updates on the data intervals corresponding to the dynamic Gaussian primitives.

[0199] When the player's timeline points to the target video frame, the terminal device retrieves the deformation data corresponding to the target video frame from the deformation data of each video frame based on the frame number or timestamp of the target video frame. For continuous playback, the terminal device retrieves the deformation data of the current video frame in order of frame rate; for paused, timeline dragged, or skipped playback, the terminal device directly locates the corresponding deformation attribute label map or the corresponding frame data in the video attribute stream based on the target video frame index.

[0200] After obtaining the deformation data corresponding to the target video frame, the terminal device performs attribute write-back on the data range corresponding to the dynamic Gaussian primitives in the Gaussian rendering object based on the changes in the dynamic Gaussian primitive attributes carried by the deformation data. Attribute write-back includes: updating the center position of the target video frame based on the reference center position of the dynamic Gaussian primitives and the position change of the target video frame; updating the scale of the target video frame based on the reference scale of the dynamic Gaussian primitives and the scale change of the target video frame; and updating the rotation of the target video frame based on the reference rotation of the dynamic Gaussian primitives and the rotation change of the target video frame. The data corresponding to the static Gaussian primitives remains unchanged when the target video frame changes.

[0201] Attribute write-back writes the Gaussian attribute array, texture data, graphics buffer, or other Gaussian rendering data structures to the terminal device. For graphics processor-based rendering implementations, after completing the dynamic Gaussian primitive attribute write-back, the terminal device performs a local graphics processor update on the data range corresponding to the dynamic Gaussian primitive, instead of uploading the complete Gaussian rendering object. For transparent Gaussian rendering, after the center position of the dynamic Gaussian primitive changes, the terminal device synchronously updates the sorting center, bounding box, or related rendering auxiliary data to keep the transparent blending order, visibility range, and the dynamic state of the current target video frame consistent.

[0202] During dynamic playback, target video frame requests and deformation data decoding occur asynchronously. To avoid multiple target video frame requests being concurrently written to the same Gaussian rendering object, the terminal device sets a frame application status flag. When the current video frame is being decoded or its attributes are being written back, if the player's timeline generates a new target video frame request based on the currently pointed-to target video frame, the terminal device records the latest target video frame and applies it after the current video frame application is completed. This method is used to merge intermediate frame requests generated by rapidly dragging the timeline, avoiding repeated decoding and uploading of outdated frames.

[0203] In one exemplary embodiment, the terminal device caches recently decoded deformation data. The cached objects include the deformation data corresponding to the current video frame and several adjacent video frames. During continuous playback, short-distance playback, or small-range dragging of the timeline, the terminal device reads the decoded deformation data from the cache; when the target video frame is not found in the cache, the corresponding frame data is located and decoded from the dynamic deformation stream. This reduces repeated decoding and repeated dequantization, improving the response speed of dynamic playback and timeline jumps.

[0204] It should be understood that, in the embodiments of this application, when loading interactive video resources for the first time, the terminal device first loads the static and dynamic bases, and displays the Gaussian rendered object after the deformation data of the initial video frame is successfully decoded and the attributes are written back. This avoids the user seeing an intermediate state where the dynamic deformation data has not yet been applied. When the dynamic deformation stream uses a video-based attribute stream and the initial decoding takes a long time, the terminal device displays a loading prompt before the initial frame application is completed, and enters playback mode after the Gaussian representation of the target video frame is generated.

[0205] In one exemplary embodiment, the step of obtaining the interactive display screen under the target video frame by real-time rendering of the Gaussian representation of the target video frame includes: The terminal device renders the Gaussian representation of the target video frame in real time and performs real-time interactive rendering based on the Gaussian representation of the target video frame. An interactive display screen that supports free viewing angle is generated under the target video frame. The interactive display screen responds to the user's dynamic playback, pause or timeline jump triggered by the target video frame and displays the four-dimensional dynamic scene by rotating, scaling, panning or changing the virtual camera perspective.

[0206] After generating the Gaussian representation of the target video frame, the terminal device renders the Gaussian representation of the target video frame in real time based on the current virtual camera parameters, obtaining an interactive display screen under the target video frame. Virtual camera parameters include viewing position, viewing direction, field of view, and projection matrix or view matrix. Users can change the virtual camera parameters or the target video frame by rotating, scaling, panning, dragging the timeline, pausing playback, or changing the viewing angle, thereby freely observing the four-dimensional dynamic scene under the target video frame. Attribute changes under a viewing angle are as follows: Figure 7 As shown, the corresponding attributes change as the viewing angle changes, that is, as... Figure 8 The attribute changes shown.

[0207] Through the above-described terminal playback embodiment, the interactive video resources generated in this application embodiment are dynamically played in the terminal device using a method of "static basis persistence, dynamic basis reuse, and target frame deformation data update". The static Gaussian primitives remain unchanged after recovery during the loading phase, while the dynamic Gaussian primitives undergo local attribute updates based on the deformation data corresponding to the target video frame during playback. This method differs from directly playing ordinary two-dimensional video frame sequences and from reconstructing a complete Gaussian scene for each video frame, thus reducing the overhead of repeated decoding, repeated memory writing, and repeated graphics resource reconstruction in the terminal device.

[0208] Therefore, under the action of the interactive video resources generated in the embodiments of this application, the terminal device can recover the Gaussian representation of the four-dimensional dynamic scene in the target video frame based on the compressed static basis, dynamic basis and dynamic deformation flow, and generate an interactive display screen that supports free viewing angle in real time, thereby providing end-to-end dynamic playback implementation for the four-dimensional dynamic scene.

[0209] In one exemplary embodiment, this application also provides a computer device including a memory, a processor, and a computer program stored in the memory, the processor executing the computer program to implement the steps of the method as described above.

[0210] In one exemplary embodiment, this application also provides a computer program product including a computer program that, when executed by a processor, implements the steps of the method as described above.

[0211] In one exemplary embodiment, this application also provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the method as described above.

[0212] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this application.

[0213] In an exemplary embodiment of this application, a computer-readable storage medium is also provided, on which computer-readable instructions are stored, which, when executed by a computer's processor, cause the computer to perform the methods described in the above method embodiments.

[0214] According to one embodiment of this application, a program product for implementing the methods in the above-described method embodiments is also provided. This product may employ a portable compact disc read-only memory (CD-ROM) and include program code, and can run on a terminal device, such as a personal computer. However, the program product of this invention is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0215] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0216] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0217] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, or any suitable combination thereof.

[0218] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0219] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0220] Furthermore, although the steps of the method in this application are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0221] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, and includes several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the method according to the embodiments of this application.

[0222] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the appended claims.

Claims

1. A method for generating interactive video resources for four-dimensional dynamic scene modeling, characterized in that, The method includes: For multiple videos of a four-dimensional dynamic scene, scene initialization training of the four-dimensional dynamic scene is performed using multi-view video frames and camera parameters from multiple videos to obtain an initial Gaussian representation of the complete scene. Based on the initial Gaussian representation, camera parameters, and region masks corresponding to multi-view video frames and camera viewpoints, the Gaussian primitives in the initial Gaussian representation are trained for static / dynamic separation to obtain a set of static Gaussian primitives and a set of dynamic Gaussian primitives. The region masks are used to distinguish dynamic subject regions and static background regions and serve as supervision information for the static / dynamic separation training. The static / dynamic separation training steps include: setting dynamic response values ​​for each Gaussian primitive in the initial Gaussian representation, where the dynamic response values ​​are time-independent Gaussian primitive-level parameters; rendering the dynamic response values ​​of each Gaussian primitive to the corresponding video frames and camera viewpoints based on the camera parameters to obtain a dynamic prediction mask; using the region masks corresponding to the video frames and camera viewpoints for supervision, and optimizing the dynamic response values ​​of each Gaussian primitive through the difference between the dynamic prediction mask and the corresponding region mask; and dividing the Gaussian primitives in the initial Gaussian representation into a set of static Gaussian primitives and a set of dynamic Gaussian primitives based on the optimized dynamic response values. The dynamic Gaussian set is trained with a deformation field to obtain the deformation field shared by the dynamic Gaussian set, and the dynamic deformation flow of the dynamic Gaussian set corresponding to each video frame is generated through the deformation field. Separate quantization compression is performed on the static basis obtained by transforming the static Gaussian set, the dynamic basis obtained by transforming the dynamic Gaussian set, and the dynamic deformation flow to generate interactive video resources of the four-dimensional dynamic scene.

2. The method according to claim 1, characterized in that, The process involves training the scene initialization of the four-dimensional dynamic scene using multi-view video frames and camera parameters from multiple video streams to obtain an initial Gaussian representation of the complete scene, including: Based on the sparse 3D cloud corresponding to the multi-view video frame, each 3D cloud in the sparse 3D cloud is initialized as a Gaussian element. Based on the camera parameters corresponding to each camera viewpoint, differentiable rendering of Gaussian elements is performed to obtain the predicted image under the corresponding camera viewpoint. Based on the image difference between the predicted image and the actual video frame corresponding to the camera viewpoint, the properties of the Gaussian unit are optimized to obtain an initial Gaussian representation representing the complete scene. The actual video frame is the video frame corresponding to the camera viewpoint among the multi-view video frames.

3. The method according to claim 1, characterized in that, Before performing static and dynamic separation training on the Gaussian primitives in the initial Gaussian representation based on the initial Gaussian representation, camera parameters, and region masks corresponding to multi-view video frames and camera viewpoints to obtain a set of static and dynamic Gaussian primitives, the method further includes: The four-dimensional dynamic scene is acquired and preprocessed to obtain the camera parameters of each channel and the region mask corresponding to each video frame. The region mask corresponds to the video frame and the camera viewpoint.

4. The method according to claim 3, characterized in that, The acquisition and preprocessing of multiple video streams of the four-dimensional dynamic scene yields camera parameters for each stream and a region mask corresponding to each frame of the video stream. The region mask corresponds to the video frame and the camera viewpoint, including: For synchronized and distortion-free multi-view video frames, candidate dynamic regions are obtained by performing region mask candidate segmentation based on the dynamic region cue words configured for the dynamic subject. Based on the candidate dynamic region and adjacent video frames, a frame-by-frame temporal mask is obtained by performing propagation, tracking and keying of the candidate dynamic region in the video. Morphological post-processing is performed on the frame-by-frame temporal mask to obtain a region mask. The region mask is then stored using the camera number and the frame number of the corresponding video frame as indexes, so that the stored region mask corresponds one-to-one with the video frame through the camera number and the frame number.

5. The method according to claim 1, characterized in that, The step of training the dynamic Gaussian set with a deformation field to obtain a shared deformation field for the dynamic Gaussian set, and generating a dynamic deformation stream for each video frame corresponding to the dynamic Gaussian set using the deformation field, includes: The spatial location of the dynamic Gaussian element in the set of dynamic Gaussian elements and the timestamp of the corresponding video frame are input into the deformation field to obtain the attribute changes of the dynamic Gaussian elements under the video frame. The attribute changes include spatial location changes, scale changes and / or rotation changes. Based on the attribute changes of the dynamic Gaussian primitives in the video frame, the Gaussian attributes of the dynamic Gaussian primitives in the video frame are updated to obtain the dynamic Gaussian representation of the corresponding video frame. The dynamic Gaussian representation of the corresponding video frame is combined with the static Gaussian set to obtain the current Gaussian scene of the corresponding video frame. Based on the corresponding camera parameters, the current Gaussian scene is rendered in a differentiable manner to obtain the predicted image from the corresponding camera view. The deformation field is optimized based on the image differences between the predicted image and the multi-view video frames from the corresponding camera perspective.

6. The method according to claim 1, characterized in that, Before training the dynamic Gaussian set with a deformation field to obtain the shared deformation field of the dynamic Gaussian set, and generating the dynamic deformation stream of the dynamic Gaussian set corresponding to each video frame through the deformation field, the method further includes: An initial deformation field is constructed and shared by the dynamic Gaussian element set, and the initial deformation field is initialized to output zero deformation or near-zero deformation.

7. The method according to claim 1, characterized in that, The step of performing separate quantization compression on the static basis obtained by transforming the static Gaussian set, the dynamic basis obtained by transforming the dynamic Gaussian set, and the dynamic deformation flow to generate the interactive video resources of the four-dimensional dynamic scene includes: The static and dynamic bases are quantized and encoded to map the quantized Gaussian attributes into a two-dimensional attribute graph. The attribute changes in the dynamic deformation stream are quantized and encoded frame by frame to obtain the Gaussian deformation attribute label map corresponding to the video frame. The two-dimensional attribute map and the Gaussian deformation attribute label map corresponding to each video frame are packaged to generate the interactive video resource of the four-dimensional dynamic scene. In this context, the dynamic Gaussian elements in the dynamic substrate maintain an index correspondence with the deformation attributes in the Gaussian element deformation attribute label map corresponding to each video frame.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1-7.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-7.

Images