Parallel reconstruction method of four-dimensional dynamic scene, computer device and program product

Abstract

The application provides a parallel reconstruction method of a four-dimensional dynamic scene, a computer device and a program product. Gaussian primitives representing the four-dimensional dynamic scene are distributed to multiple graphic processors to form multiple local Gaussian primitive partitions, and rendering contributions of the graphic processors are aggregated by cross-device differentiable Gaussian rendering, so that the Gaussian primitives stored in the partitions can jointly participate in complete scene reconstruction. Further, static Gaussian partitions and dynamic Gaussian partitions are obtained by copying state reference snapshots and area masks, and a deformation field network in the multiple graphic processors is synchronously trained to generate dynamic deformation flows corresponding to each video frame of the dynamic Gaussian partitions, and then interactive video resources for reconstructing the four-dimensional dynamic scene are generated.

Description

Technical Field

[0001] This application relates to the field of interactive graphics immersive technology, specifically to a parallel reconstruction method, computer device, and program product for a four-dimensional dynamic scene. Background Technology

[0002] With the development of virtual reality, augmented reality, free-viewpoint viewing, and interactive content display technologies, the demand for reconstructing four-dimensional dynamic scenes is constantly increasing. Four-dimensional dynamic scenes not only include three-dimensional spatial structures, but also motion, posture changes, or deformation changes that change over time. Therefore, the reconstruction process needs to process spatial information, appearance information, and time-varying information simultaneously.

[0003] Compared to static scenes or ordinary 2D videos, 4D dynamic scenes have higher data dimensions and more complex spatiotemporal relationships. Reconstructing 4D dynamic scenes typically requires processing large amounts of scene data and maintaining scene representation data that expresses spatial structure and temporal changes. When the scene duration is long, the scene details are complex, or the scene representation data is large, the reconstruction process is easily limited by the video memory capacity and computing power of the computing device, leading to reduced reconstruction efficiency, limited processing scene scale, and even difficulty in completing high-quality reconstruction with limited hardware resources.

[0004] While using multiple computing devices to increase processing power can be considered, four-dimensional dynamic scene reconstruction is not a simple data splitting task. During the reconstruction process, it is necessary to ensure the overall consistency of the scene representation in space and time, and to avoid inconsistencies in data states between different computing devices. At the same time, the handling of temporal changes is also easily affected by factors such as the instability of the initial state, which can lead to discontinuous dynamic changes or unstable reconstruction results.

[0005] Therefore, how to achieve stable reconstruction of four-dimensional dynamic scenes under limited computing resources has become a technical problem that needs to be solved. Summary of the Invention

[0006] This application provides a parallel reconstruction method, computer device, and program product for four-dimensional dynamic scenes to solve the technical problem of how to achieve stable reconstruction of four-dimensional dynamic scenes under limited computing resources.

[0007] According to one aspect of the embodiments of this application, a parallel reconstruction method for a four-dimensional dynamic scene is disclosed, the method comprising:

[0008] The Gaussian primitives representing a four-dimensional dynamic scene are trained and scheduled, and the Gaussian primitives are distributed to multiple graphics processors to form local Gaussian primitive fragments in each graphics processor. Based on the multi-view video frames of the four-dimensional dynamic scene, cross-device differentiable Gaussian rendering is used to enable each graphics processor to calculate the rendering contribution of the local Gaussian montage, and the rendering contributions of multiple graphics processors are aggregated across devices. Based on the aggregated prediction image and its backpropagation gradient, the Gaussian attributes of the local Gaussian montage in each graphics processor are updated. By replicating the reference snapshot and using the region mask corresponding to the frame number and camera view of the multi-view video frame, Gaussian elements in the local Gaussian element slice are divided to obtain static Gaussian slices and dynamic Gaussian slices. The same deformation field network is run in the multiple graphics processors, and the deformation field network is synchronously trained by using a replicated reference snapshot and dynamic Gaussian slices held by each graphics processor to obtain a deformation field network for reconstructing the dynamic changes in the four-dimensional dynamic scene. The deformation field network generates dynamic deformation flows for each video frame corresponding to dynamic Gaussian slices in each graphics processor. Interactive video resources for reconstructing the four-dimensional dynamic scene are generated by using static Gaussian slices, dynamic Gaussian slices, and the dynamic deformation flow of the dynamic Gaussian slices corresponding to each video frame on each graphics processor.

[0009] According to one aspect of the embodiments of this application, a computer device is disclosed, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method as described in any of the preceding claims.

[0010] According to one aspect of the embodiments of this application, a computer program product is disclosed, including a computer program that, when executed by a processor, implements the steps of the method as described in any of the preceding claims.

[0011] This application addresses the problem of limited computing resources hindering the reconstruction of four-dimensional dynamic scenes in related technologies. It distributes Gaussian primitives representing the four-dimensional dynamic scene to multiple graphics processors (GPUs) to form multiple local Gaussian primitive fragments. Cross-device differentiable Gaussian rendering aggregates the rendering contributions of each GPU, enabling the fragmented Gaussian primitives to collectively participate in the complete scene reconstruction. Furthermore, it obtains static and dynamic Gaussian fragments through replicated reference snapshots and region masks, and synchronously trains the deformation field network in multiple GPUs to generate dynamic deformation streams corresponding to each video frame for the dynamic Gaussian fragments. This generates interactive video resources for reconstructing the four-dimensional dynamic scene. Therefore, this application achieves synergy between Gaussian primitive fragmentation, cross-device rendering aggregation, static and dynamic fragmentation, synchronous training of the deformation field network, and dynamic deformation stream generation, enabling stable reconstruction of four-dimensional dynamic scenes with limited computing resources. The core of this application's embodiments lies in not simply assigning tasks to multiple graphics processors, but establishing a collaborative relationship between Gaussian meta-slicing, cross-device rendering aggregation, dynamic and static Gaussian slice organization, deformation field network synchronous training, and dynamic deformation flow resource generation, thereby enabling the stable reconstruction of four-dimensional dynamic scenes under limited computing resources.

[0012] Specifically, by training and scheduling Gaussian primitives and forming local Gaussian primitive fragments in each graphics processor, Gaussian primitive parameters can be distributed across multiple graphics processors for holding and maintenance, reducing the parameter scale and memory pressure required by a single graphics processor, thereby increasing the scale of four-dimensional dynamic scenes that can be processed under limited computing resources.

[0013] By using cross-device differentiable Gaussian rendering, each graphics processor can calculate its rendering contribution for local Gaussian primitive slices and aggregate the rendering contributions of multiple graphics processors across devices. This allows for the acquisition of a complete scene-corresponding predicted image even with Gaussian primitive slice storage, avoiding the fragmentation of scene representation caused by simple slice processing, thereby improving the overall consistency in the parallel reconstruction process.

[0014] By replicating the reference snapshot and using the region mask corresponding to the frame number and camera viewpoint, Gaussian elements in the local Gaussian element slice are divided to obtain static Gaussian slices and dynamic Gaussian slices. This enables the subsequent deformation field network to perform time-varying modeling around the dynamic Gaussian slices, reducing unnecessary dynamic modeling objects and improving the object stability during dynamic deformation flow generation.

[0015] By running the same deformation field network on multiple graphics processors and training the deformation field network synchronously using a replicated reference snapshot and dynamic Gaussian slices held by each graphics processor, training stability and network consistency among multiple graphics processors can be balanced during the deformation field network training process, thereby reducing the risk of unstable dynamic expression, inter-frame drift, or discontinuous reconstruction results.

[0016] By generating dynamic deformation streams for each video frame through dynamic Gaussian slices in each graphics processor via a deformation field network, and generating interactive video resources through static Gaussian slices, dynamic Gaussian slices, and dynamic deformation streams, the reconstruction results of four-dimensional dynamic scenes can be organized in a way that combines static content with frame-by-frame dynamic changes, thereby improving the efficiency of subsequent loading, frame-by-frame restoration, and interactive display.

[0017] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0018] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this application. Attached Figure Description

[0019] The above and other objectives, features and advantages of this application will become more apparent from a detailed description of exemplary embodiments thereof with reference to the accompanying drawings.

[0020] Figure 1 This is a flowchart illustrating a parallel reconstruction method for a four-dimensional dynamic scene according to an exemplary embodiment.

[0021] Figure 2 It is based on Figure 1 The flowchart described in the corresponding embodiment describes the steps of training and scheduling Gaussian primitives representing a four-dimensional dynamic scene, distributing Gaussian primitives to multiple graphics processors, and forming local Gaussian primitive slices in each graphics processor.

[0022] Figure 3 It is based on Figure 1 The flowchart shown in the corresponding embodiment describes the steps of processing multi-view video frames based on a four-dimensional dynamic scene, using cross-device differentiable Gaussian rendering, enabling each graphics processor to calculate the rendering contribution of its local Gaussian montage, aggregating the rendering contributions of multiple graphics processors across devices, and updating the Gaussian montage attributes of the local Gaussian montages in each graphics processor based on the aggregated predicted image and its backpropagation gradient.

[0023] Figure 4 It is based on Figure 1The corresponding embodiment shows a flowchart describing the steps of dividing Gaussian elements in a local Gaussian element slice by using a replicated reference snapshot and a region mask corresponding to the frame number and camera viewpoint of the multi-view video frame, to obtain static Gaussian slices and dynamic Gaussian slices.

[0024] Figure 5 It is based on Figure 1 A flowchart describing the pre-training process of step S140 in the corresponding embodiment.

[0025] Figure 6 It is based on Figure 1 A flowchart describing the joint optimization process of step S140 in the corresponding embodiment.

[0026] Figure 7 This is a schematic diagram of the interactive interface corresponding to a specific target video frame. Detailed Implementation

[0027] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided to make the description of this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art. The drawings are merely illustrative of this application and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0028] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more exemplary embodiments. Numerous specific details are provided in the following description to give a full understanding of exemplary embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced with one or more of the specific details omitted, or other methods, components, steps, etc., can be employed. In other instances, well-known structures, methods, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0029] Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0030] See Figure 1 , Figure 1 This is a flowchart illustrating a parallel reconstruction method for a four-dimensional dynamic scene according to an exemplary embodiment.

[0031] The parallel reconstruction method for four-dimensional dynamic scenes provided in this application includes: Step S110: Train and schedule the Gaussian primitives representing the four-dimensional dynamic scene, distribute the Gaussian primitives to multiple graphics processors, and form local Gaussian primitive fragments in each graphics processor. Step S120: Based on the multi-view video frames of the four-dimensional dynamic scene, cross-device differentiable Gaussian rendering is used to enable each graphics processor to calculate the rendering contribution of the local Gaussian montage, and the rendering contributions of multiple graphics processors are aggregated across devices. Based on the aggregated predicted image and its backpropagation gradient, the Gaussian montage attributes of the local Gaussian montage in each graphics processor are updated. Step S130: By replicating the reference snapshot and using the region mask corresponding to the frame number and camera view of the multi-view video frame, the Gaussian elements in the local Gaussian element slice are divided to obtain static Gaussian slices and dynamic Gaussian slices. Step S140: Run the same deformation field network in multiple graphics processors, and train the deformation field network synchronously by using a replicated reference snapshot and dynamic Gaussian slices held by each graphics processor to obtain a deformation field network for reconstructing the dynamic changes in the four-dimensional dynamic scene. Step S150: Generate dynamic deformation flow corresponding to each video frame in each graphics processor through the deformation field network; Step S160: Using the static Gaussian slices, dynamic Gaussian slices, and dynamic deformation streams corresponding to each video frame on each graphics processor, interactive video resources for reconstructing the four-dimensional dynamic scene are generated.

[0032] These steps are explained in detail below.

[0033] First, it should be noted that the four-dimensional dynamic scene in this application refers to a dynamic scene that has a three-dimensional spatial structure and undergoes movement, posture changes, deformation changes, or appearance changes over time. For example, four-dimensional dynamic scenes include handicraft production scenes, character operation scenes, product display scenes, teaching demonstration scenes, live broadcast replay scenes, and small dynamic object display scenes, etc.

[0034] Taking a handicraft production scene as an example, the scene includes the craftsman, hands, tools, materials, semi-finished products, finished products, workbench, and surrounding fixed objects. When viewing such scenes, users usually not only want to watch the production process from a fixed shooting perspective, but also want to change the viewing angle at any moment of production to see hand movements, tool contact positions, material deformation processes, or details of the finished product from the side, overhead, oblique, or magnified perspectives.

[0035] For the aforementioned four-dimensional dynamic scenes, recording them only in the form of ordinary video typically only provides a two-dimensional view from a limited shooting perspective, which is insufficient to meet the needs of free-view interactive viewing. If the four-dimensional dynamic scene is directly reconstructed completely, it is necessary to simultaneously process spatial structure, appearance information, and temporal change information, resulting in a large computational and data volume. With limited computing resources, relying solely on a single graphics processor is easily limited by video memory capacity and computing power; simply splitting the task among multiple graphics processors may lead to incomplete scene representation, inconsistent rendering results, difficulty in coordinating parameter states, or unstable temporal relationships. Therefore, the embodiments of this application are not ordinary video generation methods, nor are they simple data parallel processing methods, but rather a collaborative processing flow established for Gaussian primitive representation, cross-device differentiable rendering, static / dynamic separation, deformation field training, and interactive resource generation in the four-dimensional dynamic scene reconstruction process.

[0036] In this embodiment, the reconstruction of the four-dimensional dynamic scene uses multiple video streams of the acquired four-dimensional dynamic scene as input. These multiple video streams include video data captured by multiple cameras from different camera perspectives of the same four-dimensional dynamic scene. By preprocessing the multiple video streams, basic data for subsequent reconstruction processes is obtained.

[0037] Specifically, preprocessing includes multi-channel video synchronization, frame extraction, camera parameter estimation, image distortion correction, sparse point cloud generation, and region mask generation. The sparse point cloud provides the foundation for initializing Gaussian elements representing the four-dimensional dynamic scene in step S110. The synchronized multi-channel videos yield multi-view video frames corresponding to the same timestamp and extracted from different camera perspectives, for subsequent cross-device differentiable Gaussian rendering training. Region mask generation generates region masks corresponding to the frame number and camera perspective of the multi-view video frames, for use in step S130 for Gaussian element partitioning.

[0038] Multi-channel video is captured by multiple cameras from different perspectives of the same four-dimensional dynamic scene. For example, in a handicraft production scene, multiple cameras are arranged around the workbench or operation area to cover the main range of motion of hands, tools, materials, semi-finished products, and finished products. Multiple cameras include front cameras, side cameras, obliquely above cameras, top-down cameras, and cameras for detailed shots.

[0039] During preprocessing, synchronization of multiple video streams includes temporal synchronization. In an exemplary embodiment, temporal synchronization between multiple video streams is achieved through audio peaks and visible synchronization actions, enabling video frames captured by different cameras to be mapped to a unified reference timeline. Camera parameter estimation is obtained through feature extraction, feature matching, camera pose estimation, and joint optimization, where camera parameters include intrinsic and extrinsic parameters.

[0040] Region masks corresponding to the frame number of a multi-view video frame and the camera's viewpoint are used to indicate dynamic or non-dynamic regions within the corresponding video frame and camera viewpoint. It's important to note that region masks do not need to subdivide the specific semantic categories within dynamic regions. For example, in a handicraft production scenario, the region mask treats the entire area of ​​a hand, tool, material, or semi-finished product that is moving, displacing, or deforming as a dynamic region, without requiring separate distinctions between semantic categories such as "hand," "tool," and "material." For situations where the hand overlaps with a tool, the tool contacts a material, or the hand occludes a semi-finished product, the relevant regions are merged into dynamic regions. Thus, the region mask primarily provides two-dimensional supervision for subsequent Gaussian unit dynamic-static segmentation, rather than providing semantic classification results.

[0041] Therefore, in the execution of step S110, Gaussian elements representing the four-dimensional dynamic scene can be obtained, and then training scheduling can be implemented for the Gaussian elements of the four-dimensional dynamic scene.

[0042] In one exemplary embodiment, a Gaussian primitive set for representing a four-dimensional dynamic scene is initialized based on preprocessed sparse point clouds, multi-view video frames, and camera parameters. Specifically, three-dimensional points in the sparse point cloud are initialized as the center positions of the Gaussian primitives, and the color, scale, rotation, and opacity of the Gaussian primitives are initialized according to the color of the three-dimensional points, neighborhood distance, or preset parameters. This initialization process is only an example; in other implementations, random initialization, dense point cloud initialization, voxel initialization, or other methods capable of generating a Gaussian primitive set can also be used.

[0043] Each Gaussian primitive (GRP) has GRP attributes including center position, scale, rotation, opacity, and color or spherical harmonic coefficients, which describe its spatial distribution, appearance information, and visibility relationships in a four-dimensional dynamic scene. In other words, for the four-dimensional dynamic scene to be reconstructed, the GRP is the basic unit of representation used to characterize the spatial structure, appearance information, and visibility relationships of the four-dimensional dynamic scene. Multiple GRPs together constitute the scene representation of the four-dimensional dynamic scene.

[0044] In step S110, the Gaussian primitives representing the four-dimensional dynamic scene are trained and scheduled according to the number of graphics processors, so that each Gaussian primitive is distributed to a graphics processor. In other words, the training scheduling in this embodiment is not simply allocating multi-view video frames to different graphics processors, but rather distributing the Gaussian primitive parameters used to represent the four-dimensional dynamic scene to multiple graphics processors, so that different graphics processors hold and update different parts of the Gaussian primitives respectively.

[0045] In a specific implementation of an exemplary embodiment, let the number of graphics processors be... , No. Each graphics processor holds a local Gaussian fragment of 1. Then the complete Gaussian set is represented as:

[0046] In this embodiment, different local Gaussian fragments do not overlap. To process large-scale Gaussian sets with limited computing resources, this application does not require each graphics processor to completely copy all Gaussian elements; instead, it performs training scheduling on the Gaussian elements. A global Gaussian index is assigned to each Gaussian element, and the corresponding target graphics processor is determined based on the global Gaussian index and the number of graphics processors. Subsequently, the Gaussian attributes of each Gaussian element are written to the local parameter storage area of ​​the corresponding target graphics processor, so that each graphics processor holds a portion of the Gaussian set, thereby forming local Gaussian fragments in each graphics processor.

[0047] In a specific implementation of an exemplary embodiment, the following scheduling function is used to determine the first... The target graphics processor corresponding to each Gaussian element:

[0048] in, This represents the global Gaussian element index of the Gaussian element. Indicates the number of graphics processors. Indicates the target graphics processor number. Represents the scheduling function. Scheduling function It is a modulo function, a spatial partitioning function, a load balancing function, or other mapping function. In a specific implementation of another exemplary embodiment, according to The target graphics processor can be determined by modulo operations; in addition, the target graphics processor can also be determined based on the spatial block where the center of the Gaussian element is located, which will not be listed here.

[0049] Gaussian elements are distributed to the graphics processor (GPU) to form local Gaussian fragments on that GPU. For each local Gaussian fragment in each GPU, the data is in the form of an array of Gaussian attributes; that is, each local Gaussian fragment exists at least in the form of a Gaussian attribute array held by the corresponding GPU.

[0050] Local Gaussian fragments in different graphics processors can have different numbers of Gaussian elements. To facilitate subsequent cross-device rendering contribution aggregation, gradient backpropagation, static / dynamic separation, and resource export, in an exemplary embodiment, at least one of the following will be recorded for each local Gaussian fragment: global index range, fragment offset, and mapping relationship between local index and global Gaussian index.

[0051] After forming local Gaussian fragments in each graphics processor (GPU), a local training state is established for each Gaussian fragment. Each local training state corresponds one-to-one with a Gaussian fragment within that local Gaussian fragment. The local training state includes at least one of the following associated Gaussian fragments: optimizer state, learning rate state, visibility statistics, density statistics, pruning markers, projection radius statistics, and gradient statistics. The local training state is stored in the GPU that holds the corresponding local Gaussian fragment. Therefore, during training, each GPU not only holds the Gaussian attributes of its local Gaussian fragment but also maintains the corresponding training state, thus avoiding the need to fully copy all Gaussian attributes and training states at each training step.

[0052] Through the aforementioned training scheduling, this embodiment of the application establishes a multi-GPU fragment holding structure at the Gaussian meta-parameter level, enabling each GPU to undertake the maintenance tasks of a portion of the Gaussian meta-attributes and local training states. This approach provides a data organization foundation for subsequent cross-device differentiable Gaussian rendering, including rendering contribution aggregation, backpropagation gradient backpropagation, static / dynamic separation, and deformation field network training.

[0053] See Figure 2 , Figure 2 It is based on Figure 1 The flowchart described in the corresponding embodiment describes the steps of training and scheduling Gaussian primitives representing a four-dimensional dynamic scene, distributing Gaussian primitives to multiple graphics processors, and forming local Gaussian primitive slices in each graphics processor.

[0054] The step S110 of this application embodiment, which involves training and scheduling Gaussian primitives representing a four-dimensional dynamic scene, distributing Gaussian primitives to multiple graphics processors, and forming local Gaussian primitive fragments in each graphics processor, includes: Step S111: Obtain Gaussian elements obtained from the initialization of multi-view video frames. The Gaussian elements are used to characterize the spatial distribution of the four-dimensional dynamic scene. Step S112: Assign a global Gaussian element number to the Gaussian element; Step S113: Calculate the target graphics processor corresponding to each Gaussian element based on the global Gaussian element number and the number of graphics processors. Step S114: Write the Gaussian attributes of each Gaussian to the local parameter storage area of ​​the corresponding target graphics processor to form local Gaussian fragments in each graphics processor.

[0055] These steps are explained in detail below.

[0056] As mentioned earlier, a Gaussian set is obtained based on the preprocessed multi-view video frames, camera parameters, and sparse point cloud initialization.

[0057] The Gaussian set generated by the aforementioned preprocessing and initialization process is directly obtained. This Gaussian set serves as the object for subsequent training scheduling, used to distribute the Gaussian elements to multiple graphics processors. In a specific implementation of an exemplary embodiment, to facilitate subsequent distribution, gradient backpropagation, and resource export, the Gaussian elements in the Gaussian set are organized into a list of Gaussian elements to be scheduled.

[0058] Assign a global Gaussian element number to each Gaussian element in the list of Gaussian elements to be scheduled. The global Gaussian element number is used to uniquely identify the corresponding Gaussian element within the complete set of Gaussian elements and to maintain the correspondence in subsequent processes such as local fragmentation, cross-device rendering contribution calculation, backpropagation gradient backpropagation, replication-state reference snapshot generation, and resource export.

[0059] In one exemplary embodiment, global Gaussian element numbers are assigned sequentially according to their order of arrangement in the Gaussian list to be scheduled. For example, for N Gaussian elements in the list to be scheduled, global Gaussian element numbers from 0 to N-1 are assigned sequentially. In another implementation, Gaussian elements can be sorted first based on their center position, spatial block number, Morton order, Hilbert order, PLAS order, estimated rendering overhead, or load balancing requirements, and then global Gaussian element numbers are assigned according to the sorting results. Thus, the global Gaussian element number serves as both a unique identifier and carries certain spatial order or load balancing information.

[0060] After the global Gaussian byte order is assigned, the target graphics processor corresponding to each Gaussian byte is determined based on the number of graphics processors and the global Gaussian byte order. The target graphics processor refers to the graphics processor that subsequently holds and maintains the attributes of that Gaussian byte.

[0061] The mapping relationship between global Gaussian element numbers and target graphics processors will be recorded in the scheduling table, index table, or fragment metadata so that the corresponding Gaussian element can be determined later based on the global Gaussian element number to determine which graphics processor holds the corresponding Gaussian element.

[0062] After determining the mapping relationship between Gaussian primitives and graphics processors (GPUs), the Gaussian primitive attributes of each Gaussian primitive are written to the local parameter storage area of ​​the mapped GPU, i.e., the target GPU. The local parameter storage area is a parameter storage region managed or accessed by the corresponding target GPU, used to store the Gaussian primitive attributes allocated to that GPU. For example, the local parameter storage area includes one or more parameter arrays in video memory. Different attributes within the Gaussian primitives can be stored separately or structured in units of Gaussian primitives.

[0063] For the target graphics processor is the The Gaussian elements of a graphics processor can have their center position, scale, rotation, opacity, color, or spherical harmonics written into the Gaussian element. The local parameter storage area of ​​each graphics processor. During writing, a local local index is assigned to the Gaussian element, and each attribute is written to the corresponding storage location according to the local local index. Thus, the first... The Gaussian primitives and their attributes stored in the local parameter storage area of ​​each graphics processor form the first Gaussian primitive. Each graphics processor corresponds to a local Gaussian fragment.

[0064] In another exemplary implementation, when writing to the local parameter storage area, a mapping relationship between the global Gaussian primitive number, the local primitive number, and the graphics processor number can be established simultaneously. This mapping relationship can be expressed as: the global Gaussian primitive number corresponds to the graphics processor number and the local primitive number within that graphics processor. Through this mapping relationship, in subsequent cross-device differentiable Gaussian rendering and backpropagation processes, the rendering contribution or gradient corresponding to a certain Gaussian primitive can be accurately located to the graphics processor holding that Gaussian primitive and its local parameter storage location.

[0065] Through steps S111 to S114, the complete Gaussian metadata set is converted into multiple local Gaussian metadata fragments, each held by a different graphics processor. These multiple local Gaussian metadata fragments collectively correspond to the complete Gaussian metadata set and provide a stable data organization foundation for subsequent cross-device differentiable Gaussian rendering, gradient backpropagation, static / dynamic partitioning, and interactive video resource generation.

[0066] In step S120, a multi-view video frame refers to a group of video frames extracted from videos corresponding to multiple camera viewpoints at the same timestamp or frame number. In other words, adapted to camera viewpoints, a group of multi-view video frames includes multiple video frames, each corresponding to a camera viewpoint. For any given video frame, it corresponds to a camera viewpoint, frame number, camera parameters, and actual image content. Subsequent cross-device differentiable Gaussian rendering is performed on one video frame from the multi-view video frame group, or it can be performed separately on multiple video frames within the same multi-view video frame group.

[0067] Since Gaussian primitives have been distributed to multiple graphics processors (GPUs) in step S110, and local Gaussian primitive fragments have been formed in each GPU, a single GPU only holds a portion of the complete Gaussian primitive set. If rendering is performed independently in a single GPU, only the local rendering result corresponding to the local Gaussian primitive fragment can be obtained, which is insufficient to represent a complete four-dimensional dynamic scene. Therefore, this embodiment of the application uses cross-device differentiable Gaussian rendering, enabling the Gaussian primitives held by fragments in multiple GPUs to jointly participate in the generation of the predicted image at the same viewpoint of the same video frame, and enabling the backpropagation gradient to be propagated back to the corresponding local Gaussian primitive fragment.

[0068] For details, please refer to Figure 3 , Figure 3 It is based on Figure 1 The flowchart shown in the corresponding embodiment describes the steps of processing multi-view video frames based on a four-dimensional dynamic scene, using cross-device differentiable Gaussian rendering, enabling each graphics processor to calculate the rendering contribution of its local Gaussian montage, aggregating the rendering contributions of multiple graphics processors across devices, and updating the Gaussian montage attributes of the local Gaussian montages in each graphics processor based on the aggregated predicted image and its backpropagation gradient.

[0069] The multi-view video frames based on a four-dimensional dynamic scene provided in this application embodiment, through cross-device differentiable Gaussian rendering, enable each graphics processor to calculate the rendering contribution of its local Gaussian meta-plots, and aggregate the rendering contributions of multiple graphics processors across devices. The step S120, which updates the Gaussian meta-attributes of the local Gaussian meta-plots in each graphics processor based on the aggregated predicted image and its backpropagation gradient, includes: Step S121: Based on the camera parameters corresponding to the camera viewpoint of the multi-view video frame, project the local Gaussian primitives in each graphics processor onto the image plane under the camera viewpoint. Step S122: In each graphics processor, the rendering contribution of the local Gaussian pixel is calculated by projecting the local Gaussian pixel onto the image plane. Step S123: Aggregate the rendering contributions calculated by each graphics processor across devices to obtain the predicted images corresponding to the multi-view video frames. Step S124: Based on the backpropagation gradient of the predicted image, the gradient corresponding to each local Gaussian piece is backpropagated to the corresponding graphics processor to update the Gaussian attributes of the local Gaussian pieces in the graphics processor.

[0070] In this exemplary embodiment, a multi-view video frame can be understood as a target video frame in a group of multi-view video frames. That is, it should be clear that the training view used to carry out training can be a video frame in a group of multi-view video frames as the target video frame to execute steps S121 to S124 respectively, or one of the video frames can be used as the target video frame as the training view. This is not limited here.

[0071] The target video frame has a corresponding frame number and camera viewpoint. Based on the camera parameters corresponding to this viewpoint, the local Gaussian pixel slices held by each graphics processor are projected onto the same image plane. Camera parameters include intrinsic and extrinsic parameters. The intrinsic parameters describe the imaging relationship from 3D points to the image plane, while the extrinsic parameters describe the pose relationship between the camera viewpoint and the unified world coordinate system.

[0072] In one implementation, for the first... Each graphics processor holds a local Gaussian fragment. Local Gaussian pixel fragments can be created based on the camera parameters corresponding to the camera's viewpoint of the target video frame. The center position, scale, and rotation attributes of each Gaussian primitive are used for projection calculations to obtain at least one of the following: projection position, projection range, depth information, and visibility information of the local Gaussian primitive slice in the image plane. Different graphics processors use the camera parameters corresponding to the same target video frame to project their respective local Gaussian primitive slices, so the projection results of multiple local Gaussian primitive slices are under the same camera viewpoint and the same image coordinate system.

[0073] Through the above processing, the embodiments of this application do not use different graphics processors to process different camera viewpoints, nor do they allow each graphics processor to generate independent local scene images. Instead, they enable multiple graphics processors to calculate the projection contribution of their local Gaussian meta-plots around the same predicted image under the same camera viewpoint, providing a basis for subsequent cross-device aggregation of rendering contributions.

[0074] Each graphics processor calculates the rendering contribution of its local Gaussian pixel piece to the predicted image corresponding to the target video frame, based on the projection position, projection range, depth relationship, and Gaussian pixel attributes of the local Gaussian pixel piece. Rendering contributions include color contribution, transparency contribution, depth contribution, visibility contribution, pixel coverage contribution, or other contributions related to differentiable Gaussian rendering.

[0075] In a specific implementation of an exemplary embodiment, for the camera viewpoint to which the target video frame belongs... , No. Each graphics processor is based on its local Gaussian pixel fragments. Calculate local rendering contribution This local rendering contribution It can be an intermediate rendering result corresponding to pixels in the image plane, or it can be intermediate contribution data that participates in subsequent transparency compositing, depth sorting, or pixel accumulation. Since each graphics processor only calculates the rendering contribution corresponding to its local Gaussian byte fragment, a single graphics processor does not need to hold the complete set of Gaussian bytes to complete the local contribution calculation.

[0076] Each graphics processor can also record rendering statistics for local Gaussian pixel slices when calculating rendering contributions. These statistics include at least one of the following: Gaussian pixel visibility, projection radius, number of pixels involved in rendering, gradient strength, alpha contribution, and depth range. This rendering statistics are used for subsequent local training state updates, densification, pruning, or resource export sorting.

[0077] The local rendering contributions from multiple graphics processors are aggregated across devices to generate a predicted image corresponding to the target video frame. This predicted image is not a local image independently generated by any single graphics processor, but rather a complete scene prediction image jointly contributed by multiple local Gaussian slices from the same camera viewpoint.

[0078] In a specific implementation of an exemplary embodiment, let the number of multiple graphics processors be... , No. One graphics processor is designed for the camera's field of view. The calculated local rendering contribution is The predicted image is then represented as:

[0079] in, This refers to cross-device rendering contribution aggregation operations. Cross-device rendering contribution aggregation operations include at least one of the following: pixel-level contribution accumulation, transparency compositing, depth-related sorting, contribution merging, and communication reduction. Through cross-device aggregation, local Gaussian pixel pieces in different graphics processors can jointly represent a four-dimensional dynamic scene in the same predicted image, thereby avoiding missing predicted images or fragmented scene representation caused by Gaussian pixel piece storage.

[0080] In addition, cross-device aggregation is performed while maintaining differentiability, allowing the training loss corresponding to the predicted image to be backpropagated to each graphics processor through the aggregation operation. Thus, cross-device aggregation is used not only for the generation of the complete image in the forward rendering but also for gradient allocation in the backward training phase.

[0081] The training loss is calculated based on the difference between the predicted image and the real image of the target video frame. The training loss includes at least one of color reconstruction loss, structural similarity loss, mask constraint loss, depth constraint loss, or other differentiable training loss. After backpropagation, the gradient of the predicted image is fed back to each graphics processor through a cross-device rendering contribution aggregation operation, and further applied to the local Gaussian fragments held by the corresponding graphics processor.

[0082] Specifically, for the first Each graphics processor holds a local Gaussian fragment. Only with Gradient backpropagation of the high-order skewness element to the first One graphics processor. Each graphics processor updates the Gaussian metadata attributes in its local Gaussian metadata slices based on the gradient. Gaussian metadata attributes include at least one of center position, scale, rotation, opacity, color, or spherical harmonics. Thus, the updating of Gaussian metadata attributes follows the local parameter holding relationships formed in step S110, eliminating the need to copy the complete Gaussian metadata set to all graphics processors during each backpropagation, and also eliminating the need for global synchronization of all Gaussian metadata attributes.

[0083] In one embodiment, the first Local Gaussian fragmentation in a graphics processor The parameters are denoted as Then, based on the training loss corresponding to the predicted image... You can get the same as The corresponding gradient:

[0084] Among them, each Each graphics processor updates its local parameters based on this gradient. In this embodiment, different graphics processors (GPUs) update their respective local Gaussian meta-parameters, thereby achieving fragmented holding and updating of Gaussian meta-parameters. This method differs from ordinary data parallel training, which typically requires each GPU to hold complete model parameters and synchronize model gradients. In this embodiment, however, the Gaussian meta-parameters are held by different GPUs, the predicted image is obtained through rendering contribution aggregation, and the backpropagation gradient is propagated back to the corresponding GPU according to the local Gaussian meta-parameter fragments.

[0085] When updating the Gaussian attributes of local Gaussian fragments, the corresponding local training state can also be updated. For example, the optimizer state is updated based on the backpropagation gradient, and at least one of visibility statistics, densification statistics, and pruning tags is updated based on the rendering statistics of local Gaussian fragments in cross-device differentiable Gaussian rendering. Furthermore, when some Gaussian fragments in a local Gaussian fragment meet the densification condition, the Gaussian fragments can be multiplied or split in the corresponding graphics processor; when some Gaussian fragments meet the pruning condition, the Gaussian fragments can be deleted or marked as invalid in the corresponding graphics processor. Thus, local Gaussian fragments and their training states can be continuously maintained in the corresponding graphics processor.

[0086] Through steps S121 to S124, this embodiment of the application, based on Gaussian primitive fragment storage, enables multiple graphics processors to jointly complete the differentiable rendering of the complete scene from the same video frame's corresponding viewpoint, and ensures that the backpropagation gradient is returned to the corresponding graphics processor according to the parameter ownership of the local Gaussian primitive fragment. This process reduces the parameter and memory pressure on individual graphics processors while maintaining the rendering consistency of the complete four-dimensional dynamic scene from the training viewpoint, providing a stable scene representation foundation for subsequent motion-static separation, deformation field training, and interactive video resource generation.

[0087] In step S130, it should first be noted that the replicated reference snapshot is a complete reference Gaussian representation obtained by aggregating the local Gaussian fragments currently held by multiple graphics processors. Specifically, after steps S110 and S120, each graphics processor holds and updates its corresponding local Gaussian fragment. In order to make a unified judgment based on the complete scene during the static / dynamic scene segmentation stage, the local Gaussian fragments in multiple graphics processors are aggregated to obtain a reference Gaussian representation containing the complete Gaussian set, and this reference Gaussian representation is provided to multiple graphics processors as the replicated reference snapshot.

[0088] The replicated reference snapshot serves as a stable reference copy used in the static / dynamic partitioning stage. Its purpose is twofold: firstly, it enables static / dynamic partitioning to be performed based on the complete Gaussian representation of the scene, rather than solely on local Gaussian fragments within a single graphics processor; secondly, optimizations in the static / dynamic partitioning stage primarily involve determining the dynamic response values ​​of static / dynamic attributes, without simultaneously updating the geometric and appearance attributes of Gaussian fragments, thus reducing the risk of instability in static / dynamic judgments due to continuous changes in the Gaussian fragments themselves. The replicated reference snapshot does not alter the local fragment holding relationships of each Gaussian fragment in subsequent parallel training. After static / dynamic partitioning is completed, it is still necessary to map the partitioning results back to the local Gaussian fragments held by each graphics processor based on the global Gaussian fragment index, fragment offset, or local local index of the Gaussian fragments.

[0089] A region mask is dynamic region supervision information corresponding to the frame number of a video frame and the camera's viewpoint. For the same frame number, the projection position of dynamic objects differs under different camera viewpoints; therefore, the region mask needs to correspond to both the frame number and the camera viewpoint. For example, the [frame number missing in original text]... Frame number, the The area mask corresponding to each camera viewpoint can be denoted as... The region mask is used to indicate the dynamic and non-dynamic regions in the video frame image. The dynamic region may include the area of ​​a hand, tool, material being manipulated, semi-finished product, or other object that is moving, displacing, changing its posture, or deforming.

[0090] In one exemplary embodiment, region masks are generated frame-by-frame during the preprocessing stage. Specifically, cue words are first configured for dynamic objects in the four-dimensional dynamic scene, including people, hands, tools, manipulated objects, materials, semi-finished products, moving parts, etc. Then, for synchronized and distortion-reduced multi-view video frames, candidate dynamic regions are generated using image segmentation models, video segmentation models, object tracking models, or keying models. Next, video propagation, tracking, or keying processing is performed based on adjacent video frames to obtain a temporal mask with temporal continuity. Subsequently, post-processing is performed on the temporal mask, including opening, closing, hole filling, small connected region removal, boundary smoothing, or edge feathering, to obtain a region mask corresponding to the frame number and camera viewpoint. The quality of the region mask does not require complete pixel-level error-free operation, but it should avoid systematically missing dynamic subjects or mislabeling large areas of static background as dynamic regions. For regions where hands and tools overlap, tools and materials contact each other, or dynamic objects occlude each other, the relevant regions are merged into the same dynamic region.

[0091] For further explanation, please refer to Figure 4 , Figure 4 It is based on Figure 1 The corresponding embodiment shows a flowchart describing the steps of dividing Gaussian elements in a local Gaussian element slice by using a replicated reference snapshot and a region mask corresponding to the frame number and camera viewpoint of the multi-view video frame, to obtain static Gaussian slices and dynamic Gaussian slices.

[0092] The step S130 provided in this application embodiment, which involves dividing Gaussian elements in a local Gaussian element slice using a replicated reference snapshot and a region mask corresponding to the frame number and camera viewpoint of the multi-view video frame, to obtain static and dynamic Gaussian slices, includes: Step S131: Set a dynamic response value on the Gaussian cell in the replicated reference snapshot; Step S132: Based on the camera parameters corresponding to the camera view of the multi-view video frame, render the dynamic response value onto the image plane under the camera view to obtain the dynamic prediction mask corresponding to the frame number and camera view of the multi-view video frame. Step S133: Optimize the dynamic response value by dynamically predicting the difference between the mask and the corresponding region mask; Step S134: Based on the optimized dynamic response value, divide the Gaussian elements in the local Gaussian element partition to obtain static Gaussian partition and dynamic Gaussian partition.

[0093] Specifically, a dynamic response value is set for each Gaussian cell in the replicated reference snapshot. The dynamic response value represents the degree to which the corresponding Gaussian cell belongs to a dynamic region. The dynamic response value is a trainable scalar parameter and also an intermediate parameter for obtaining the dynamic probability after mapping. For the ... Each high-level element can be used to set a dynamic response value. and through Sigmoid The function obtains the dynamic probability :

[0094] in, The closer it is to 1, the more likely the corresponding Gaussian element is to be a dynamic Gaussian element; The closer it is to 0, the more likely the corresponding Gaussian element is to be a static Gaussian element.

[0095] It should be noted that the dynamic response value is a time-independent Gaussian parameter. The same Gaussian corresponds to the same dynamic response value throughout the entire video timeline. Therefore, the dynamic response value ensures that the Gaussian motion / static state judgment for each video frame does not change over time, thus enabling dynamic Gaussian fragments to have stable membership and index relationships. This provides a stable object foundation for subsequent deformation field training, dynamic deformation flow generation, and interactive video resource export.

[0096] After configuring the dynamic response value for the replicated reference snapshot in step S131, step S132 treats the dynamic response value as a renderable scalar of Gaussian elements, instead of using the color attributes of Gaussian elements as the mask prediction output. For frame number... t and camera perspective c Based on the camera parameters corresponding to camera viewpoint c, the Gaussian elements in the replicated reference snapshot are projected onto the image plane under that camera viewpoint, and the dynamic probabilities of each Gaussian element are rendered in a differentiable manner to obtain the dynamic prediction mask. .

[0097] For example, dynamic prediction mask Medium pixel The values ​​of are represented as follows:

[0098] in, Indicates the first A Gaussian element in frame number t and camera perspective c Lower pixel Rendering contribution weight Indicates the first The dynamic probability of a Gaussian element. This rendering contribution weight can be related to projection coverage, depth relationship, cumulative transparency, or visibility.

[0099] Although the dynamic response value itself can be a time-independent Gaussian element-level parameter, the projection position, occlusion relationship, and visibility of Gaussian elements differ at different frame numbers and camera viewpoints. Therefore, the rendered dynamic prediction mask can still produce different 2D prediction results for different frame numbers and camera viewpoints. Thus, the embodiments of this application can transform 2D region mask supervision into 3D Gaussian element-level dynamic and static partitioning supervision.

[0100] The obtained dynamic prediction mask is processed by executing step S133 with the same frame number. t Same camera perspective c Corresponding area mask The dynamic response value is optimized based on the difference between the two, i.e., the result of the dynamic-static separation loss.

[0101] In specific calculations, the difference can be represented by binary cross-entropy loss, L1 loss, Dice loss, focus loss, or a combination of the above losses.

[0102] For example, the dynamic-static separation loss is expressed by the following formula:

[0103] in, Represents a set of pixels. This represents the numerically stable term. This dynamic-static separation loss ensures that Gaussian elements projected onto the dynamic region have a higher dynamic probability, while those projected onto the non-dynamic region have a lower dynamic probability.

[0104] During the dynamic-static partitioning optimization, only the dynamic response value is optimized, while the Gaussian meta-attributes in the replicated reference snapshot are not updated. By optimizing only the dynamic response value, the optimization degrees of freedom for dynamic-static separation are reduced, making the dynamic-static partitioning more focused on region mask supervision, rather than adapting to the mask by changing the Gaussian meta-attributes. This improves the stability and interpretability of the resulting static and dynamic Gaussian partitions.

[0105] In addition, sparsity constraints, class balancing weights, dynamic region expansion suppression constraints, or cross-view consistency constraints can be added to the dynamic response values. Sparsity constraints are used to prevent too many Gaussian elements from being included in the dynamic region; class balancing weights are used to alleviate the problem of imbalance in the area ratio between the dynamic and non-dynamic regions; and cross-view consistency constraints are used to ensure that the same Gaussian element has a consistent dynamic and static tendency under the supervision of different camera viewpoints.

[0106] Thus, with the optimization of the dynamic response value, the Gaussian elements in the local Gaussian element partition are divided to obtain static Gaussian partitions and dynamic Gaussian partitions.

[0107] For example, in step S134, a dynamic threshold is set. When the first The dynamic probability of a Gaussian element Greater than or equal to the dynamic threshold When the Gaussian element is defined as a dynamic Gaussian element; when the first Gaussian element is defined as a dynamic Gaussian element. The dynamic probability of a Gaussian element Less than the dynamic threshold When this happens, the Gaussian element is identified as a static Gaussian element, that is, the dynamic and static labels are obtained for the Gaussian element.

[0108] Since the Gaussian elements in the replicated reference snapshot retain the global Gaussian element sequence number, after obtaining the dynamic and static tags, the dynamic tags are mapped to local Gaussian element fragments in each graphics processor according to the global Gaussian element sequence number. For a local Gaussian element fragment held by a graphics processor, Gaussian elements marked as dynamic are assigned to the graphics processor's dynamic Gaussian fragment, and Gaussian elements marked as static are assigned to the graphics processor's static Gaussian fragment.

[0109] In this way, instead of reselecting dynamic Gaussian elements in each video frame, this embodiment obtains relatively stable dynamic and static Gaussian partitions throughout the entire video timeline based on replicated reference snapshots, frame-by-frame region mask supervision, and time-independent dynamic response values. This stable partitioning reduces temporal instability caused by frequent changes in the dynamic and static identities of Gaussian elements between adjacent video frames. Furthermore, it provides stable input objects for subsequent deformation field networks to model temporal variations only in dynamic Gaussian partitions, and establishes a one-to-one correspondence between dynamic deformation flows and dynamic Gaussian elements.

[0110] In summary, the embodiments of this application utilize replicated reference snapshots obtained by aggregating multiple local Gaussian element slices to provide a complete and stable dynamic-static partitioning benchmark; they use frame-by-frame region masks corresponding to frame numbers and camera viewpoints to supervise dynamic response values, transforming 2D dynamic region information into Gaussian element-level dynamic-static labels; and they form stable static and dynamic Gaussian slices through time-independent dynamic response values. These processes eliminate the need for subsequent deformation field training to be performed on all Gaussian elements and avoid the dynamic Gaussian element set switching with each frame, thereby improving the stability of parallel reconstruction of 4D dynamic scenes.

[0111] After the dynamic Gaussian partitioning performed in step S130 is completed, the deformation field network will be trained in step S140.

[0112] In step S140, it should first be noted that the deformation field network is used to predict the attribute changes of the dynamic Gaussian elements in the corresponding video frame based on the reference position and time information of the dynamic Gaussian elements. The attribute changes, which are adapted to the Gaussian element attributes, include at least one of position changes, scale changes, and rotation changes. In addition, they may also include opacity changes or appearance changes.

[0113] In one exemplary embodiment, the deformation field network includes at least one of a hash grid encoder, a multilayer perceptron decoder, a spatiotemporal coding module, a position output head, a scale output head, and a rotation output head. Interference between changes in different properties is reduced by predicting changes in position, scale, and rotation separately.

[0114] For example, deformation field networks Used to determine the reference position based on the dynamic Gaussian element. and video frame timestamps Predict the attribute changes of this dynamic Gaussian primitive in the corresponding video frame:

[0115] in, Indicates a change in position; Indicates scale change. This indicates rotational changes. The Gaussian attributes of the dynamic Gaussian elements in the corresponding video frame are calculated based on these attribute changes.

[0116] Similarly, deformation field networks can also output changes in transparency, color, or other Gaussian meta-attributes.

[0117] The training of the Deformation Field Network (DNF) further includes a pre-training phase and a joint optimization phase. Pre-training uses a replicated reference snapshot as the baseline Gaussian representation for DNF training and determines the reference Gaussian elements in the replicated reference snapshot that participate in the DNF training based on the dynamic Gaussian slices held by each graphics processor. These reference Gaussian elements correspond to the dynamic Gaussian elements that require subsequent time-varying modeling. The DNF is trained using these reference Gaussian elements, and the Gaussian attributes in the replicated reference snapshot are kept unchanged during training. In this way, the DNF learns the initial time-varying mapping relationship first, without being affected by the unstable changes in the dynamic Gaussian attributes themselves in the early stages of training.

[0118] For details on the pre-training process in step S140, please refer to [link / reference]. Figure 5 , Figure 5 It is based on Figure 1 A flowchart describing the pre-training process of step S140 in the corresponding embodiment.

[0119] The pre-training process in step S140 includes: Step S141a: Load the same deformation field network in each graphics processor; Step S142a: Use the replicated reference snapshot as the baseline Gaussian representation for training the deformation field network, and determine the reference Gaussian primitives in the replicated reference snapshot that participate in the training of the deformation field network based on the dynamic Gaussian slices held by each graphics processor. Step S143a: Train the deformation field network by referencing Gaussian elements, and keep the Gaussian element attributes in the replicated state reference snapshot from being updated during training. Step S144a: Synchronize the gradients of the deformation field network trained in each graphics processor, and update the deformation field network in each graphics processor according to the synchronized deformation field network gradients.

[0120] Each graphics processing unit (GPU) calculates the attribute changes of the deformation field network (DNF) output based on the reference Gaussian unit's baseline position and the timestamp corresponding to the video frame, and obtains the DNF gradient based on the training loss. The DNF gradients trained across multiple GPUs are aggregated or synchronized to obtain the synchronized DNF gradient, and the DNF in each GPU is updated based on this synchronized gradient. Since each GPU runs the same DNF and uses the synchronized gradient for updates, the DNF parameters across all GPUs remain consistent.

[0121] For details on the pre-training process in step S140, please refer to [link / reference]. Figure 6 , Figure 6 It is based on Figure 1A flowchart describing the joint optimization process of step S140 in the corresponding embodiment.

[0122] The joint optimization process in step S140 includes: Step S141b: After training the deformation field network with a replicated reference snapshot, the deformation field network is jointly optimized using dynamic Gaussian slices held by each graphics processor. Step S142b: During the joint optimization process, the dynamic Gaussian primitive attributes in the dynamic Gaussian slices held by each graphics processor are locally updated, and the training gradients of the deformation field network obtained by multiple graphics processors are aggregated to obtain the synchronous gradient. Step S143b: Update the deformation field network in each graphics processor according to the synchronization gradient.

[0123] In the joint optimization phase, the deformation field network is jointly optimized using dynamic Gaussian slices held by each graphics processor (GPU). Specifically, the dynamic Gaussian primitives from the dynamic Gaussian slices held by each GPU are input into the deformation field network of the corresponding GPU to obtain the attribute changes of the dynamic Gaussian primitives under the corresponding video frame. Updated dynamic Gaussian slices are obtained based on these attribute changes, and these updated slices, along with the static Gaussian slices held by the corresponding GPU, participate in cross-device differentiable Gaussian rendering to obtain the predicted image for the joint optimization phase.

[0124] Subsequently, backpropagation is performed based on the training loss corresponding to the predicted image from the joint optimization phase to obtain the attribute gradients of the dynamic Gaussian slices and the training gradients of the deformation field network in each graphics processor. The attribute gradients of the dynamic Gaussian slices are used to locally update the dynamic Gaussian primitive attributes in the corresponding graphics processor; the training gradients of the deformation field network obtained from multiple graphics processors are aggregated to obtain the synchronization gradient, and the deformation field network in each graphics processor is updated according to the synchronization gradient.

[0125] Each graphics processor (GPU) updates its loaded deformation field network parameters based on the synchronization gradient. Thus, the Gaussian primitive properties of the dynamic Gaussian partitions can be updated locally within their respective GPUs, while the deformation field network remains consistent across multiple GPUs through the synchronization gradient.

[0126] In addition to the above, deformation field training can also introduce at least one of the following constraints: temporal smoothing constraint, deformation amplitude constraint, scale stability constraint, rotational continuity constraint, mask constraint, and cross-viewpoint consistency constraint. Temporal smoothing constraint is used to suppress abrupt changes in attributes between adjacent video frames; deformation amplitude constraint is used to avoid excessively large and unreasonable offsets in dynamic Gaussian elements; scale stability constraint and rotational continuity constraint are used to improve the geometric continuity of dynamic Gaussian elements between different video frames. Through these constraints, the stability of dynamic relationships is further improved.

[0127] Furthermore, the joint optimization phase can employ a delayed unfreezing strategy or a phased learning rate strategy. For example, during the pre-training phase, only the deformation field network is trained; after pre-training, dynamic Gaussian trigonometric attribute updates in the dynamic Gaussian slices are enabled; during the phase where the learning rate is high and the model still has convergence space, only the dynamic Gaussian slices are densified; and in the later stages of training, the addition of new Gaussian trigonometric units is disabled to reduce the impact of later additions on temporal consistency and playback stability.

[0128] For the trained deformation field network, step S150 generates the dynamic deformation flow of each video frame corresponding to the dynamic Gaussian slice in each graphics processor.

[0129] After the deformation field network is trained, the dynamic Gaussian primitives of each GPU's dynamic Gaussian slice are sequentially input into the trained deformation field network according to multiple video frames in the exported timeline, obtaining the attribute changes of the dynamic Gaussian primitives corresponding to each video frame. For any video, the attribute changes of each dynamic Gaussian primitive under that video frame are organized according to the order of the dynamic Gaussian primitives in the dynamic Gaussian slice, obtaining the dynamic deformation data corresponding to that video frame. The dynamic deformation data corresponding to multiple video frames are organized in chronological order, thus forming a dynamic deformation stream.

[0130] When generating a dynamic deformation stream, the order of dynamic Gaussian primitives in the dynamic Gaussian slices should be consistent with the order of attribute changes in the dynamic deformation data. In other words, each attribute change in the dynamic deformation stream should correspond to a dynamic Gaussian primitive in the dynamic basis. This correspondence can be recorded through the global Gaussian primitive index, the index within the dynamic Gaussian slice, the exported sorting index, the resource index, or metadata fields. A stable index correspondence can prevent the incorrect application of attribute changes from a particular video frame to other dynamic Gaussian primitives during playback or reconstruction.

[0131] After obtaining the dynamic deformation flow of each video frame corresponding to the dynamic Gaussian primitive slice in step S150, the static Gaussian slice, dynamic Gaussian slice, and dynamic deformation flow of each video frame corresponding to the dynamic Gaussian slice on each graphics processor are generated under the action of step S160, generating interactive video resources for reconstructing four-dimensional dynamic scenes.

[0132] In step S160, the static Gaussian slices, dynamic Gaussian slices, and corresponding dynamic deformation streams held by each graphics processor are aggregated. During aggregation, Gaussian primitives and deformation data from multiple graphics processors are uniformly arranged according to the global Gaussian primitive index, slice offset, dynamic Gaussian slice intra-index, or derived sorting rules. Subsequently, a static base, a dynamic base, and a dynamic deformation stream are generated. The static base is obtained by converting the Gaussian primitives corresponding to the static Gaussian slices and is used to store the geometric, transparency, and appearance attributes of the static Gaussian primitives. The dynamic base is obtained by converting the dynamic Gaussian primitives corresponding to the dynamic Gaussian slices and is used to store the basic attributes of the dynamic Gaussian primitives in the baseline state. The dynamic deformation stream is used to store the attribute changes of the dynamic Gaussian primitives corresponding to each video frame.

[0133] During the resource generation process described above, it is necessary to maintain the index correspondence between the dynamic basis and the dynamic deformation flow. For example, in the dynamic basis, the first... The dynamic Gaussian element corresponds to the first dynamic Gaussian element in each video frame of the dynamic deformation stream. Group attribute change data. This correspondence can be written into the resource metadata. Resource metadata includes at least one of the following: static Gaussian pixel count, dynamic Gaussian pixel count, attribute graph size, quantization parameters, sorting method, dynamic representation method, frame rate, frame count, timestamp index, dynamic deformation stream index, resource version number, and verification fields. Recording the above information through metadata reduces the risk of data inconsistencies between the exporting and playback ends.

[0134] To further explain, quantization compression is performed separately on the static basis, dynamic basis, and dynamic deformation flow to generate interactive video resources. The static and dynamic basis are quantized according to Gaussian meta-attributes, such as normalization, codebook quantization, linear quantization, or other compression processing of position, scale, rotation, opacity, color, or spherical harmonic coefficients. Position attributes are quantized using signed logarithmic transform and multi-bit quantization; scale attributes are quantized or codebook compressed in the log-scale domain; rotation attributes are encoded using quaternions; and color and opacity are quantized using channel quantization, codebook quantization, or cluster compression.

[0135] The quantized Gaussian properties can be mapped to a two-dimensional property graph. Each Gaussian primitive corresponds to a pixel location or fixed patch in the two-dimensional property graph. The two-dimensional property graph can be arranged in spatial order, Morton order, PLAS order, or other compression-friendly arrangements. For dynamic basis, the arrangement order of dynamic Gaussian primitives in the two-dimensional property graph should be consistent with the arrangement order of property changes in the dynamic deformation flow.

[0136] For dynamic deformation streams, the attribute changes of dynamic Gaussian elements in each video frame are quantized and encoded into a Gaussian element attribute label map. This Gaussian element attribute label map is not a plain image for direct display, but rather a data map carrying labels for dynamic Gaussian element attribute changes. The Gaussian element attribute label map uses a two-dimensional atlas layout, for example, placing low-order labels, high-order labels, scale change labels, and rotation change labels in different regions or channels. The Gaussian element attribute label maps corresponding to multiple video frames can be stored as frame-by-frame image files, binary files, image sequences, or video encoded streams, and can also be compressed using WebP, H.265 / HEVC, or other encoding methods. During encoding, the color space, sampling format, and compression parameters are configured according to the label value fidelity requirements to reduce label value distortion.

[0137] By using separate quantization compression of static basis, dynamic basis, and dynamic deformation flow, interactive video resources do not need to repeatedly store the complete four-dimensional dynamic scene representation for each video frame. Instead, relatively stable content is expressed in the static basis, the baseline attributes of dynamic Gaussian units are expressed in the dynamic basis, and the time-varying attribute changes are expressed in the dynamic deformation flow. This reduces the data volume of interactive video resources and improves the efficiency of subsequent frame-by-frame reconstruction and interactive display on the terminal.

[0138] Therefore, on the terminal reconstruction and playback side, the terminal device loads the generated interactive video resources and parses the resource metadata to obtain the relationship between static basis, dynamic basis, dynamic deformation flow, quantization parameters, frame index and dynamic Gaussian meta-index.

[0139] The terminal device decodes and dequantizes the static and dynamic bases, recovers the baseline properties of the static and dynamic Gaussian primitives, and merges them into a unified Gaussian rendering object. In addition, the terminal device records the dynamic range of the dynamic Gaussian primitives within the unified Gaussian rendering object.

[0140] When the player's timeline points to the target video frame, the terminal device reads or decodes the attribute change data corresponding to the target video frame from the dynamic deformation stream, and restores the dynamic Gaussian primitive attributes under the target video frame based on the dynamic basis and attribute change data. Subsequently, the terminal device only performs attribute write-back on the dynamic range in the unified Gaussian rendering object, while the static Gaussian primitives remain unchanged. For transparent Gaussian rendering scenes, after the position of the dynamic Gaussian primitives changes, the sorting center, bounding box, or GPU rendering auxiliary data can also be updated to improve the accuracy of transparent blending and rendering results. In this way, the terminal device can restore the four-dimensional dynamic scene frame by frame without reconstructing the complete Gaussian scene frame by frame, and supports interactive operations such as rotation, scaling, translation, pausing, dragging the timeline, or changing the virtual camera viewpoint, i.e., in a scene like... Figure 7Various interactive operations are performed when the interactive controls shown are triggered.

[0141] In summary, the embodiments of this application form a complete process from four-dimensional dynamic scene data processing to parallel reconstruction and interactive video resource generation through Gaussian primitive fragmentation scheduling, cross-device rendering contribution aggregation, replicated reference snapshot-assisted dynamic-static separation and deformation field pre-training, dynamic Gaussian fragmentation joint optimization, dynamic deformation flow generation, and interactive video resource separate compression. The embodiments of this application can improve the feasibility and stability of four-dimensional dynamic scene reconstruction under limited computing resources and provide a data foundation for subsequent frame-by-frame restoration and free-viewpoint interactive display in terminal devices.

[0142] In one exemplary embodiment, this application also provides a computer device including a memory, a processor, and a computer program stored in the memory, the processor executing the computer program to implement the steps of the method as described above.

[0143] In one exemplary embodiment, this application also provides a computer program product including a computer program that, when executed by a processor, implements the steps of the method as described above.

[0144] In one exemplary embodiment, this application also provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the method as described above.

[0145] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this application.

[0146] In an exemplary embodiment of this application, a computer-readable storage medium is also provided, on which computer-readable instructions are stored, which, when executed by a computer's processor, cause the computer to perform the methods described in the above method embodiments.

[0147] According to one embodiment of this application, a program product for implementing the methods in the above-described method embodiments is also provided. This product may employ a portable compact disc read-only memory (CD-ROM) and include program code, and can run on a terminal device, such as a personal computer. However, the program product of this invention is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0148] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0149] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0150] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, or any suitable combination thereof.

[0151] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0152] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0153] Furthermore, although the steps of the method in this application are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0154] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, and includes several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the method according to the embodiments of this application.

[0155] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the appended claims.

Claims

1. A parallel reconstruction method for a four-dimensional dynamic scene, characterized in that, The method includes: The Gaussian primitives representing a four-dimensional dynamic scene are trained and scheduled, and the Gaussian primitives are distributed to multiple graphics processors to form local Gaussian primitive fragments in each graphics processor. Based on the multi-view video frames of the four-dimensional dynamic scene, cross-device differentiable Gaussian rendering is used to enable each graphics processor to calculate the rendering contribution of the local Gaussian montage, and the rendering contributions of multiple graphics processors are aggregated across devices. Based on the aggregated prediction image and its backpropagation gradient, the Gaussian attributes of the local Gaussian montage in each graphics processor are updated. By using a replicated reference snapshot and a region mask corresponding to the frame number and camera view of the multi-view video frame, Gaussian elements in the local Gaussian element slice are divided to obtain static Gaussian slices and dynamic Gaussian slices. The replicated reference snapshot is a complete reference Gaussian representation obtained by summing up the local Gaussian element slices currently held by multiple graphics processors. The same deformation field network is run on multiple graphics processors (GPUs), and synchronously trained using a replicated reference snapshot and dynamic Gaussian slices held by each GPU to obtain a deformation field network for reconstructing the dynamic changes in the four-dimensional dynamic scene. The synchronous training step includes a pre-training phase and a joint optimization phase: In the pre-training phase, the same deformation field network is loaded on each GPU; the replicated reference snapshot is used as the baseline Gaussian representation for training the deformation field network, and reference Gaussian elements participating in the training of the deformation field network in the replicated reference snapshot are determined based on the dynamic Gaussian slices held by each GPU; the deformation field network is trained using the reference Gaussian elements, and the network is maintained during training. The Gaussian meta-attributes in the replicated reference snapshot are not updated during training; the gradients of the deformation field network trained in each graphics processor are synchronized, and the deformation field network in each graphics processor is updated according to the synchronized deformation field network gradient; in the joint optimization phase, after training the deformation field network with the replicated reference snapshot, the deformation field network is jointly optimized through the dynamic Gaussian slices held by each graphics processor; during the joint optimization process, the dynamic Gaussian meta-attributes in the dynamic Gaussian slices held by each graphics processor are locally updated, and the training gradients of the deformation field network obtained by multiple graphics processors are aggregated to obtain the synchronized gradient; the deformation field network in each graphics processor is updated according to the synchronized gradient; The deformation field network generates dynamic deformation flows for each video frame corresponding to dynamic Gaussian slices in each graphics processor. Interactive video resources for reconstructing the four-dimensional dynamic scene are generated by using static Gaussian slices, dynamic Gaussian slices, and the dynamic deformation flow of the dynamic Gaussian slices corresponding to each video frame on each graphics processor.

2. The method according to claim 1, characterized in that, The training and scheduling of Gaussian primitives representing a four-dimensional dynamic scene, distributing the Gaussian primitives to multiple graphics processing units (GPUs), and forming local Gaussian primitive slices in each GPU includes: Gaussian elements are obtained from the initialization of multi-view video frames, and these Gaussian elements are used to characterize the spatial distribution of the four-dimensional dynamic scene. Assign a global Gaussian number to the Gaussian; Based on the global Gaussian element number and the number of graphics processors, calculate the target graphics processor corresponding to each Gaussian element; The Gaussian attributes of each Gaussian are written to the local parameter storage area of ​​the corresponding target graphics processor to form local Gaussian fragments in each graphics processor.

3. The method according to claim 2, characterized in that, The method further includes: For each local Gaussian element slice formed in a graphics processor, a local training state is established, wherein the local training state corresponds one-to-one with the Gaussian elements in the local Gaussian element slice. The local training state includes at least one of the following: optimizer state associated with the corresponding Gaussian element, visibility statistics, densification statistics, and pruning tags, and the local training state is stored in the graphics processor holding the local Gaussian element fragment.

4. The method according to claim 1, characterized in that, The multi-view video frames based on the four-dimensional dynamic scene are rendered using cross-device differentiable Gaussian rendering, enabling each graphics processor to calculate its rendering contribution to the local Gaussian metadata slices. The rendering contributions from multiple graphics processors are then aggregated across devices. Based on the aggregated predicted image and its backpropagation gradient, the Gaussian metadata attributes of the local Gaussian metadata slices in each graphics processor are updated, including: Based on the camera parameters corresponding to the camera viewpoint of the multi-view video frame, the local Gaussian primitive slices in each graphics processor are projected onto the image plane under the camera viewpoint. In each graphics processor, the rendering contribution of the local Gaussian pixel is calculated by projecting the local Gaussian pixel onto the image plane; The rendering contributions calculated by each graphics processor are aggregated across devices to obtain the predicted image corresponding to the multi-view video frame; Based on the backpropagation gradient of the predicted image, the gradient corresponding to each local Gaussian piece is backpropagated to the corresponding graphics processor to update the Gaussian attributes of the local Gaussian pieces in the graphics processor.

5. The method according to claim 1, characterized in that, The process of dividing Gaussian elements in a local Gaussian element slice using a replicated reference snapshot and a region mask corresponding to the frame number and camera viewpoint of the multi-view video frame results in static and dynamic Gaussian slices, including: A dynamic response value is set on the Gaussian cell in the replicated reference snapshot, and the dynamic response value is used to indicate the degree to which the corresponding Gaussian cell belongs to the dynamic region; Based on the camera parameters corresponding to the camera view of the multi-view video frame, the dynamic response value is rendered onto the image plane under the camera view to obtain a dynamic prediction mask corresponding to the frame number and camera view of the multi-view video frame. The dynamic response value is optimized by using the difference between the dynamic prediction mask and the corresponding region mask; Based on the optimized dynamic response value, the Gaussian elements in the local Gaussian element partition are divided to obtain static Gaussian partitions and dynamic Gaussian partitions.

6. The method according to claim 1, characterized in that, The joint optimization of the deformation field network using dynamic Gaussian partitioning held by each graphics processor includes: The dynamic Gaussian primitives in the dynamic Gaussian slices held by each graphics processor are input into the deformation field network in the corresponding graphics processor to obtain the attribute changes of the dynamic Gaussian primitives in the corresponding video frame. The updated dynamic Gaussian slice is obtained based on the attribute changes, and the updated dynamic Gaussian slice and the static Gaussian slice held by the corresponding graphics processor are used together to participate in cross-device differentiable Gaussian rendering to obtain the predicted image in the joint optimization stage. Backpropagation is performed based on the training loss corresponding to the predicted image in the joint optimization stage to obtain the attribute gradient of the dynamic Gaussian slice in each graphics processor and the training gradient of the deformation field network. Based on the attribute gradient of the dynamic Gaussian slice, the dynamic Gaussian primitive attributes in the corresponding graphics processor are updated.

7. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1-6.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-6.

Images