A dynamic scene reconstruction method and device based on Gaussian point cloud

Abstract

This application discloses a method and apparatus for dynamic scene reconstruction based on Gaussian point clouds. First, at least six synchronized cameras are used to acquire video from at least six different camera positions, obtaining a multi-view image sequence. Based on the multi-view image sequence and a four-dimensional Gaussian model, a Gaussian point cloud containing a time dimension and a Gaussian identification code is generated. The Gaussian point cloud containing the time dimension and Gaussian identification code is integrated to generate an enhanced single-unit PLY file, wherein the enhanced single-unit PLY file contains a Gaussian identification code, a parameter sequence corresponding to keyframes of the time sequence within a specified time period, and an attribute sequence. The parameter sequence and attribute sequence of the enhanced single-unit PLY file are compressed to obtain a compressed enhanced single-unit PLY file. Then, a dedicated GPU renderer is used to parse and dynamically render the compressed enhanced single-unit PLY file to obtain a multi-view dynamic virtual scene. Thus, high-fidelity dynamic scene reconstruction can be achieved.

Description

Technical Field

[0001] This application relates to the fields of computer graphics and virtual reality technology, and in particular to a method and apparatus for dynamic scene reconstruction based on Gaussian point clouds. Background Technology

[0002] Currently, scene reconstruction based on Gaussian point clouds mainly focuses on static scenes. To achieve dynamic scene reconstruction, it is necessary to record the Gaussian point cloud data of each keyframe and its corresponding time relationship, and then reconstruct the scene frame by frame in chronological order to achieve a dynamic effect.

[0003] However, there is currently no standard format for storing Gaussian point clouds with temporal relationships. When rendering and presenting, existing tools often only support Gaussian point cloud data in standard format, reconstructing a static scene at a certain moment frame by frame.

[0004] Therefore, how to associate Gaussian point clouds with time, and how to reconstruct static scenes in chronological order to present dynamic effects, have become research topics and challenges that need to be addressed and overcome one by one for dynamic scene reconstruction based on Gaussian point cloud data.

[0005] Specifically, in cross-frame Gaussian point cloud data, the Gaussian identities are not inherently related. That is, there is no inherent connection between the Gaussian identity G1_A representing an object in the first frame and the Gaussian identity G2_B representing the same object in the second frame. This often leads to the loss of Gaussian identities, and the inability to perform reliable trajectory reconstruction and temporal interpolation is one of the major challenges.

[0006] Furthermore, since the Gaussian point cloud data for each frame is enormous, storing it frame by frame could result in terabyte-level data, which places extremely high demands on storage and computing resources, and may even be difficult to achieve.

[0007] Due to these difficulties, current solutions for reconstructing dynamic scenes based on Gaussian point clouds mostly focus on reducing the number of frames, using a smaller number of keyframes for prediction and interpolation, or combining static Gaussian point clouds based on keyframes with deformation fields. However, these methods are only suitable for dynamic scenes with simple movements and few changes. For scenes that require accurate reproduction of complex movements and real-time interaction, they cannot guarantee high-fidelity image quality and accurate capture of complex movements, and it is also difficult to meet the requirements of real-time rendering and presentation during real-time interaction. Summary of the Invention

[0008] This application provides a method and apparatus for dynamic scene reconstruction based on Gaussian point clouds.

[0009] According to a first aspect of the embodiments of this application, a dynamic scene reconstruction method based on Gaussian point clouds is provided, applied to a scene acquisition end, comprising: using at least six synchronized cameras to acquire video from at least six different camera positions to obtain a multi-view image sequence; generating a Gaussian point cloud containing a time dimension and a Gaussian identification code based on the multi-view image sequence and a four-dimensional Gaussian model; integrating the Gaussian point cloud containing the time dimension and the Gaussian identification code to generate an enhanced single-unit PLY file, the enhanced single-unit PLY file containing a Gaussian identification code, a parameter sequence corresponding to a key frame of the time sequence within a specified time period, and an attribute sequence; compressing the parameter sequence and attribute sequence of the enhanced single-unit PLY file to obtain a compressed enhanced single-unit PLY file.

[0010] According to one embodiment of this application, a Gaussian point cloud containing a time dimension and a Gaussian identity code is generated based on a multi-view image sequence and a four-dimensional Gaussian model. The method includes: determining camera parameters and sparse point clouds for at least six synchronized cameras based on the multi-view image sequence; training the four-dimensional Gaussian model based on the camera parameters and sparse point clouds to generate Gaussian point clouds frame by frame, resulting in a Gaussian point cloud containing a time dimension; matching Gaussian identities within the Gaussian point cloud containing the time dimension across multiple consecutive frames based on nearest distance, appearance similarity, and motion prior information, determining the Gaussian identity code corresponding to each Gaussian identity; and adding the Gaussian identity code corresponding to each Gaussian identity to the Gaussian point cloud containing the time dimension, resulting in a Gaussian point cloud containing both the time dimension and the Gaussian identity code.

[0011] According to one embodiment of this application, an enhanced single-unit PLY file is generated by integrating a Gaussian point cloud containing a time dimension and a Gaussian identification code. This includes: determining world coordinates; performing coordinate system transformation on the Gaussian point cloud containing the time dimension and Gaussian identification code using the world coordinates to obtain a Gaussian point cloud containing the time dimension and Gaussian identification code in coordinate system one; performing time alignment on the Gaussian point cloud containing the time dimension and Gaussian identification code in coordinate system one based on the time dimension to obtain a time-aligned Gaussian point cloud containing the time dimension and Gaussian identification code in coordinate system one; and merging and deduplicating trajectories, as well as unifying attributes and parameters, on the time-aligned Gaussian point cloud containing the time dimension and Gaussian identification code in coordinate system one based on the Gaussian identification code and the time dimension to obtain an enhanced single-unit PLY file containing the complete spatiotemporal trajectory of the Gaussian point cloud.

[0012] According to one embodiment of this application, the parameter sequence and attribute sequence of an enhanced single-unit PLY file are compressed to obtain a compressed enhanced single-unit PLY file, including: determining a set of key coefficients for frequency domain transformation compression based on the motion trajectory and / or frequency characteristics of each Gaussian identity; using the set of key coefficients to perform frequency domain transformation compression on the parameter sequence and attribute sequence of the enhanced single-unit PLY file; and storing the set of key coefficients in the enhanced single-unit PLY file to obtain the compressed enhanced single-unit PLY file.

[0013] According to one embodiment of this application, a set of key coefficients for frequency domain transform compression is determined based on the motion trajectory and / or frequency characteristics of each Gaussian identity, including: obtaining the motion trajectory and / or frequency characteristics of each Gaussian identity; if the motion trajectory and / or frequency characteristics indicate that the Gaussian identity is a smooth background region, then a set of key coefficients with a lower compression ratio is used; if the motion trajectory and / or frequency characteristics indicate that the Gaussian identity is a high-speed or complex action, then a set of key coefficients with a higher compression ratio is used.

[0014] According to one embodiment of this application, the enhanced single-entity PLY file also includes traceable metadata.

[0015] According to a second aspect of the embodiments of this application, a dynamic scene reconstruction method based on Gaussian point clouds is provided, applied to a scene rendering end. The method includes: using a first GPU renderer to parse a compressed enhanced single-unit PLY file to obtain a Gaussian identity code, a parameter sequence and an attribute sequence corresponding to key frames of a time series within a specified time period; using the first GPU renderer, dynamically rendering the Gaussian identity based on the Gaussian identity code, the parameter sequence and attribute sequence corresponding to the Gaussian identity to key frames of a time series within a specified time period, to obtain a multi-view dynamic virtual scene.

[0016] According to one embodiment of this application, parsing a compressed enhanced single-unit PLY file using a first GPU renderer includes: writing commonly used sine / cosine frequency values ​​into a time table; loading the time table into a high-speed constant area of ​​video memory; querying the time table to obtain the frequency values ​​required for parsing the enhanced single-unit PLY file; and parsing the compressed enhanced single-unit PLY file using the first GPU renderer based on the frequency values.

[0017] According to a third aspect of the embodiments of this application, a dynamic scene reconstruction device based on Gaussian point clouds is provided, applied to a scene acquisition end. The device includes: a video acquisition module for acquiring video from at least six different camera positions using at least six synchronized cameras to obtain a multi-view image sequence; a Gaussian point cloud generation module for generating a Gaussian point cloud containing a time dimension and a Gaussian identification code based on the multi-view image sequence and a four-dimensional Gaussian model; a single-file generation module for integrating the Gaussian point cloud containing the time dimension and the Gaussian identification code to generate an enhanced single-file PLY file, wherein the enhanced single-file PLY file contains a Gaussian identification code, a parameter sequence corresponding to a key frame of the time sequence within a specified time period, and an attribute sequence; and a compression module for compressing the parameter sequence and attribute sequence of the enhanced single-file PLY file to obtain a compressed enhanced single-file PLY file.

[0018] According to a fourth aspect of the embodiments of this application, a dynamic scene reconstruction device based on Gaussian point clouds is provided, applied to a scene presentation end. The device includes: a parsing module, used to parse a compressed enhanced single-unit PLY file using a first GPU renderer to obtain a Gaussian identity code, a parameter sequence and an attribute sequence corresponding to key frames of a time series within a specified time period; and a dynamic rendering module, using the first GPU renderer to dynamically render the Gaussian identity based on the Gaussian identity code, the parameter sequence and attribute sequence corresponding to the Gaussian identity to key frames of a time series within a specified time period, to obtain a multi-view dynamic virtual scene.

[0019] According to a fifth aspect of the present application, a computer storage medium is provided, the storage medium including a set of computer-executable instructions, which, when executed, are used to perform any of the above-described dynamic scene reconstruction methods based on Gaussian point clouds.

[0020] This application provides a method and apparatus for dynamic scene reconstruction based on Gaussian point clouds. First, the following method is executed at the scene acquisition end: using at least six synchronized cameras, video is acquired from at least six different camera positions to obtain a multi-view image sequence; based on the multi-view image sequence and a four-dimensional Gaussian model, a Gaussian point cloud containing a time dimension and a Gaussian identification code is generated; the Gaussian point cloud containing the time dimension and the Gaussian identification code is integrated to generate an enhanced single-unit PLY file, wherein the enhanced single-unit PLY file contains a Gaussian identification code, a parameter sequence corresponding to the keyframe of the time sequence within a specified time period, and an attribute sequence; the parameter sequence and attribute sequence of the enhanced single-unit PLY file are compressed to obtain a compressed enhanced single-unit PLY file. Subsequently, at the scene rendering end, a dedicated GPU renderer (the first GPU renderer) is used to parse the compressed enhanced single-unit PLY file, obtaining the Gaussian identity code, parameter sequence, and attribute sequence corresponding to the time series keyframes within a specified time period. Then, based on the Gaussian identity code and the parameter and attribute sequences corresponding to the Gaussian identity within the specified time period, the Gaussian identity is dynamically rendered to obtain a multi-view dynamic virtual scene. In this way, by using the enhanced single-unit PLY file containing the Gaussian identity code and the parameter and attribute sequences corresponding to the time series keyframes within a specified time period, Gaussian point cloud data is associated with time, and the Gaussian identity code enables identity unification and association across frame data, ensuring the reliability of trajectory reconstruction and time interpolation. Furthermore, compressing the parameter and attribute sequences of the enhanced single-unit PLY file significantly reduces its size, saving considerable storage space while ensuring the fidelity of complex actions. Using a dedicated GPU to parse and render the compressed enhanced single-unit PLY file can significantly reduce the dependence of the scene rendering end on the four-dimensional Gaussian model, reduce the hardware and software costs of the scene rendering end, and make the dynamic scene reconstruction method and device based on Gaussian point cloud of this application easier to implement, deploy and promote. Attached Figure Description

[0021] The above and other objects, features, and advantages of exemplary embodiments of this application will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. Several embodiments of this application are illustrated in the drawings by way of example and not limitation, in which:

[0022] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts.

[0023] Figure 1 This is a schematic diagram of the implementation process of the dynamic scene reconstruction method based on Gaussian point cloud at the scene acquisition end according to an embodiment of this application;

[0024] Figure 2 For this application Figure 1The illustrated embodiment is a schematic diagram of the implementation process of the dynamic scene reconstruction method based on Gaussian point clouds at the scene rendering end.

[0025] Figure 3 This is a schematic diagram illustrating the implementation process of a dynamic scene reconstruction method based on Gaussian point clouds according to another embodiment of this application.

[0026] Figure 4 For this application Figure 3 The illustrated embodiment is a flowchart of the process executed during the four-dimensional Gaussian point cloud generation stage;

[0027] Figure 5 For this application Figure 3 The illustrated embodiment is a flowchart of the data integration and enhanced monolithic PLY file generation phases.

[0028] Figure 6 For this application Figure 3 The illustrated embodiment is a flowchart of the process executed during the compression stage of the enhanced single-unit PLY file.

[0029] Figure 7 For this application Figure 3 The illustrated embodiment is a flowchart of the process executed during the compressed file parsing stage;

[0030] Figure 8 This application embodiment presents a schematic diagram of the composition structure of the scene acquisition terminal device for dynamic scene reconstruction based on Gaussian point clouds;

[0031] Figure 9 This application presents a schematic diagram of the composition structure of a scene acquisition device for dynamic scene reconstruction based on Gaussian point clouds. Detailed Implementation

[0032] To make the objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0033] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0034] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0035] In this application, scene reconstruction refers to obtaining multi-angle videos of real-world scenes captured from multiple cameras, and then virtually reconstructing these videos in a digital virtual world for display and presentation from different angles. Dynamic scene reconstruction, on the other hand, refers to densely reconstructing scenes according to a certain temporal sequence based on different scenes presented at different times, thereby presenting a continuously changing virtual scene to achieve dynamic simulation of various perspectives, actions, expressions, and scene changes. Dynamic scene reconstruction can be widely applied in various virtual reality and augmented reality scenarios.

[0036] The process of realizing dynamic scene reconstruction can be roughly divided into two parts: dynamic scene acquisition and dynamic scene reproduction.

[0037] Dynamic scene acquisition is primarily used to capture real-world scenes from multiple cameras, obtaining multi-angle videos. These videos are then digitally analyzed and converted to produce a computer-readable digital representation. Dynamic scene acquisition is part of the preliminary data preparation phase, typically completed during the research and development stage before dynamic scene presentation. Its main outputs are a digitally represented image capable of dynamic presentation and traceable raw data.

[0038] Dynamic scene rendering is primarily used to render and present digital representations generated from dynamic scene captures, showcasing them as images and videos from different angles to create an immersive visual experience. Dynamic scene rendering is part of the application and typically runs on the user's display device.

[0039] Therefore, the dynamic scene reconstruction method based on Gaussian point clouds in this application is typically implemented in two parts: one part is applied to the dynamic scene acquisition end during the data preparation and R&D phase, and the other part is applied to the dynamic scene presentation end during the actual application and operation phase. These two parts can run on the same electronic device (e.g., during the R&D and testing phase, or in a scenario where dynamic scene acquisition and presentation are performed simultaneously), or they can run on different electronic devices (e.g., the dynamic scene acquisition method runs on the R&D vendor's equipment, while the dynamic scene presentation method runs on the user's electronic device).

[0040] Figure 1 This paper illustrates the implementation flow of a dynamic scene reconstruction method based on Gaussian point clouds according to an embodiment of this application at the scene acquisition end. (Reference) Figure 1 The method includes:

[0041] Operation 110: Use at least six synchronized cameras to acquire video from at least six different camera positions to obtain a multi-view image sequence.

[0042] Among them, a synchronous camera refers to a camera that performs synchronous shooting, which can be any camera with video shooting capabilities.

[0043] Using at least six synchronized cameras, video is captured from at least six different camera positions. The resulting multi-view image sequence includes multiple images captured at the same time from different perspectives of the same scene, as well as a series of images captured at different times from the same perspective of the same scene. Each frame in the multi-view image sequence carries a timestamp of the shooting time. In the implementation of the embodiments of this application, the inventors of this application discovered that at least six synchronized cameras from at least six camera positions are required to obtain a digital representation that can present a dynamic scene in 360 degrees. If there are fewer than six camera positions or fewer than six synchronized cameras, it is difficult to guarantee the presentation effect of the dynamic scene.

[0044] Operation 120 generates a Gaussian point cloud containing a time dimension and a Gaussian identification code based on a multi-view image sequence and a four-dimensional Gaussian model.

[0045] Among them, the four-dimensional Gaussian model refers to the model data obtained by adding a time dimension to the three-dimensional Gaussian model. The three-dimensional Gaussian model is used to reconstruct the three-dimensional object model of objects in the virtual world using digital representation methods, while the four-dimensional Gaussian model can be simply understood as a series of three-dimensional object models with a time sequence relationship.

[0046] Gaussian point clouds refer to a 3D dataset formed by processing raw point cloud data using a Gaussian kernel function. Each Gaussian point cloud contains attributes such as center point location, covariance matrix, and radiance intensity (color and transparency), and can be used in scenes rendered using Gaussian splashing techniques. A 3D Gaussian model generates a set of Gaussian point clouds based on multiple images of the same scene taken from different perspectives. A 4D Gaussian model generates multiple sets of Gaussian point clouds with a time-series relationship. This time-series relationship can be achieved by adding a timestamp attribute to each Gaussian point cloud, storing Gaussian point clouds from the same time in the same storage unit and recording the association between that storage unit and the timestamp, or simply storing them sequentially according to their timestamp chronological order.

[0047] A Gaussian identity is a identifier used to distinguish each Gaussian identity (which can also be understood as a Gaussian object, such as a Gaussian sphere). This attribute is typically not present in Gaussian point clouds. In this embodiment, it is specifically added to ensure that different frames captured at different times are accurately associated with the same Gaussian identity.

[0048] The process of generating Gaussian point clouds containing temporal dimensions and Gaussian identification codes based on multi-view image sequences and a four-dimensional Gaussian model typically involves several steps. For example: first, the multi-view image sequences are input into a COLMAP / RealityCapture adapter to obtain camera parameters and sparse point clouds; then, the camera parameters and sparse point clouds are input into a four-dimensional Gaussian model for training to obtain Gaussian point clouds with temporal dimensions; based on this, adjacent frame data are compared to find Gaussian point clouds that are suspected to have the same Gaussian identity, and these Gaussian point clouds are assigned the same Gaussian identification code. In this way, Gaussian point clouds containing temporal dimensions and Gaussian identification codes can be obtained.

[0049] In this embodiment of the application, by adding the Gaussian identity code, a series of problems caused by the loss of Gaussian identity can be avoided, so that the motion trajectory of the Gaussian identity can be reconstructed more accurately and smoothly, which is especially suitable for scenarios with complex movements, requirements for accurate reproduction and high image quality.

[0050] Operation 130 integrates the Gaussian point cloud containing the time dimension and Gaussian identification code to generate an enhanced single-unit PLY file. The enhanced single-unit PLY file contains the Gaussian identification code, the parameter sequence and attribute sequence corresponding to the time series keyframes within the specified time period.

[0051] Among them, PLY files, or point cloud standard files (Polygon Files), are typically used to store attribute information such as the position, color, normal, and texture coordinates of points, and can represent triangular patches and polygons.

[0052] In this application, a single-unit PLY file refers to a PLY file containing point cloud information corresponding to a set of Gaussian identities, generated by merging image sequences from various viewpoints. An enhanced single-unit PLY file refers to a single-unit PLY file with extended attributes added.

[0053] Typically, a single PLY file stores only the point cloud information (spatial location) and Gaussian attribute information (covariance matrix, color, and transparency, etc.) of a Gaussian identity at a specific moment. However, in the dynamic scene reconstruction method based on Gaussian point clouds proposed in this application, the enhanced single PLY file stores the parameter sequence and attribute sequence corresponding to the Gaussian identity and time-series keyframes over a certain time period. This further expands the functionality of the single PLY file, enabling it to not only be a carrier of static information but also a carrier of dynamic information.

[0054] Time-series keyframes refer to a set of keyframes stored sequentially according to their timestamps. By playing these keyframes in the order they are stored, the dynamic scene captured within the corresponding time period can be presented.

[0055] In most existing solutions, individual PLY files are typically stored in specific storage units or storage areas according to time. When performing dynamic rendering, the corresponding static information is retrieved from the corresponding storage units or storage areas in chronological order and presented continuously to achieve the presentation of dynamic scenes.

[0056] The dynamic scene reconstruction method based on Gaussian point clouds in this application stores time, parameters, and attributes in the form of sequences in an enhanced single PLY file, without the need to divide storage units or storage areas, making storage more flexible and convenient.

[0057] Furthermore, since an enhanced single-file stores various parameters and attributes of a group of Gaussian identities at different times within a specified period, a single enhanced single-file PLY can be used to dynamically present a group of Gaussian identities within a specified period without having to query or filter the relevant information of the Gaussian identity from the corresponding storage unit or storage area.

[0058] Furthermore, in most existing solutions, in order to utilize some established functions of 3D or 4D Gaussian models and reduce the information stored in individual PLY files, individual PLY files are usually bound to a certain 3D or 4D Gaussian model. This results in the need to build a corresponding 3D or 4D Gaussian model on the dynamic scene rendering end, which places higher demands on the hardware and software conditions of the dynamic scene rendering end and brings many limitations to the promotion and implementation of dynamic scene rendering.

[0059] In the dynamic scene reconstruction method based on Gaussian point clouds in this application, the parameter sequence and attribute sequence corresponding to the time-series keyframes stored in the enhanced single-entity PLY file can contain any parameters or attributes related to dynamic scene reproduction. Thus, all the information required for dynamic scene reproduction can be stored in the enhanced single-entity PLY file, eliminating the need to build corresponding 3D or 4D Gaussian models, greatly relaxing the hardware and software requirements for dynamic scene presentation, and making it easier to promote and implement.

[0060] Operation 140 compresses the parameter sequence and attribute sequence of the enhanced single-object PLY file to obtain the compressed enhanced single-object PLY file.

[0061] The parameter and attribute sequences corresponding to keyframes in a time series are usually huge within a certain period of time. Storing them in a single PLY file may result in an extremely large file that occupies a huge amount of storage space. Therefore, the dynamic scene reproduction method based on Gaussian point clouds in this application compresses the enhanced single PLY file to significantly reduce the file size and facilitate storage.

[0062] Unlike common methods that compress the entire file, this application's dynamic scene reconstruction method based on Gaussian point clouds compresses only the parameter and attribute sequences corresponding to time-series keyframes. For example, redundant data in the parameter and attribute sequences is removed through functional relationships. Thus, when parsing a parameter or attribute, it is unnecessary to expand the entire file; only the corresponding parameter or attribute sequence needs to be expanded, which significantly reduces the memory storage space required for dynamic scene rendering.

[0063] Through the above operations, the dynamic scene reconstruction method based on Gaussian point clouds in this application can unify the same Gaussian identity in each frame through Gaussian identity recognition codes, avoiding the loss of Gaussian identity and reproducing the motion trajectory of the Gaussian identity more accurately and reliably. Furthermore, storing the parameter sequence and attribute sequence corresponding to the time series keyframes of the same Gaussian identity within a specified time period in the same enhanced single-unit PLY file realizes the storage of the temporal correlation between 3D Gaussian data, laying a data foundation for subsequent dynamic scene presentation. Compressing the parameter sequence and attribute sequence of the enhanced single-unit PLY file not only significantly reduces the storage space requirements of the dynamic scene acquisition end but also reduces the video memory requirements of the dynamic scene presentation end. Thus, it is possible to perfectly realize the accurate reconstruction and presentation of real-world dynamic scenes in the virtual world, preserving as much detailed data as possible from frames, thereby greatly improving the user's visual experience. In addition, since the enhanced single-unit PLY file is used directly to store the parameters and attributes related to the time series keyframes, the hardware and software requirements of the dynamic scene presentation end are also greatly reduced, making the dynamic scene reconstruction method based on Gaussian point clouds in this application easier to implement and deploy.

[0064] It should be noted that operations 130 and 140 are separated into two operations for ease of description or functional division. In practical applications, they can also be combined into a single step and performed simultaneously. For example, after obtaining a certain attribute information of the Gaussian identity, it can be compressed, and the compressed value can be directly stored in the enhanced singleton PLY file, without first storing all attribute information in the enhanced singleton PLY file and then performing compression.

[0065] Figure 2 This application shows Figure 1 The illustrated embodiment describes the implementation process of the dynamic scene reconstruction method based on Gaussian point clouds at the scene rendering end. (Reference) Figure 2 The method includes:

[0066] Operation 210: Use the first GPU renderer to parse the compressed enhanced single-unit PLY file to obtain the Gaussian identification code, the parameter sequence and attribute sequence corresponding to the time series keyframes within the specified time period.

[0067] Because the enhanced single-unit PLY file used in this application embodiment includes Gaussian identification codes, parameter sequences corresponding to keyframes in the time series within a specified time period, and attribute sequences, existing GPU renderers do not understand the meaning and format of these data and therefore cannot identify and parse these attributes. Therefore, in this application embodiment, a dedicated GPU renderer, the first GPU renderer, was developed specifically for the compressed enhanced single-unit PLY file generated at the dynamic scene acquisition end. This first GPU renderer can run on standard GPU hardware to perform dynamic real-time rendering and achieve dynamic display effects.

[0068] Specifically, the first GPU renderer can parse the compressed enhanced single-unit PLY file according to the agreed-upon data meaning and format to obtain various information required for dynamic rendering.

[0069] For example, after loading an enhanced single-entity PLY file, the value of a specific attribute is read according to rendering needs. If the attribute value is a compressed value, an expansion operation is performed to obtain the original value of the attribute before compression, so as to obtain the required information.

[0070] Operation 220 uses the first GPU renderer to dynamically render the Gaussian identity based on the Gaussian identity identification code, the parameter sequence and attribute sequence corresponding to the key frames of the time series within a specified time period, and obtain a multi-view dynamic virtual scene.

[0071] In this process, the first GPU renderer organizes and sorts all the data for the Gaussian identity according to the time series, reading all parameters and attributes corresponding to a specific time series. Then, based on the sorted time series, it sequentially obtains all parameters and attributes corresponding to each point in time within that time series, performing real-time rendering to present the 3D scene at that point in time. Thus, once the 3D scene corresponding to each point in the time series is presented sequentially, the user can see a dynamic scene that changes over time.

[0072] In this embodiment of the application, the functions of data organization and sorting, as well as real-time rendering based on various parameters and attributes in the first GPU renderer, are also implemented through self-development.

[0073] Thus, in dynamic scene rendering, the Gaussian point cloud-based dynamic scene reconstruction method of this application eliminates the need to build a 3D or 4D Gaussian model. As long as the compressed PLY file is available and the first GPU renderer developed specifically for compressed PLY files is installed, dynamic scene reconstruction can be performed. This makes dynamic scene rendering exceptionally easy, with lower hardware and software requirements, making it easier to promote and implement.

[0074] It should be noted that, Figure 1 and Figure 2 The embodiments shown in this application are merely basic embodiments of the dynamic scene reconstruction method based on Gaussian point clouds. Implementers can refine and expand upon these embodiments according to specific implementation scenarios, conditions, and requirements.

[0075] Figure 3 This paper illustrates the implementation flow of a dynamic scene reconstruction method based on Gaussian point clouds, according to another embodiment of this application. This embodiment is applied to the digital preservation and teaching scenarios of intangible cultural heritage (ICH).

[0076] As we all know, there are many intangible cultural heritages with dynamic scenes and performances, such as dance, paper cutting and musical instrument playing. Not only are the movements complex, but the facial expressions of the characters and their positional relationships with the objects of the movements also need to be meticulously recorded.

[0077] There are also more requirements when digitally protecting intangible cultural heritage. For example, intangible cultural heritage performances / processes emphasize the verifiability of action details and object relationships, requiring spatiotemporal continuity, high fidelity, and low distortion dynamic recording; most are on-site demonstrations, making it difficult to deploy high-density sensors or motion capture markers, but to achieve a perfect presentation, stable reconstruction is still required even in the absence of markers; long-term archiving and reinterpretation (education, exhibition, VR reproduction) are emphasized, so format universality, readability, and transferability are particularly important; involving master-apprentice collaboration and temporal changes in artifacts and materials, it is necessary to focus on the fidelity of local actions and details and to magnify and replay local parts; the software and hardware resources available for dynamic scene acquisition and presentation ends, especially dynamic scene presentation ends, are limited.

[0078] Previously, some organizations attempted to preserve these intangible cultural heritages with dynamic scenes and performances using traditional methods, such as pictures, photos, or videos, but none of these methods could achieve a complete record and reproduction from all angles without blind spots.

[0079] Later, some organizations attempted to solve this problem by combining 3D scanning and reconstruction with motion capture. However, this approach is essentially "recreation / re-enactment," meaning that skeletal movement and surface models are first captured, and then the skin, clothing, and expressions are "reconstructed" using art techniques. In this process, the performer's core essence, spirit, and physical movement under specific actions are lost, ultimately resulting in only a CG simulation effect. This not only causes information loss but also easily leads to the "uncanny valley effect" in live-action 3D animation.

[0080] In recent years, the popular Gaussian splashing technology has been able to solve the problem of full-angle and complete recording and replication. However, existing dynamic Gaussian splashing schemes mostly describe motion through a deformable network. This method is difficult to handle topological changes and large-amplitude movements. It is only suitable for reconstructing dynamic scenes with simple movements and small changes, and cannot replicate more complex movements and fine-grained dynamic scenes such as subtle changes in human facial expressions.

[0081] In the above application scenarios, the dynamic scene reconstruction method based on Gaussian point clouds in this application can record and replicate the three-dimensional reconstruction information of intangible cultural heritage performances at different times frame by frame, and accurately reconstruct the motion trajectory through Gaussian identification codes, thereby realizing the full-angle accurate replication, recording and presentation of intangible cultural heritage, which can solve the above problems and become a feasible solution.

[0082] Furthermore, the compression mechanism for attribute and parameter sequences in the enhanced single-file PLY file significantly reduces the storage space required for frame-by-frame recording and replication of dynamic scenes, further increasing the feasibility of this technical solution. The lightweight application of dynamic scene rendering is also more suitable for widespread adoption and promotion in grassroots institutions such as schools, museums, and even communities.

[0083] Figure 3 This paper illustrates the main process of the dynamic scene reconstruction method based on Gaussian point clouds in the above-mentioned application of digital preservation of intangible cultural heritage with dynamic scenes and performance characteristics, as shown in one embodiment of this application.

[0084] refer to Figure 3 "The real-life scene of intangible cultural heritage performance" serves as the external input to the entire system. At the scene acquisition end, execution... Figure 3 The data processing flow shown on the left involves data collection and dynamic scene reconstruction of the "real scene of intangible cultural heritage performance," ultimately generating a compressed, enhanced single-unit PLY file, which is then stored in a file storage medium for use by the dynamic scene presentation end. The file storage medium can be any storage device used to store the enhanced single-unit PLY file, including network shared directories, cloud drives, or external hard drives.

[0085] Then, on the dynamic scene rendering end, execute Figure 3 The data processing flow shown on the right retrieves the compressed enhanced single-unit PLY file from the file storage medium and loads it into a dedicated GPU renderer. The dedicated GPU renderer then parses and dynamically renders the compressed enhanced PLY file, ultimately presenting a "virtual scene for demonstration and teaching" dynamically on the outside.

[0086] See Figure 3 On the left side, in this embodiment of the application, the data processing flow executed at the dynamic scene acquisition end includes:

[0087] Phase 1: Data Collection.

[0088] In this embodiment of the application, the data acquisition stage mainly involves using at least six synchronized cameras to acquire video from at least six different camera positions to obtain multi-view image sequences.

[0089] Specifically, in the actual scene of the intangible cultural heritage performance, 8 different positions were set up for filming, with 6 camera positions set up at each filming position (evenly distributed at the top, middle and bottom).

[0090] In this way, as many different image sequences as possible can be obtained, thus enabling full-angle presentation and reconstruction.

[0091] Second stage: Generation of four-dimensional Gaussian point clouds.

[0092] In this embodiment, the main operations performed during the four-dimensional Gaussian point cloud generation stage are to generate a Gaussian point cloud containing a time dimension and a Gaussian identification code based on a multi-view image sequence and a four-dimensional Gaussian model. For details of the specific operations, please refer to... Figure 4 Mainly includes:

[0093] Operation 410: Determine the camera parameters and sparse point cloud of at least six synchronized cameras based on the multi-view image sequence.

[0094] Specifically, SfM / COLMAP is used to calculate the intrinsic and extrinsic parameters of the cameras to obtain the parameters of each camera at each of the eight sets (six camera positions each) and generate a sparse point cloud to describe the sparse structure.

[0095] Operation 420: Based on camera parameters and sparse point cloud, train a four-dimensional Gaussian model and generate Gaussian point cloud frame by frame to obtain Gaussian point cloud containing the time dimension.

[0096] Specifically, the existing four-dimensional Gaussian model can be trained using the camera parameters and sparse point clouds from the eight different locations mentioned above. During training, a frame-by-frame optimization and generation method is used to generate Gaussian point clouds containing the time dimension, focusing on the time-varying Gaussian parameters.

[0097] Operation 430: Based on the nearest distance, appearance similarity and motion prior information, the Gaussian identity is matched in a Gaussian point cloud containing the time dimension within a continuous range of multiple frames to determine the Gaussian identity identification code.

[0098] Specifically, the possible corresponding points are found by "nearest distance judgment", and then the Gaussian identity code of each Gaussian identity is determined in multiple consecutive frames by the "global optimal matching" method, thereby maintaining the consistency of Gaussian identity in cross-frame data.

[0099] By explicitly assigning Gaussian identification codes in this embodiment, more accurate cross-frame tracking and trajectory reproduction can be achieved, which in turn enables better handling of non-rigid, large-amplitude real-world intangible cultural heritage performance movements.

[0100] Furthermore, in this embodiment, the Gaussian identity code for each Gaussian identity is determined by performing matching across multiple consecutive frames and combining appearance similarity (SH cosine similarity) and motion prior (constant velocity / constant acceleration). Compared to single-frame comparison, the Gaussian identity determined by the above method in this embodiment is more accurate, significantly reduces ID jumps, and the reconstructed trajectory is more reliable.

[0101] Operation 440: Add the Gaussian identity code corresponding to each Gaussian identity to the Gaussian point cloud containing the time dimension, and obtain the Gaussian point cloud containing the time dimension and the Gaussian identity code.

[0102] Specifically, add an attribute to the description file of the Gaussian point cloud to record the Gaussian identification code corresponding to the Gaussian point cloud.

[0103] Phase 3: Data integration and enhanced single-unit PLY file output.

[0104] In this embodiment, the main operations performed during the data integration and enhanced single-unit PLY file output stage are to integrate the Gaussian point cloud containing the time dimension and Gaussian identification code to generate the enhanced single-unit PLY file. For details, please refer to [link to specific operations]. Figure 5 Mainly includes:

[0105] Operation 510: Determine the world coordinates, and use the world coordinates to perform coordinate system transformation on the Gaussian point cloud containing the time dimension and Gaussian identification code, to obtain the Gaussian point cloud containing the time dimension and Gaussian identification code in coordinate system one.

[0106] Specifically, world coordinates are selected based on the reconstructed coordinates of SfM / RC. Rigid 3D spatial transformations (rotation and translation) are performed on the Gaussian point clouds corresponding to each camera to achieve registration. When performing rotational transformations on the Gaussian ellipsoids or point volumes, rigorous mathematical transformations of their covariance matrix are required to ensure their shape remains correct after registration. This is a crucial detail for ensuring the quality of multi-source data fusion.

[0107] Operation 520: Based on the time dimension, perform time alignment on the Gaussian point cloud of coordinate system one, which includes the time dimension and Gaussian identification code, to obtain a time-aligned Gaussian point cloud of coordinate system one that includes the time dimension and Gaussian identification code.

[0108] Specifically, time synchronization is performed using the capture timestamps or audio pulses of each frame to unify timestamps with errors within a certain range into a single time point, thus constructing a global time series [t0, tN]. For example, timelines are aligned using features and key actions, with the starting dance action as the reference time t0, the next time interval as time t1, and so on. If a Gaussian point cloud for a certain time point is missing, resampling or interpolation can be performed as necessary.

[0109] Operation 530: Based on the Gaussian identification code and time dimension, merge and deduplicate the trajectories of the Gaussian point cloud containing the time dimension and Gaussian identification code that are time-aligned and have a unified coordinate system, and unify the attributes and parameters to obtain an enhanced single PLY file containing the complete spatiotemporal trajectory of the Gaussian point cloud.

[0110] Specifically, this includes: merging duplicate Gaussian point cloud data based on Gaussian identification codes or similarity; splicing multi-source segments with the same Gaussian identity, for example, merging cross-source (from different camera positions) trajectories based on the consistency of motion trajectory and color SH; and unifying the order of SH and the method of covariance parameterization.

[0111] Then, Gaussian point cloud data from different shooting positions and angles are stored in blocks, and a block index and global directory are built for use when presenting dynamic scenes from all angles.

[0112] The fourth stage involves enhanced single-unit PLY file compression.

[0113] In this embodiment, the compression stage of the enhanced singleton PLY file mainly involves compressing the parameter sequence and attribute sequence to obtain the compressed enhanced singleton PLY file. For details, please refer to [link to specific operations]. Figure 6 Mainly includes:

[0114] Operation 610 determines a set of key coefficients for frequency domain transform compression based on the motion trajectory and / or frequency characteristics of each Gaussian identity.

[0115] Frequency domain transform compression primarily refers to the technique of transforming the content to be compressed, represented by digital signals, from the time domain to the frequency domain, and then utilizing the sparsity characteristics of the signal in the frequency domain for data compression. Its core idea is to remove redundant frequency components and retain key features, thereby reducing storage and transmission burden. Frequency domain transform compression mainly includes: time-frequency analysis, decomposing the signal into a superposition of different frequency components to form a frequency domain matrix; sparsity processing, eliminating redundant coefficients with small amplitudes through threshold filtering or encoding methods, retaining only important features; and multi-dimensional compression: performing joint compression in the time, frequency, or time-frequency plane structural dimensions to further reduce data volume, among other key processing steps.

[0116] Specifically, the motion trajectory and / or frequency characteristics of each Gaussian identity are obtained; if the motion trajectory and / or frequency characteristics indicate that the Gaussian identity is a smooth background area, a set of key coefficients with a low compression ratio is used to retain as few key frames as possible to reduce storage space; if the motion trajectory and / or frequency characteristics indicate that the Gaussian identity is a high-speed or complex action, a set of key coefficients with a high compression ratio is used to retain as many key frames as possible to reduce action omissions or missing frames.

[0117] In this way, a better balance can be achieved between compression ratio and fidelity.

[0118] In one embodiment of this application, Fourier compression is used as the method for frequency domain transform compression, and the number of Fourier terms is used as the key coefficients. In other embodiments of this application, any other frequency domain transform compression method may be used.

[0119] Operation 620 uses a set of key coefficients to perform frequency domain transformation compression on the parameter and attribute sequences of the enhanced single-unit PLY file.

[0120] In this embodiment, frequency domain transformation compression is performed on the parameter and attribute sequences of the enhanced single-unit PLY file. A set of key coefficients from the frequency domain transformation compression records the functional relationships between each parameter or attribute in the parameter and attribute sequences corresponding to the key frames of the time series, while the specific numerical values ​​that can be obtained from these functional relationships are deleted (i.e., redundant data is removed). Then, during presentation, only the corresponding data values ​​need to be calculated based on this set of key coefficients. This significantly reduces the amount of data that needs to be stored.

[0121] Operation 630 saves a set of key coefficients in the enhanced single-unit PLY file to obtain the compressed enhanced single-unit PLY file.

[0122] The following is an exemplary fragment (attribute definition) of the enhanced singleton PLY file generated in the embodiments of this application:

[0123] ...

[0124] format binary_little_endian 1.0

[0125] comment dply_version 1.0

[0126] comment time_range 0.0 12.0# seconds

[0127] comment fps_native 60# Frames per second

[0128] Comment license cc-by-nc 4.0

[0129] comment capture_info site="Mentougou" project="ICH-xxx"

[0130] comment element "gaussian" is a custom type for dynamic 3D Gaussianprimitives

[0131] How many Gaussian point clouds are there in element Gaussian N?

[0132] property float x

[0133] property float y

[0134] property float z

[0135] property float qx# quaternion (unit)

[0136] property float qy

[0137] property float qz

[0138] property float qw

[0139] property float sx

[0140] property float sy

[0141] property float sz

[0142] property float opacity #Opacity

[0143] multiple property uint track_id

[0144] property uchar t_scheme# 0=keyframe,1=fourier,2=mixed

[0145] property list uint float t_key_times # Time series keyframes

[0146] property list uint float t_key_params # Packed Δ parameters

[0147] property ushort fbc_K# Fourier terms

[0148] property list uint float fbc_ax

[0149] property list uint float fbc_bx # Store each property separately

[0150] property list uchar float sh# Spherical Harmony Coefficient

[0151] ...

[0152] Wherein, time_range is the time range of the Gaussian point cloud data stored in the enhanced single-unit PLY file, which is the time period length specified by each enhanced single-unit PLY file; t_key_times are the time series keyframes; t_key_params and sh are the parameter sequences corresponding to the time series keyframes; float fbc_ax and floatfbc_bx are the attribute sequences corresponding to the time series keyframes; and fbc_K is the number of Fourier terms used for compression.

[0153] In this way, a compressed, enhanced single-unit PLY file can be generated, storing the Gaussian identity code, time-series keyframes, Fourier terms, compressed parameters and attributes corresponding to the time-series keyframes, and the Gaussian identity's location information all in one file. Thus, during dynamic rendering of a dynamic scene, only this enhanced single-unit PLY file is needed for dynamic rendering and presentation, without the need to reconstruct a 3D or 4D Gaussian model or rely on any third-party decoding tools.

[0154] In one embodiment of this application, a mode attribute, such as "uchar t_scheme", can be set to identify whether the enhanced monolithic PLY file uses keyframes, Fourier transforms, or a hybrid approach. This allows for flexible determination of which method to use to store the Gaussian point cloud based on different time periods, actions, or the characteristics of the foreground and background. For example, for backgrounds with little variation, storing only a small number of keyframes can be used; for more complex actions, frame-by-frame storage combined with Fourier transform compression can be employed; and for scenarios with both simple backgrounds and complex actions, a hybrid approach can be used. This results in higher compression efficiency while balancing storage space and image quality.

[0155] In one embodiment of this application, traceable metadata, such as shooting time, location, and licensing information, can be added to the enhanced single-unit PLY. This allows the digitized information of intangible cultural heritage to have a certain degree of traceability.

[0156] The above is Figure 3 The data processing flow executed at the dynamic scene acquisition end in the embodiment of this application is shown.

[0157] Next, return to Figure 3 The embodiments shown below illustrate the data processing flow executed at the dynamic scene presentation end. See also... Figure 3 On the right side, the data processing flow executed at the dynamic scene presentation end in this application embodiment mainly includes:

[0158] Phase 1: Parsing the compressed, enhanced single-unit PLY file.

[0159] In this embodiment, the main operation performed during the compressed file parsing stage is to use the first GPU renderer to parse the compressed enhanced single-unit PLY file, obtain the Gaussian identification code, the parameter sequence and attribute sequence corresponding to the time series keyframes within a specified time period. For details of the operation, please refer to [link to specific operations]. Figure 7 Mainly includes:

[0160] Operation 710 writes commonly used sine / cosine frequency values ​​to the fast table.

[0161] These commonly used sine / cosine frequency values ​​are mainly used to expand parameter sequences and attribute sequences compressed using frequency domain transform.

[0162] Operation 720 loads the TLB into the high-speed constant area of ​​the video memory;

[0163] Operation 730: Query the TLB to obtain the frequency values ​​required to parse the enhanced single-unit PLY file;

[0164] Operation 740 uses the first GPU renderer to parse the compressed, enhanced single-unit PLY file based on the frequency value.

[0165] The first GPU renderer also receives an attribute and parameter definition file identical to the aforementioned enhanced single-unit PLY file, enabling it to understand the meaning of each attribute and parameter in the compressed enhanced single-unit PLY file (especially the extended attribute part). Furthermore, the first GPU renderer has a decompression function, which can expand the attribute and parameter sequences based on a set of key coefficients to obtain the complete sequence before compression, for use in subsequent rendering and dynamic scene presentation.

[0166] Typically, when expanding attributes compressed using Fourier transform, the sine / cosine must be calculated online each time to reconstruct the spectrum, which is very time-consuming. However, in this embodiment, commonly used frequency values ​​are pre-calculated and placed in a fast table for all threads within the GPU to share. A lookup table (LUT) operation is used instead of real-time calculation. These tables are then placed in the high-speed constant area of ​​the video memory, which can significantly shorten the calculation time and greatly improve the parsing efficiency.

[0167] Furthermore, in this embodiment, frequently used Gaussian point cloud data is cached to further accelerate access to repetitive data. For Gaussian point cloud data from different angles, a storage block combined with an index can be used to achieve on-demand loading, further improving the parsing efficiency of enhanced single-file PLY parsing.

[0168] Phase Two: Dynamic Scene Rendering.

[0169] In this embodiment, the dynamic scene rendering stage mainly uses the first GPU renderer to dynamically render the Gaussian identity based on the Gaussian identity identification code, the parameter sequence and attribute sequence corresponding to the Gaussian identity within a specified time period, and obtain related operations for a multi-view dynamic virtual scene.

[0170] Specifically, the Gaussian point cloud data at each time point of the specified Gaussian identity are put into the rendering pipeline in chronological order. A dedicated renderer retrieves the 3D Gaussian point cloud data to be rendered at the next moment from the rendering pipeline, and renders and presents it until all the data to be rendered is rendered.

[0171] Since Gaussian point clouds can be rendered in real time using Gaussian splashing technology, in this embodiment, the display angle can be adjusted, the image can be zoomed in or out, and playback and fast forward operations can be performed based on user interaction, making it more convenient for the observation and teaching of intangible cultural heritage performances.

[0172] In conclusion, Figures 3 to 7The embodiments of this application, when using the dynamic scene reconstruction method based on Gaussian point clouds for the digital protection of performative intangible cultural heritage, can significantly reduce the size of a single 3D Gaussian file; enable multiple frame-by-frame 3D Gaussian point clouds to be presented continuously and smoothly like a video; and ultimately achieve real-time, multi-angle, and freely viewable dynamic rendering display.

[0173] Furthermore, the addition of traceable Gaussian identification codes in cross-frame data makes the reproduction of motion trajectories more accurate, thus preserving and replicating the essence, spirit, and energy of each movement in intangible cultural heritage performances. This has extremely high social significance and promotional value for the digital protection project of intangible cultural heritage performances.

[0174] It should be noted that, Figures 1 to 7 The embodiments shown in this application are merely illustrative examples of the dynamic scene reconstruction method based on Gaussian point clouds and are not intended to limit the implementation methods or application scenarios of the dynamic scene reconstruction method based on Gaussian point clouds. Implementers can adopt any applicable implementation method and apply it to any applicable application scenario according to specific needs and actual implementation conditions.

[0175] According to a third aspect of the embodiments of this application, the embodiments of this application also provide an apparatus for dynamic scene reconstruction based on Gaussian point clouds at the scene acquisition end. For example... Figure 8 As shown, the device 80 includes: a video acquisition module 801, used to acquire video from at least six different camera positions using at least six synchronized cameras to obtain a multi-view image sequence; a Gaussian point cloud generation module 802, used to generate a Gaussian point cloud containing a time dimension and a Gaussian identification code based on the multi-view image sequence and a four-dimensional Gaussian model; a single-file generation module 803, used to integrate the Gaussian point cloud containing the time dimension and the Gaussian identification code to generate an enhanced single-file PLY file, the enhanced single-file PLY file containing a Gaussian identification code, a parameter sequence and an attribute sequence corresponding to the keyframes of the time series within a specified time period; and a compression module 804, used to compress the parameter sequence and attribute sequence of the enhanced single-file PLY file to obtain a compressed enhanced single-file PLY file.

[0176] According to one embodiment of this application, the Gaussian point cloud generation module 802 includes: a camera parameter and sparse point cloud determination submodule, used to determine the camera parameters and sparse point clouds of at least six synchronized cameras based on a multi-view image sequence; a Gaussian point cloud generation submodule, used to train a four-dimensional Gaussian model based on the camera parameters and sparse point clouds, and generate Gaussian point clouds frame by frame to obtain Gaussian point clouds containing a time dimension; a Gaussian identity code determination submodule, used to match Gaussian identities in the Gaussian point clouds containing a time dimension within a continuous range of multiple frames based on nearest distance, appearance similarity, and motion prior information, and determine the Gaussian identity code corresponding to each Gaussian identity; and a Gaussian identity code addition submodule, used to add the Gaussian identity code corresponding to each Gaussian identity to the Gaussian point clouds containing a time dimension to obtain Gaussian point clouds containing a time dimension and Gaussian identity codes.

[0177] According to an embodiment of this application, the single-unit file generation module 803 includes: a coordinate system one submodule, used to determine world coordinates and use the world coordinates to perform coordinate system transformation on the Gaussian point cloud containing the time dimension and Gaussian identification code, to obtain a Gaussian point cloud containing the time dimension and Gaussian identification code in coordinate system one; a time alignment submodule, used to perform time alignment on the Gaussian point cloud containing the time dimension and Gaussian identification code in coordinate system one according to the time dimension, to obtain a time-aligned Gaussian point cloud containing the time dimension and Gaussian identification code in coordinate system one; and a data integration and enhanced single-unit PLY file generation submodule, used to merge and deduplicate trajectories and unify attributes and parameters on the time-aligned Gaussian point cloud containing the time dimension and Gaussian identification code in coordinate system one according to the Gaussian identification code and the time dimension, so as to obtain an enhanced single-unit PLY file containing the complete spatiotemporal trajectory of the Gaussian point cloud.

[0178] According to one embodiment of this application, the compression module 804 includes: a key coefficient determination submodule, used to determine a set of key coefficients for frequency domain transformation compression based on the motion trajectory and / or frequency characteristics of each Gaussian identity; a parameter compression submodule, used to perform frequency domain transformation compression on the parameter sequence and attribute sequence of the enhanced single-unit PLY file using the set of key coefficients; and a key coefficient storage submodule, used to store a set of key coefficients in the enhanced single-unit PLY file to obtain the compressed enhanced single-unit PLY file.

[0179] According to an embodiment of this application, the Fourier term determination submodule includes: a feature acquisition unit, used to acquire the motion trajectory and / or frequency characteristics of each Gaussian identity; and a key coefficient determination unit, specifically used to use a set of key coefficients with a lower compression ratio if the motion trajectory and / or frequency characteristics indicate that the Gaussian identity is a smooth background area; and to use a set of key coefficients with a higher compression ratio if the motion trajectory and / or frequency characteristics indicate that the Gaussian identity is a high-speed or complex action.

[0180] According to a fourth aspect of the embodiments of this application, the embodiments of this application also provide an apparatus for dynamic scene reconstruction based on Gaussian point clouds used at the scene rendering end. For example... Figure 9 As shown, the device 90 includes: a parsing module 901, used to parse the compressed enhanced single-unit PLY file using a first GPU renderer to obtain a Gaussian identity code, a parameter sequence and an attribute sequence corresponding to key frames of the time series within a specified time period; and a dynamic rendering module 902, used by the first GPU renderer to dynamically render the Gaussian identity based on the Gaussian identity code, the parameter sequence and attribute sequence corresponding to the Gaussian identity to key frames of the time series within a specified time period, to obtain a multi-view dynamic virtual scene.

[0181] According to one embodiment of this application, the parsing module 901 includes: a TLB writing submodule for writing commonly used sine / cosine frequency values ​​into the TLB; a video memory loading submodule for loading the TLB into the high-speed constant area of ​​the video memory; a frequency value query submodule for querying from the TLB to obtain the frequency values ​​required for parsing the enhanced single-unit PLY file; and a parsing submodule for parsing the compressed enhanced single-unit PLY file using a first GPU renderer based on the frequency values.

[0182] In one embodiment of this application, the various modules of the aforementioned dynamic scene reconstruction device based on Gaussian point clouds are independently developed, tested, and maintained by different teams. The modules interact and communicate with each other based on a predefined data format, exhibiting plug-and-play characteristics. This significantly simplifies development and maintenance costs and facilitates iteration, upgrades, and expansion.

[0183] According to a fifth aspect of the present application, a computer storage medium is provided, the storage medium including a set of computer-executable instructions, which, when executed, are used to perform any of the above-described dynamic scene reconstruction methods based on Gaussian point clouds.

[0184] It should be noted that the above descriptions of the dynamic scene reconstruction device based on Gaussian point clouds and the embodiment of the computer storage medium are similar to the descriptions of the foregoing method embodiments and have similar beneficial effects, therefore, they will not be repeated. For technical details not disclosed in the descriptions of the embodiments of the dynamic scene reconstruction device based on Gaussian point clouds and the embodiment of the computer storage medium in this application, please refer to the descriptions of the foregoing method embodiments of this application for understanding; to save space, they will not be repeated here.

[0185] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0186] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components may be combined, or integrated into another device, or some features may be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0187] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0188] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be set as a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0189] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage media, read-only memory (ROM), magnetic disks, or optical disks.

[0190] Alternatively, if the integrated units described above are implemented as software functional modules and set up as independent products for sale or use, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage media, ROM, magnetic disks, or optical disks.

[0191] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A dynamic scene reproduction method based on Gaussian point clouds, applied to a scene acquisition terminal, characterized in that, The method includes: Use at least six simultaneous cameras to capture video from at least six different camera positions to obtain a multi-view image sequence; Based on the multi-view image sequence and the four-dimensional Gaussian model, a Gaussian point cloud containing a time dimension and a Gaussian identification code is generated. The Gaussian point cloud containing the time dimension and Gaussian identification code is integrated to generate an enhanced single-unit PLY file. The enhanced single-unit PLY file contains the Gaussian identification code, the parameter sequence and attribute sequence corresponding to the time series keyframes within a specified time period. The parameter sequence and attribute sequence of the enhanced single-unit PLY file are compressed to obtain the compressed enhanced single-unit PLY file; The step of generating a Gaussian point cloud containing a temporal dimension and a Gaussian identity code based on the multi-view image sequence and the four-dimensional Gaussian model includes: determining the camera parameters and sparse point cloud of at least six synchronized cameras based on the multi-view image sequence; training the four-dimensional Gaussian model based on the camera parameters and the sparse point cloud to generate Gaussian point clouds frame by frame, obtaining Gaussian point clouds containing a temporal dimension; matching Gaussian identities in the Gaussian point cloud containing a temporal dimension within a continuous range of multiple frames based on nearest distance, appearance similarity, and motion prior information, determining the Gaussian identity code corresponding to each Gaussian identity; and adding the Gaussian identity code corresponding to each Gaussian identity to the Gaussian point cloud containing a temporal dimension, obtaining a Gaussian point cloud containing both a temporal dimension and a Gaussian identity code. The step of compressing the parameter sequence and attribute sequence of the enhanced single-unit PLY file to obtain the compressed enhanced single-unit PLY file includes: determining a set of key coefficients for frequency domain transformation compression based on the motion trajectory and / or frequency characteristics of each Gaussian identity; performing frequency domain transformation compression on the parameter sequence and attribute sequence of the enhanced single-unit PLY file based on the set of key coefficients; and storing the set of key coefficients in the enhanced single-unit PLY file to obtain the compressed enhanced single-unit PLY file.

2. The method according to claim 1, characterized in that, The process of integrating the Gaussian point cloud containing the time dimension and Gaussian identification code to generate an enhanced monolithic PLY file includes: Determine the world coordinates, and use the world coordinates to perform coordinate system transformation on the Gaussian point cloud containing the time dimension and Gaussian identification code to obtain a Gaussian point cloud containing the time dimension and Gaussian identification code in coordinate system one. Based on the time dimension, the Gaussian point cloud of coordinate system one, which includes the time dimension and Gaussian identification code, is time-aligned to obtain a time-aligned Gaussian point cloud of coordinate system one that includes the time dimension and Gaussian identification code. Based on the Gaussian identification code and the time dimension, the trajectory of the time-aligned and coordinate system-unified Gaussian point cloud containing the time dimension and the Gaussian identification code is merged and deduplicated, and the attributes and parameters are unified to obtain an enhanced single PLY file containing the complete spatiotemporal trajectory of the Gaussian point cloud.

3. The method according to claim 1, characterized in that, The determination of a set of key coefficients for frequency domain transform compression based on the motion trajectory and / or frequency characteristics of each Gaussian identity includes: Obtain the motion trajectory and / or frequency characteristics of each Gaussian identity; If the motion trajectory and / or frequency characteristics indicate that the Gaussian identity is a smooth background region, then a set of key coefficients with a lower compression ratio is used; If the motion trajectory and / or frequency characteristics indicate that the Gaussian identity is a high-speed or complex motion, then a set of key coefficients with a higher compression ratio is used.

4. The method according to claim 1, characterized in that, The enhanced single-unit PLY file also includes traceable metadata.

5. A dynamic scene reproduction method based on Gaussian point clouds, applied to a scene presentation end, wherein the scene presentation end corresponds to the scene acquisition end described in claim 1, characterized in that, The method includes: The compressed, enhanced single-unit PLY file is parsed using the first GPU renderer to obtain the Gaussian identification code, the parameter sequence and attribute sequence corresponding to the time series keyframes within the specified time period; Using the first GPU renderer, the Gaussian identity is dynamically rendered based on the Gaussian identity identification code, the parameter sequence and attribute sequence corresponding to the key frames of the time series within a specified time period, to obtain a multi-view dynamic virtual scene.

6. The method according to claim 5, characterized in that, The step of parsing the compressed, enhanced single-unit PLY file using the first GPU renderer includes: Write commonly used sine / cosine frequency values ​​into the fast table; Load the TLB into the high-speed constant area of ​​the video memory; The frequency values ​​required to parse the enhanced single-unit PLY file are obtained by querying the fast table. Based on the frequency value, the compressed enhanced single-unit PLY file is parsed using a first GPU renderer.

7. A dynamic scene reproduction device based on Gaussian point clouds, applied at a scene acquisition end, characterized in that, The device includes: The video acquisition module is used to acquire video from at least six different camera positions using at least six synchronized cameras to obtain a multi-view image sequence; The Gaussian point cloud generation module is used to generate a Gaussian point cloud containing a time dimension and a Gaussian identification code based on the multi-view image sequence and the four-dimensional Gaussian model. The single-unit file generation module is used to integrate the Gaussian point cloud containing the time dimension and Gaussian identification code to generate an enhanced single-unit PLY file. The enhanced single-unit PLY file contains the Gaussian identification code, a parameter sequence and an attribute sequence corresponding to the time series keyframes within a specified time period. A compression module is used to compress the parameter sequence and attribute sequence of the enhanced single-unit PLY file to obtain a compressed enhanced single-unit PLY file. The Gaussian point cloud generation module includes: a camera parameter and sparse point cloud determination submodule, used to determine the camera parameters and sparse point clouds of at least six synchronized cameras based on a multi-view image sequence; a Gaussian point cloud generation submodule, used to train a four-dimensional Gaussian model based on the camera parameters and sparse point clouds, and generate Gaussian point clouds frame by frame to obtain Gaussian point clouds containing the time dimension; a Gaussian identity code determination submodule, used to match Gaussian identities in the Gaussian point clouds containing the time dimension within a continuous range of multiple frames based on nearest distance, appearance similarity, and motion prior information, and determine the Gaussian identity code corresponding to each Gaussian identity; and a Gaussian identity code addition submodule, used to add the Gaussian identity code corresponding to each Gaussian identity to the Gaussian point clouds containing the time dimension, to obtain Gaussian point clouds containing both the time dimension and the Gaussian identity code. The compression module includes: a key coefficient determination submodule, used to determine a set of key coefficients for frequency domain transformation compression based on the motion trajectory and / or frequency characteristics of each Gaussian identity; a parameter compression submodule, used to perform frequency domain transformation compression on the parameter sequence and attribute sequence of the enhanced single-unit PLY file using a set of key coefficients; and a key coefficient saving submodule, used to save a set of key coefficients in the enhanced single-unit PLY file to obtain the compressed enhanced single-unit PLY file.

8. A dynamic scene reproduction device based on Gaussian point clouds, applied to a scene presentation end, wherein the scene presentation end corresponds to the scene acquisition end described in claim 7, characterized in that, The device includes: The parsing module is used to parse the compressed enhanced single PLY file using the first GPU renderer to obtain the Gaussian identification code, the parameter sequence and attribute sequence corresponding to the time series keyframes within the specified time period; The dynamic rendering module uses the first GPU renderer to dynamically render the Gaussian identity based on the Gaussian identity identification code, the parameter sequence and attribute sequence corresponding to the Gaussian identity and the time series keyframes within a specified time period, thereby obtaining a multi-view dynamic virtual scene.

Images