Method for transmitting dynamic three-dimensional scene and electronic device

Abstract

The embodiment of the specification discloses a dynamic three-dimensional scene transmission method and electronic equipment, and relates to the technical field of three-dimensional scene data transmission. In view of the technical problems that there is a contradiction between large dynamic three-dimensional scene data and network fluctuation, there is a lack of incremental quality refinement capability, and the existing layered method is difficult to effectively expand to a dynamic scene, the embodiment comprises the following steps: calculating the importance score of each Gaussian primitive in the pre-trained dynamic three-dimensional scene model; dividing the model into a basic layer and multiple enhancement layers based on the importance score and storing them independently; in response to a transmission request, determining the receiving bandwidth of the client and deciding at least one target layer to be transmitted according to the receiving bandwidth, and sequentially transmitting the at least one target layer to the client. The embodiment of the specification can be applied to volumetric video streaming scenarios, can realize low startup delay transmission and step-by-step quality improvement of dynamic three-dimensional scenes, and can ensure full reuse of transmitted data.

Description

Technical Field

[0001] This specification relates to one or more embodiments in the field of artificial intelligence, and more particularly to a method and electronic device for transmitting dynamic three-dimensional scenes. Background Technology

[0002] With the increasing demand for immersive media experiences, applications such as free-viewpoint video, telepresence, and extended reality are placing increasingly higher demands on the real-time rendering and efficient transmission of dynamic 3D scenes. Among numerous scene representation methods, 3D Gaussian splashing technology and its dynamic extension scheme have become a highly promising technical approach due to their ability to achieve real-time rendering of over 100 frames per second and high visual fidelity through explicit Gaussian primitive representation.

[0003] However, there is still a lack of mature solutions for efficiently transmitting such dynamic 3D scene content to end users. Summary of the Invention

[0004] This specification aims to disclose a progressive transmission technology solution for dynamic 3D scenes. Its core concept is to sort the Gaussian primitives in the pre-trained dynamic 3D scene model according to their importance to rendering quality, divide them into a base layer and multiple enhancement layers, and store them separately. During transmission, the target layer to be transmitted is adaptively determined according to the client's receiving bandwidth and buffer status, so that the client can start visualization rendering immediately after receiving the base layer, and gradually improve the rendering effect when the enhancement layers are received subsequently.

[0005] According to a first aspect of the embodiments of this specification, a method for transmitting a dynamic three-dimensional scene is provided, the dynamic three-dimensional scene including a three-dimensional scene model constructed based on Gaussian elements that dynamically change in the time dimension; the method includes: Calculate the importance score of each Gaussian primitive in the pre-trained dynamic 3D scene; wherein the importance score represents the degree of importance of the Gaussian primitive to the rendering quality of the dynamic 3D scene; Based on the importance score, the dynamic 3D scene is divided into a base layer and multiple enhancement layers, and the base layer and the multiple enhancement layers are stored as independent transportable units. The base layer contains the Gaussian elements with the highest importance scores in the dynamic 3D scene; the multiple enhancement layers contain the remaining Gaussian elements in the dynamic 3D scene excluding the aforementioned Gaussian elements; the importance scores of the Gaussian elements in the multiple enhancement layers decrease sequentially. In response to a request to transmit a dynamic 3D scene to a client, the receiving bandwidth of the client is determined, and based on the receiving bandwidth, a target dynamic 3D scene to be transmitted is determined from the stored dynamic 3D scenes. A base layer and multiple enhancement layers contained in the target dynamic 3D scene are sequentially transmitted to the client, so that the client performs preliminary visualization rendering of the target dynamic 3D scene based on the Gaussian elements contained in the received base layer, obtaining an initial display effect corresponding to the target dynamic 3D scene. Upon receiving subsequent transmissions of enhancement layers, the client further performs visualization rendering of the target dynamic 3D scene based on the Gaussian elements contained in the enhancement layers, obtaining a display effect that enhances the initial display effect based on the enhancement layers.

[0006] Optionally, the target dynamic 3D scene, comprising a base layer and multiple enhancement layers, is sequentially transmitted to the client, including: The target dynamic 3D scene, consisting of a base layer and multiple enhancement layers, is transmitted sequentially to the client according to a progressive transmission strategy that prioritizes the transmission of the base layer.

[0007] Optionally, the importance score corresponding to any Gaussian element includes a score calculated based on the gradient magnitude of the rendering loss corresponding to each Gaussian element generated during the pre-training process of the dynamic 3D scene. The importance score of each Gaussian primitive in the pre-trained dynamic 3D scene is calculated, including: for each Gaussian primitive in the pre-trained dynamic 3D scene, the sum of the gradient magnitudes of the rendering loss generated during the pre-training process of the dynamic 3D scene is calculated for each of the multiple sets of Gaussian parameters of the Gaussian primitive, and the original gradient magnitude of the Gaussian primitive on each set of Gaussian parameters is obtained. The original gradient magnitude of the Gaussian element on each set of Gaussian parameters is normalized to obtain the normalized gradient magnitude of the Gaussian element on each set of Gaussian parameters. The importance score corresponding to the Gaussian element is obtained by summing the normalized gradients of the Gaussian parameters of all groups.

[0008] Optionally, the base layer contains a preset initial number of Gaussian elements; the number of Gaussian elements in the multi-level enhancement layers increases sequentially according to a preset coefficient based on the initial number; the dynamic 3D scene is divided into one base layer and multiple enhancement layers based on the importance score, including: Obtain the initial quantity preset for the base layer; The first number of Gaussian elements contained in the first-level enhancement layer is calculated based on the initial number, and the Gaussian elements with the lowest importance scores are stripped from all Gaussian elements in the dynamic 3D scene according to the first number, and the first-level enhancement layer is created based on the stripped Gaussian elements. Based on the first quantity, the second quantity of Gaussian elements contained in the second-level enhancement layer is calculated, and the Gaussian elements with the lowest importance scores are stripped from the remaining Gaussian elements in the dynamic 3D scene according to the second quantity, and the second-level enhancement layer is created based on the stripped Gaussian elements. This process continues until the number of enhancement layers reaches a specified number, at which point the base layer is created based on the remaining Gaussian elements in the dynamic 3D scene.

[0009] Optionally, after creating any level of enhancement layer, the importance scores of the remaining Gaussian elements in the dynamic 3D scene are recalculated.

[0010] Optionally, the number of Gaussian elements contained in the multiple enhancement layers increases exponentially based on the initial number according to a preset coefficient; the preset coefficient is a coefficient calculated based on the total number of Gaussian elements in the dynamic 3D scene, the specified number preset for the enhancement layers, and the initial number.

[0011] Optionally, the multiple sets of Gaussian parameters include spatial opacity parameters; the method further includes: after dividing the dynamic 3D scene into a base layer and multiple levels of enhancement layers, further training is performed on the divided base layer and each level of enhancement layer; The further training of the base layer and each level of the enhancement layer includes: The base layer is trained by using all the Gaussian parameters of each Gaussian unit contained in the base layer as trainable parameters. After training of the base layer is completed, the Gaussian parameters of each Gaussian element in the base layer, except for the spatial opacity parameter, are frozen. The spatial opacity parameter of each Gaussian element in the base layer and the full set of Gaussian parameters of each Gaussian element in the first level of the multi-level enhancement layer are used as trainable parameters, and the base layer and the first level enhancement layer are jointly trained. After the joint training of the base layer and the first-level enhancement layer is completed, the Gaussian parameters of each Gaussian element in the base layer, except for the spatial opacity parameter, are frozen. The spatial opacity parameter of each Gaussian element in the base layer and the full set of Gaussian parameters of each Gaussian element in the first and second-level enhancement layers in the multi-level enhancement layers are used as trainable parameters, and the base layer, the first-level enhancement layer and the second-level enhancement layer are jointly trained. This process continues until all enhancement layers at each level have been trained.

[0012] Optionally, the method further includes: After joint training of any level of enhancement layer and the base layer is completed, the updated spatial opacity parameter obtained through training is obtained; the obtained updated spatial opacity parameter is saved as data to be transmitted to the enhancement layer, so that when the client performs further visualization rendering of the target dynamic 3D scene based on the Gaussian meta-generators contained in the enhancement layer, it can perform visualization rendering of the target dynamic 3D scene based on the updated spatial opacity parameter saved in the enhancement layer.

[0013] Optionally, before storing the base layer and the multi-level enhancement layers as independent transferable units, the method further includes: The Gaussian parameters of each Gaussian element in the base layer and the multi-level enhancement layer are quantized and compressed using a differentiated quantization method; wherein the degree of quantization compression of at least some of the Gaussian parameters of each Gaussian element in the enhancement layer is higher than the degree of quantization compression of the Gaussian parameters of each Gaussian element in the base layer; and entropy encoding is performed on the quantized and compressed base layer and the multi-level enhancement layer.

[0014] Optionally, before quantizing and compressing the Gaussian parameters of each Gaussian element contained in the base layer and the multi-level enhancement layers using a differentiated quantization method, the method further includes: Discard the last or more enhancement layers in the multi-level enhancement layers, and retain the base layer and the remaining enhancement layers at each level.

[0015] Optionally, determining the client's receiving bandwidth includes: Obtain the most recent multiple receive rate samples reported by the client; The harmonic mean of the multiple received rate samples is calculated as an estimate of the received bandwidth of the client.

[0016] Optionally, based on the received bandwidth, the target dynamic 3D scene to be transmitted is determined from the stored dynamic 3D scenes, and the base layer and multiple enhancement layers contained in the target dynamic 3D scene are sequentially transmitted to the client according to a progressive transmission strategy that prioritizes the transmission of the base layer, including: Obtain the remaining space in the cache reported by the client; Based on the received bandwidth and the remaining space, the target dynamic 3D scene to be transmitted is determined from the stored dynamic 3D scene. From the base layer and multiple enhancement layers contained in the target dynamic 3D scene, at least one target layer to be transmitted is adaptively decided for the client. According to the progressive transmission strategy of prioritizing the transmission of the base layer, the at least one target layer is transmitted to the client as an independent transmittable unit in sequence.

[0017] Optionally, based on the received bandwidth, a target dynamic 3D scene to be transmitted is determined from the stored dynamic 3D scenes. From the base layer and multiple enhancement layers included in the target dynamic 3D scene, at least one target layer to be transmitted is adaptively determined for the client. Following a progressive transmission strategy that prioritizes the transmission of the base layer, the at least one target layer is sequentially transmitted to the client as an independent transmittable unit. This includes: Determine whether the remaining space is lower than a preset first threshold; If the remaining space is lower than the first threshold, based on the recorded transmission progress for all stored dynamic 3D scenes, find the first dynamic 3D scene that has not yet been transmitted to the client from all stored dynamic 3D scenes; The first dynamic 3D scene is identified as the target dynamic 3D scene to be transmitted, and the base layer contained in the target dynamic 3D scene is transmitted to the client as at least one target layer to be transmitted.

[0018] Optionally, the method further includes: If the remaining space is not lower than the first threshold, it is further determined whether the receiving bandwidth is lower than a preset second threshold; If the received bandwidth is lower than the second threshold, based on the recorded transmission progress for all stored dynamic 3D scenes, the first dynamic 3D scene that has not yet been transmitted to the client from all stored dynamic 3D scenes is searched. The first dynamic 3D scene is identified as the target dynamic 3D scene to be transmitted, and the quality scores of the base layer and each level of enhancement layer contained in the target dynamic 3D scene are calculated; wherein, the quality score is used to represent the rendering quality; The layer with the highest quality score in the target dynamic 3D scene is identified as at least one target layer to be transmitted. If the at least one target layer contains only a base layer, the base layer is transmitted to the client. If the target layer includes any one of the multiple levels of enhancement layers, the base layer, the enhancement layers at each level preceding the target enhancement layer, and the target enhancement layer are transmitted to the client sequentially according to a progressive transmission strategy that prioritizes the transmission of the base layer.

[0019] Optionally, the quality scores of the base layer and each level of enhancement layers contained in the target dynamic 3D scene are calculated, including: Calculate the quality gain score of the base layer and each level of enhancement layer contained in the target dynamic 3D scene; wherein, the quality gain score represents the rendering quality gain relative to the base layer; Calculate the bitrate of the base layer and each level of enhancement layer contained in the target dynamic 3D scene; Based on the rendering quality gain, the bitrate, and the remaining space in the buffer, the quality scores of the base layer and each level of enhancement layers contained in the target dynamic 3D scene are further calculated; wherein, the quality score is positively correlated with the rendering quality gain, negatively correlated with the bitrate, and negatively correlated with the remaining space in the buffer.

[0020] Optionally, the method further includes: If the received bandwidth is greater than or equal to the second threshold, the byte budget for data transmission is calculated based on the remaining space in the buffer. Based on the recorded transmission progress of all stored dynamic 3D scenes, find the dynamic 3D scene that has been transmitted to the client at least once a base layer, and the first dynamic 3D scene that has not been transmitted to the client at any layer. A set of dynamic 3D scenes is created based on the found dynamic 3D models for optimization solutions; From the set of dynamic 3D scenes, at least one dynamic 3D scene is further identified where the display effect on the client does not reach the final display effect enhanced by the highest-level enhancement layer on the initial display effect. All candidate transmission actions for each of the at least one dynamic 3D scene are listed. Each candidate transmission action indicates the transmission of at least one layer of the base layer and the multi-level enhancement of each dynamic 3D scene to the client. The number of bytes required to execute each candidate transmission action and the corresponding user experience score are pre-calculated. The user experience score represents the user's perceived experience of the display effect obtained by the client visually rendering the target dynamic 3D scene based on the Gaussian elements contained in the at least one layer transmitted according to the candidate transmission action after the execution of the candidate transmission action. The optimization objective is to find the candidate action combination with the highest cumulative value of the user experience score, provided that the total number of bytes consumed does not exceed the byte budget. The optimization solution is then performed on all the listed candidate transmission actions to find the target candidate action combination for the target dynamic 3D scene to be transmitted in the set of dynamic 3D scenes. The target candidate transmission action with the highest user experience score in the target candidate action combination is determined, and the target candidate transmission action is executed to transmit the client as at least one target layer to be transmitted, in accordance with a progressive transmission strategy that prioritizes the transmission of the base layer.

[0021] Optionally, the optimization solution includes: dynamic programming related to the group knapsack problem, which uses the candidate action combination with the highest cumulative value of user experience score as the planning objective, provided that the total number of bytes consumed does not exceed the byte budget; correspondingly, the optimization solution algorithm includes the dynamic programming algorithm corresponding to the group knapsack problem.

[0022] Optionally, the 3D scene model is pre-trained in the following manner: a non-overlapping group of images is obtained by segmenting the video data source used to train the dynamic 3D scene model; the image group is pre-trained based on the spatiotemporal Gaussian splashing modeling method to obtain a 3D scene model constructed based on Gaussian meta-units that change dynamically in the time dimension.

[0023] Optionally, the video data source includes multi-dimensional volumetric videos with multiple perspectives.

[0024] According to a second aspect of the embodiments of this specification, an electronic device is provided, including one or more processors and a memory, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described in any of the first aspects above.

[0025] According to a third aspect of the embodiments of this specification, a computer program product includes a computer program / instructions that, when executed by a processor, implement the method described in any of the first aspects above.

[0026] The technical solutions provided in the embodiments of this specification have the following beneficial effects.

[0027] This specification describes an embodiment that calculates the importance score of Gaussian units in a pre-trained dynamic 3D scene model and divides the model into a base layer and multiple enhancement layers based on this score. This allows each layer to be organized according to the contribution of Gaussian units to rendering quality. Furthermore, each layer is stored as an independent, transmittable unit. During transmission, the target dynamic 3D scene to be transmitted can be adaptively determined based on the client's receiving bandwidth, and the base layer and multiple enhancement layers contained in the target dynamic 3D scene are sequentially transmitted to the client.

[0028] This processing method allows the client to begin visualization rendering immediately after receiving a small amount of data containing the most important Gaussian primitives from the base layer, without waiting for the complete model to arrive. This significantly reduces startup latency and avoids the bandwidth waste of having to acquire a full, high-quality version whenever visualization rendering quality needs to be improved. Moreover, since each subsequent enhancement layer is a new Gaussian primitive superimposed on the base layer, all transmitted data is preserved and continues to function, thus significantly improving the efficiency of bandwidth utilization. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the architecture of a dynamic three-dimensional scene transmission system provided in an exemplary embodiment.

[0030] Figure 2 This is a flowchart of a dynamic three-dimensional scene transmission method provided in an exemplary embodiment.

[0031] Figure 3 This is a schematic diagram of an importance score calculation process provided in an exemplary embodiment.

[0032] Figure 4 This is a schematic diagram of a hierarchical division process provided in an exemplary embodiment.

[0033] Figure 5 This is a schematic diagram illustrating how the cumulative number of primitives varies with the hierarchy, provided in an exemplary embodiment.

[0034] Figure 6 This is a schematic diagram of a client-side progressive rendering process provided in an exemplary embodiment.

[0035] Figure 7 This is a schematic diagram of a layer-by-layer fine-tuning training process provided in an exemplary embodiment.

[0036] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an exemplary embodiment.

[0037] Figure 9 This is a block diagram of a transmission device based on a dynamic three-dimensional scene, provided in an exemplary embodiment. Detailed Implementation

[0038] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0039] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties. The collection, use and processing of related data shall comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals shall be provided for users to choose to authorize or refuse.

[0040] With the rise of immersive media applications such as free-viewpoint video, telepresence, and extended reality, the real-time rendering and efficient transmission of dynamic 3D scenes have become urgent technical problems to be solved.

[0041] Dynamic 3D scenes are typically represented and rendered using the Spatiotemporal Gaussian Splatting technique. Spatiotemporal Gaussian Splatting is a representation and rendering method for dynamic 3D scenes. This method uses a set of spatiotemporal Gaussian primitives with position, scale, rotation, color, transparency, and time-dependent parameters (such as time center, time scale, polynomial motion coefficients, rotation offset, and time-dependent appearance features) to represent the scene, and further models the changes of these primitives over time, thereby supporting high-quality free-viewpoint rendering of dynamic scenes.

[0042] This type of scene model consists of a large number of Gaussian primitives carrying spatial position, scale, rotation, opacity, and time-related parameters, enabling it to achieve a high rendering frame rate while supporting free-viewpoint browsing.

[0043] However, the amount of data contained in a dynamic 3D scene model usually reaches tens to hundreds of megabits, which is far higher than the amount of data in traditional 2D video with the same perceptual quality, thus placing higher demands on transmission bandwidth.

[0044] In real-world network environments, the available receiving bandwidth for clients often exhibits time-varying and unpredictable characteristics.

[0045] When network conditions are good, transmitting a complete dynamic 3D scene model can achieve high rendering quality; when network conditions deteriorate, if data is still sent according to the same transmission strategy, it may lead to increased transmission delay or even playback interruption.

[0046] It is evident that the core technical challenge in the field of dynamic 3D scene streaming is how to enable clients to begin visualization rendering as quickly as possible under conditions of bandwidth fluctuations, gradually improve rendering quality during subsequent transmission, and avoid wasting transmitted data.

[0047] In related technologies, one approach is to transmit the complete dynamic 3D scene model as a single file, requiring the client to wait for all data to be received before rendering and playback can begin.

[0048] However, this method can lead to a long startup delay when the file size is large, making it difficult to meet the real-time requirements of interactive application scenarios.

[0049] Another approach draws inspiration from HTTP-based dynamic adaptive streaming media technology.

[0050] This method divides the dynamic 3D scene content into multiple independent segments over time, and each segment is pre-encoded into multiple discrete quality versions. The client independently selects one quality version for each segment to download based on the currently estimated network bandwidth.

[0051] However, this approach lacks data reuse between different quality versions. When a client switches from a low-quality version to a high-quality version, the previously downloaded low-quality fragment data cannot be used to refine the fragment's quality; the complete high-quality version of the fragment must be downloaded again, resulting in wasted bandwidth resources. Furthermore, since each fragment can only be selected for a single quality version, downloaded fragments cannot be upgraded when network conditions improve, leading to lower transmission efficiency in scenarios with fluctuating bandwidth.

[0052] In view of this, this specification provides a transmission technology solution for dynamic 3D scenes. Its core is: by evaluating the importance of Gaussian units in the pre-trained dynamic 3D scene model, the model is divided into a base layer and multiple enhancement layers according to the evaluation results and stored independently. During transmission, the target layer to be transmitted is adaptively determined according to the client's receiving bandwidth and buffer status, so that the client can start visualization rendering immediately based on the base layer and gradually improve the rendering effect when the enhancement layer is received.

[0053] The exemplary embodiments of this specification will now be described in detail with reference to the accompanying drawings.

[0054] Please see Figure 1 , Figure 1 This is a schematic diagram of the architecture of a dynamic three-dimensional scene transmission system provided in the embodiments of this specification.

[0055] like Figure 1 As shown, the system includes a server side and a client side.

[0056] The server-side may include an offline preprocessing module and an online scheduling module. The offline preprocessing module further includes an importance scoring unit, a hierarchical unit, and a storage unit.

[0057] The importance scoring unit is used to calculate the importance score of each Gaussian unit in the pre-trained dynamic 3D scene model. This importance score represents the degree of importance of the Gaussian unit to the rendering quality of the dynamic 3D scene.

[0058] The hierarchical unit is used to divide a dynamic 3D scene model into a base layer and multiple enhancement layers based on importance scores. The base layer contains the Gaussian units with the highest importance scores, and the multiple enhancement layers contain the remaining Gaussian units with importance scores decreasing sequentially.

[0059] The storage unit is used to store the base layer and multiple enhancement layers as independent transportable units, and to generate a list containing layer identifier, byte size and quality information for each transportable unit.

[0060] The online scheduling module may include a bandwidth estimation unit, a buffer monitoring unit, and a decision-making unit.

[0061] The bandwidth estimation unit is used to acquire the received rate sample reported by the client and estimate the client's received bandwidth. The buffer monitoring unit is used to acquire the remaining buffer space reported by the client. The decision unit is used to determine the target layer to be transmitted from the stored transmittable units based on the received bandwidth and the remaining buffer space.

[0062] The client side can include a receiving unit, a rendering unit, and a feedback unit.

[0063] The receiving unit receives data from each layer sent by the server. The rendering unit merges the Gaussian elements from the received layer data into the rendering buffer and performs visualization rendering based on the Gaussian elements in the rendering buffer. The feedback unit measures the receiving rate during data reception, monitors the remaining buffer space and playback progress, and periodically or event-triggeredly reports the above information to the server.

[0064] The server and client sides exchange data via a communication network. After determining the target layer to be transmitted, the server-side decision unit sends the target layer to the client-side receiving unit via the communication network. The client-side feedback unit reports the receiving rate, remaining buffer space, and playback progress to the server-side bandwidth estimation unit and buffer monitoring unit via the communication network.

[0065] Based on the above system architecture, those skilled in the art will understand that the various modules of this system can be implemented in many different ways.

[0066] In some implementations, importance score calculation and stratification can be performed on a cloud server, while online scheduling and transmission are performed by an edge server.

[0067] Specifically, the cloud server utilizes its powerful computing resources to perform model pre-training, importance score calculation, layering, and compression, distributing the layered data to the storage nodes of each edge server. The edge servers respond to client requests, performing adaptive scheduling and transmission based on real-time network conditions. This distributed architecture helps reduce client access latency and improves service scalability.

[0068] In another implementation, the client can also participate in part of the decision-making process. For example, the client can proactively request a specific layer from the server based on its own playback status and network measurement results, without completely relying on the server's scheduling decisions. This client-driven model can coexist with the server-driven model, with the client's request serving as a reference factor for the server's decision-making.

[0069] In other implementations, the system can also support concurrent transmission scenarios with multiple clients. When multiple clients simultaneously request the same dynamic 3D scene content, the server can utilize multicast or multi-cast technology to send the same layer data to multiple clients at the same time, thereby saving bandwidth resources. For different clients with varying cache states and quality requirements, the server can maintain independent transmission schedules and scheduling strategies for each client.

[0070] The above description is merely a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of protection of this specification.

[0071] Please see Figure 2 , Figure 2 This is a flowchart illustrating a method for transmitting a dynamic three-dimensional scene as shown in this specification. This method can be... Figure 1 The server-side execution is shown; the method includes the following execution process: Step 202: Calculate the importance score of each Gaussian unit in the pre-trained dynamic 3D scene; wherein the importance score represents the degree of importance of the Gaussian unit to the rendering quality of the dynamic 3D scene; The entity that performs the above method can be Figure 1 The server shown can be a standalone physical server, a server cluster consisting of multiple servers, or a virtual server instance provided by a cloud computing platform. The server is typically deployed at the edge node of a content delivery network or data center to provide data transmission services to clients from the nearest location. During the offline phase, the server completes the pre-training and hierarchical storage of the dynamic 3D scene model; during the online phase, it responds to client transmission requests and provides adaptive scheduling services.

[0072] In the embodiments described in this specification, the aforementioned dynamic 3D scene may specifically include a 3D scene model constructed based on Gaussian elements that dynamically change over time.

[0073] In practical applications, this 3D scene model can be pre-trained from a group of pictures (GoP) obtained by segmenting a video data source containing multiple perspectives.

[0074] In some embodiments, the video data source may include multi-dimensional volumetric video with multiple perspectives. Volumetric video is a form of three-dimensional dynamic video that can provide a free perspective and immersive viewing experience. It not only records the content of the picture, but also includes the three-dimensional spatial structure of the scene, allowing users to freely change the viewing angle in three-dimensional space.

[0075] An image group is a non-overlapping sequence of consecutive frames. Each image group contains a predetermined number of consecutive time frames, for example, 60 frames in 2-second durations. Each image group trains an independent 3D scene model. There is no reference dependency between models corresponding to different image groups. Therefore, each image group model can be used as an independent streaming segment.

[0076] In some embodiments, pre-training may employ a modeling approach using spatiotemporal Gaussian splatting.

[0077] Spatiotemporal Gaussian splashing is a representation and rendering method for dynamic 3D scenes. It uses a set of spatiotemporal Gaussian primitives with position, scale, rotation, color, transparency, and time-related parameters to represent the scene and models the changes of these primitives over time, thereby supporting free-view rendering of dynamic scenes.

[0078] The pre-trained dynamic 3D scene model contains N Gaussian elements, each carrying multiple sets of Gaussian parameters. These parameters include at least position parameters, scale parameters, rotation parameters, spatial opacity parameters, temporal center parameters, temporal scale parameters, motion coefficient parameters, rotation offset parameters, basic appearance feature parameters, dynamic appearance feature parameters, and time-varying appearance feature parameters.

[0079] Among them, the time center parameter and the time scale parameter constitute a time radial basis function, which is used to control the activation interval and decay characteristics of the Gaussian element on the time axis; the motion coefficient parameter is a cubic polynomial coefficient, which is used to calculate the trajectory of the position of the Gaussian element as the time changes; the rotation offset parameter is used to calculate the linear offset of the rotation of the Gaussian element as the time changes; the basic appearance feature parameter, the dynamic appearance feature parameter, and the time-varying appearance feature parameter together encode the color and appearance information of the Gaussian element at different times and different viewpoints.

[0080] The aforementioned importance score can be used to quantify the importance of each Gaussian primitive to the rendering quality of a dynamic 3D scene.

[0081] It should be noted that the method for quantifying the importance of Gaussian elements to the rendering quality of dynamic 3D scenes is not specifically limited in this specification. In practical applications, different quantization methods can be flexibly selected based on specific needs.

[0082] In some embodiments, after pre-training is completed, the importance score can be calculated based on the degree of impact of the rendering loss generated by Gaussian elements during training on the rendering quality.

[0083] In practical applications, the impact of rendering loss on rendering quality can be represented by the magnitude of the gradient of rendering loss. In this case, the importance score of each Gaussian unit can be calculated based on the magnitude of the gradient of rendering loss generated during pre-training.

[0084] Specifically, calculating the importance score of each Gaussian unit in the pre-trained dynamic 3D scene can include: For each Gaussian primitive in the pre-trained dynamic 3D scene, calculate the sum of the gradient magnitudes of the rendering loss generated during the pre-training process of the dynamic 3D scene for each of the multiple sets of Gaussian parameters of the Gaussian primitive, and obtain the original gradient magnitude of the Gaussian primitive on each set of Gaussian parameters. Furthermore, the original gradient magnitude of the Gaussian element on each set of Gaussian parameters is normalized to obtain the normalized gradient magnitude of the Gaussian element on each set of Gaussian parameters; the normalized gradient magnitudes of the Gaussian element on all sets of Gaussian parameters are summed to obtain the importance score corresponding to the Gaussian element.

[0085] Please see Figure 3 , Figure 3 This is a schematic diagram illustrating the importance score calculation process provided in the embodiments of this specification.

[0086] like Figure 3 As shown, for each Gaussian element First, obtain the Gaussian parameters for each group during the pre-training process. to The gradient vectors are calculated, and the magnitudes of each gradient vector are calculated. to The gradient magnitude is represented by the modulus; then, the modulus of each group is normalized to obtain... to Finally, the normalized values ​​are summed to obtain the importance score of the Gaussian unit. .

[0087] For example, for each Gaussian element i and its j-th group of Gaussian parameters The original gradient magnitude can be calculated as follows: in, Indicates the time frame index. This indicates the total number of frames contained in an image group. Indicates the camera view index. This represents the total number of camera views used during training. This represents the rendering loss at frame t and viewpoint v. Representing vectors Modulus length. The physical meaning is: the sum of the gradient magnitudes of the j-th group of Gaussian parameters of the Gaussian element for rendering loss over all time and all viewpoints. The larger this value is, the more important the group of parameters is to the rendering quality.

[0088] Then, for each parameter group Perform the original gradient magnitude on all Gaussian elements. Normalization. The normalization method is as follows: for the th Set parameters, calculate the mean of the original gradient magnitudes of all N Gaussian elements. and standard deviation Then, calculate the normalized gradient magnitude using the following formula: in, for The standard deviation. After z-score normalization, the gradient magnitudes of the parameters in each group are adjusted to a uniform scale with a mean of 0 and a standard deviation of 1.

[0089] Of course, in practical applications, other types of normalization methods can also be used when normalizing importance scores, which will not be listed one by one in this manual.

[0090] Finally, the normalized gradients of each Gaussian element on all P groups of Gaussian parameters are summed to obtain the importance score of that Gaussian element: This importance score comprehensively reflects the contribution of the Gaussian primitive to the rendering quality across multiple dimensions, including position, scale, rotation, opacity, temporal parameters, motion parameters, and appearance features.

[0091] In the above embodiments, when calculating the importance score, the original gradient magnitude is obtained by summing the rendering loss gradient magnitudes of each Gaussian unit on each set of Gaussian parameters. This is then normalized and summed to obtain the importance score. Since the dimensions and numerical ranges of the various Gaussian parameters (such as position parameters, opacity parameters, and time parameters) differ significantly, directly using the original gradient magnitudes for summation would lead to a single parameter group dominating the score. Through normalization, the contributions of each parameter group are adjusted to the same scale before summation, allowing the final importance score to more accurately reflect the overall contribution of each Gaussian unit to the rendering quality.

[0092] In some embodiments, the method for calculating rendering loss may include: For each frame and each viewpoint, the image rendered by the model is compared with the real-world image, and the rendering loss is calculated as a weighted sum of L1 loss and D-SSIM loss.

[0093] It is important to note that the gradient magnitude reflects the degree of change in rendering loss when a set of Gaussian parameters undergoes a small change. The larger the gradient magnitude, the greater the impact of the change in that set of parameters on rendering quality, meaning that the parameters of that Gaussian unit are more important to rendering quality.

[0094] Because the magnitudes and numerical ranges of the Gaussian parameters can vary significantly—for example, the gradient value of the position parameter might be much larger than that of the opacity parameter—directly summing the original gradient magnitudes would lead to the larger parameter group dominating the scoring result. Therefore, each group of Gaussian parameters is normalized separately to adjust the importance contribution of each group to the same scale before summing. This ensures that the final importance score more comprehensively reflects the overall importance of each Gaussian unit across different attribute dimensions.

[0095] For example, during the calculation process, rendering loss You can follow Loss and Weighted sum calculation of losses: in, The model represents the first time. Frame and the The image rendered from the viewpoint L1 represents the sum of absolute errors per pixel, where L1 represents the actual captured image. Represents structural similarity loss. The preset weighting coefficient (e.g., 0.8) is used to balance the relative importance of the two losses.

[0096] By employing a weighted combination of L1 loss and D-SSIM loss, pixel-level accuracy can be guaranteed while perceptual quality can be improved, making the subsequent importance scoring based on gradient calculation more consistent with the characteristics of human vision.

[0097] It should be noted that in practical applications, the importance score can be calculated offline only once on the server. After calculation, the score result is associated with each Gaussian unit and stored for subsequent hierarchical operations. Since both pre-training and importance score calculation are offline preprocessing steps and do not involve real-time transmission decisions, higher computational precision and longer processing time can be used to ensure the accuracy of the importance score.

[0098] Of course, in practical applications, in addition to using the degree of influence of rendering loss on rendering quality to characterize the above importance score, other methods can also be used to quantify the importance score, which will not be listed one by one in this specification.

[0099] For example, in some embodiments, besides using the impact of rendering loss on rendering quality to calculate importance scores, the importance of Gaussian elements can be directly inferred from their Gaussian parameters using machine learning. For instance, a neural network model (such as a lightweight MLP) can be trained, and the Gaussian parameters of the Gaussian elements can be input into the neural network to directly predict their importance scores.

[0100] In other alternative implementations, the importance score can also be calculated based on the perceived quality contribution of each Gaussian unit.

[0101] Specifically, after pre-training, the server performs a "removal-measurement" operation on each Gaussian primitive: temporarily removing a Gaussian primitive from the complete model, then rendering a test image using the remaining Gaussian primitives, and measuring the perceptual quality difference between this rendered image and the reference image rendered by the complete model. The perceptual quality difference can be measured using the Learned Perceptual Image Patch Similarity (LPIPS) metric. This metric extracts image features through a deep neural network and calculates the distance between features, effectively reflecting the subjective perception of image quality by the human eye. The magnitude of the decrease in perceptual quality after removing the primitive is used as the primitive's importance score; the greater the decrease, the more important the primitive is in maintaining perceptual quality. The importance score based on perceptual quality contribution directly targets the subjective quality of the final rendered image, better reflecting the primitive's importance in the actual viewing experience. For example, since removing primitives one by one and re-rendering is computationally expensive, approximate methods can be used, such as removing a batch of primitives instead of individual primitives, or measuring only on a few keyframes and key viewpoints.

[0102] In some other implementations, the importance score can also be calculated based on the geometric properties of the Gaussian meta-element itself, without the need for rendering operations, thereby significantly reducing computational overhead.

[0103] Specifically, for each Gaussian element, its importance can be assessed based on its geometric characteristics such as spatial coverage volume, curvature variation, and location distribution density.

[0104] For example, primitives with a large spatial volume or located in areas of dramatic curvature (such as object edges and textured areas) are generally more critical to representing the scene's geometry and can be assigned higher scores; while primitives located in flat areas or heavily occluded by other primitives contribute less and can be assigned lower scores. The importance scoring method based on geometric attributes is decoupled from the rendering process, allowing it to be independent of the specific rendering viewpoint, and it is computationally efficient, making it suitable for use in offline preprocessing scenarios with limited computing resources.

[0105] In some implementations, importance scores can also be learned based on attention mechanisms.

[0106] Specifically, during the pre-training phase, an attention module is introduced into the rendering network of the dynamic 3D scene model. This attention module takes the parameters of each Gaussian primitive (such as position, scale, opacity, etc.) as input and outputs the contribution weight of each primitive to the final rendering result. This contribution weight can serve as the source of importance scoring. During training, the attention module is jointly optimized with other parts of the scene model, and backpropagation enables the attention weights to reflect the degree of contribution of each primitive to the rendering loss. After training, the weights of each primitive can be directly extracted from the attention module as importance scores. The importance scoring based on the attention mechanism can adaptively learn the contribution distribution of each primitive without additional post-processing calculations and can be dynamically adjusted as the model is updated. Compared with gradient-based or removal-measurement-based methods, the attention mechanism can simultaneously complete importance evaluation during training, reducing the additional overhead of offline preprocessing. In addition, the attention weights can also serve as a direct basis for subsequent layering operations, making the layering process more tightly coupled with model training.

[0107] In some embodiments, the aforementioned multiple sets of Gaussian parameters may include position parameters, scale parameters, rotation parameters, spatial opacity parameters, time center parameters, time scale parameters, motion coefficient parameters, rotation offset parameters, basic appearance feature parameters, dynamic appearance feature parameters, and appearance feature parameters.

[0108] Taking the spatiotemporal Gaussian splash model as an example, each Gaussian element specifically contains the following P=10 sets of Gaussian parameters: 3D position parameters , used to control the position of primitives in space; Scale parameters This is used to control the distribution range of primitives in each dimension; Rotation parameters , used to control the orientation of the primitives; Spatial opacity parameter , used to control the occlusion intensity of primitives in the spatial domain; Time center parameters , used to control the activation center position of the primitive on the time axis; Time scale parameters This is used to control the width of the activation interval of a primitive on the time axis; Motion coefficient parameters , where is the coefficient of a cubic polynomial, used to calculate the trajectory of the primitive position as it changes over time; Rotation offset parameters , used to calculate the linear offset of the primitive rotation as it changes over time; Basic appearance feature parameters Dynamic appearance feature parameters and time-varying appearance characteristic parameters It is used to jointly encode the color and appearance information of primitives at different times and from different perspectives.

[0109] The above parameter set covers the spatial geometry, temporal dynamics, and appearance representation attributes of Gaussian primitives in dynamic 3D scenes, enabling the importance score to comprehensively reflect the combined contribution of each primitive in different attribute dimensions.

[0110] In some embodiments, the server needs to pre-train a dynamic 3D scene model before calculating the importance score.

[0111] The video data sources used during training can include dynamic scene video sequences captured from different perspectives by multiple synchronized cameras. The server divides these video sequences into non-overlapping image groups along the timeline, with each image group serving as an independent training unit. For each image group, the server can pre-train using a spatiotemporal Gaussian splashing method, optimizing the parameters of each Gaussian unit to minimize the rendering loss between the multi-view images rendered by the model and the actually captured images. After pre-training, the server obtains a dynamic 3D scene model containing N Gaussian units.

[0112] Based on this, the server can perform a complete forward-backward propagation on the pre-trained model, calculating the rendering loss gradient of each Gaussian primitive on each set of Gaussian parameters. To obtain a stable importance score, this forward-backward propagation process can cover all time frames and all training views within the image set. For each set of parameters for each primitive, the gradient magnitudes under each frame and view are summed to obtain the original gradient magnitude. Then, z-score normalization is performed on each set of parameters, and finally, the normalized values ​​of each set are summed to obtain the importance score.

[0113] Step 204: Based on the importance score, the dynamic 3D scene is divided into a base layer and multiple enhancement layers, and the base layer and the multiple enhancement layers are stored as independent transportable units. The base layer contains the Gaussian elements with the highest importance scores in the dynamic 3D scene; the multiple enhancement layers contain the remaining Gaussian elements in the dynamic 3D scene excluding the aforementioned Gaussian elements; the importance scores of the Gaussian elements in the multiple enhancement layers decrease sequentially. After obtaining the importance score of each Gaussian unit, the server can divide the N Gaussian units of the dynamic 3D scene model into a base layer and multiple enhancement layers based on the score.

[0114] The base layer contains the Gaussian units with the highest importance scores; multiple enhancement layers contain the remaining Gaussian units, and the importance scores of the Gaussian units in each enhancement layer decrease sequentially.

[0115] In other words, primitives are assigned to the base layer, the first enhancement layer, the second enhancement layer, and so on, in descending order of importance score. Primitives in the base layer are the most critical to rendering quality, and the importance of primitives in each subsequent enhancement layer decreases progressively.

[0116] Storing each layer as an independent, transmittable unit means that the data in each layer can be read, transmitted, and decoded independently of the data in other layers.

[0117] When storing data, the server generates an independent file or data block for each layer and records the identification information, byte size, number of primitives contained in each layer, and quality information (such as the expected peak signal-to-noise ratio of the layer).

[0118] In addition, the server can generate a manifest containing the storage location, size, and quality information of all layers for subsequent transmission scheduling. This independent storage of each layer allows the server to select which layers to send based on actual network conditions, rather than sending the entire model.

[0119] In some embodiments, the number of Gaussian elements contained in the base layer can be a preset initial number; the number of Gaussian elements contained in multiple enhancement layers increases sequentially according to a preset coefficient based on the initial number.

[0120] Specifically, based on importance scoring, a dynamic 3D scene is divided into a base layer and multiple enhancement layers, which may include: Obtain the initial number preset for the base layer; calculate the first number of Gaussian elements contained in the first-level enhancement layer based on the initial number, and extract the Gaussian elements with the lowest importance score from all Gaussian elements in the dynamic 3D scene according to the first number, and create the first-level enhancement layer based on the extracted Gaussian elements. Based on the first quantity, the second quantity of Gaussian elements contained in the second-level enhancement layer is calculated, and the Gaussian elements with the lowest importance scores are stripped from the remaining Gaussian elements in the dynamic 3D scene according to the second quantity. The second-level enhancement layer is then created based on the stripped Gaussian elements. This process is repeated until the number of enhancement layers created reaches the specified number. Finally, a base layer is created based on the remaining Gaussian elements in the dynamic 3D scene.

[0121] Please see Figure 4 , Figure 4 This is a schematic diagram of the hierarchical division process provided in the embodiments of this specification.

[0122] like Figure 4 As shown, the complete model contains N Gaussian units, which are stripped away sequentially according to their importance scores from low to high. First, the unit with the lowest importance score is stripped from the complete model. Each primitive is used as the (L-1)th level enhancement layer (the highest level enhancement layer); then the importance score of the remaining primitives is recalculated, and the primitives with the lowest importance scores are stripped away. Each primitive is used as the L-2 level enhancement layer; And so on, until the remaining Each primitive as the base layer The diagram uses circular areas of different gray levels and sizes to represent the number and importance of primitives in each layer, with the base layer being the most prominent. Contains the fewest primitives but has the highest importance (dark area), various enhancement layers to The number of primitives included increases progressively while their importance decreases progressively (the light-colored area gradually expands).

[0123] The number of Gaussian elements contained in multiple enhancement layers is in the initial number Based on this, the coefficients are increased sequentially according to a preset coefficient, and the increase can be geometric progression.

[0124] In other words, if the base layer contains The first enhancement layer brings the cumulative number of primitives to [number]. (i.e., the first enhancement layer itself contains) (each primitive), the second enhancement layer brings the cumulative primitive count to [number of primitives]. One, and so on, until the l-th enhancement layer brings the cumulative number of primitives to 1. indivual.

[0125] in The geometric growth factor is preset. Since the rate-distortion curve after importance ranking usually exhibits diminishing marginal returns, a small number of primitives initially can bring a large quality improvement, while the quality gain brought by the same number of primitives gradually decreases. The geometric growth method makes the granularity of the early enhancement layer finer, which can achieve a more significant quality improvement with less data; the granularity of the later enhancement layer is coarser, which is suitable for transmission when bandwidth is sufficient to provide fine-grained supplementation.

[0126] In the above embodiments, by obtaining the initial number of basic layers, the number of primitives that each level of enhancement layer should contain is calculated based on the initial number in a geometric growth manner, and the primitives with the lowest importance scores are sequentially stripped from all the Gaussian primitives to create each level of enhancement layer, and the remaining Gaussian primitives constitute the basic layer.

[0127] Because the importance distribution of Gaussian primitives in dynamic 3D scenes typically exhibits a long-tail characteristic, a small number of primitives contribute the vast majority of rendering quality, while the contribution of a large number of primitives is relatively limited. By retaining a small number of the most important primitives in the base layer, the amount of data in the base layer can be controlled within a very small range, so that it can be quickly delivered to the client during the transmission startup phase. At the same time, the size of each enhancement layer increases geometrically, so that the bytes transmitted subsequently can continuously bring perceptible quality improvements, thereby achieving a balance between transmission efficiency and rendering quality.

[0128] Of course, in practical applications, in addition to using the reverse stripping method mentioned in the above examples, which involves separating the model in order of importance scores from low to high, other layering methods can obviously also be used.

[0129] In some implementations, the hierarchical division of the model can also be carried out in a forward construction manner, that is, in order of importance score from high to low, Gaussian primitives are selected from the complete model to construct the base layer and each enhancement layer, rather than starting from the complete model and peeling off the lowest score primitives step by step.

[0130] Specifically, this forward construction method may include the following steps: First, select the Gausky units with the highest importance scores from all Gausky units, in descending order. A high-level element, based on this Each high-level element creates the basic layer. .

[0131] Then, from the remaining Gaussian elements, select the one with the second highest importance score according to the number of targets. A high-level element, based on this Each Gaussian element creates the first enhancement layer. .

[0132] Next, from the remaining Gaussian elements, select those with higher importance scores based on the number of targets. A high-level element is used to create a second enhancement layer. .

[0133] This process continues until all enhancement layers have been created.

[0134] In this forward construction method, the number of Gaussian units contained in each enhancement layer is the same as in the aforementioned geometric progression method, i.e. This forward construction method and the aforementioned reverse stripping method are essentially equivalent in the final layered result; both ensure that the base layer contains the most important elements. Each enhancement layer contains primitives of decreasing importance.

[0135] The difference lies in the construction order: the reverse stripping method gradually discards low-scoring primitives starting from the complete model. Its advantage is that it can optimize the boundary of each layer in the sense of rate distortion by iteratively recalculating the importance score; while the forward construction method completes the allocation of all layers at once, without the need for multiple rounds of recalculation, making it simpler and more direct, with less computational overhead.

[0136] In practical applications, the appropriate method can be selected based on specific computing resources and time budget.

[0137] On the one hand, if computing resources are plentiful and optimal rate-distortion performance is desired, a reverse stripping method can be adopted; On the other hand, if computational resources are limited or the requirements for hierarchical accuracy are relatively lenient, a forward construction approach can be adopted. For example, when the model size is small or the total number of primitives is small, the global importance score of each primitive in the complete model can already reflect its contribution well, without the need to optimize the marginal contribution through iterative recalculation. In this case, the forward construction approach becomes a suitable alternative due to its simplicity and computational efficiency.

[0138] Furthermore, the forward construction method is also easier to combine with the aforementioned importance scoring method based on attention mechanism. After obtaining the attention weights of each primitive during training, it can be directly sorted by weight for layering without additional gradient calculation or recalculation steps.

[0139] In other implementations, forward construction and reverse stripping can be combined to form a hybrid layering strategy.

[0140] For example, the initial partitioning of the base layer and some lower enhancement layers can be determined first using a forward construction method. Then, the boundaries of the higher enhancement layers can be fine-tuned through a finite number of iterations of recalculation, thus achieving a trade-off between computational efficiency and layer quality. Specifically, after completing the forward construction, several lower enhancement layers (such as...) close to the base layer can be selected. and ), and jointly recalculate and fine-tune its primitive set and the basic layer, while the higher-level enhancement layers (such as , The partitioning result of the forward construction is retained to reduce computational overhead.

[0141] In other implementations, the layering approach can also be based on the spatial clustering results of primitives. Specifically, all Gaussian primitives can be spatially clustered first, grouping primitives with similar spatial locations into the same cluster. Then, within each cluster, primitives are sorted according to their importance scores, and a corresponding proportion of high-scoring primitives are extracted from each cluster to form the base layer. The remaining primitives are distributed to each enhancement layer in a balanced manner between clusters. This layering approach based on spatial clustering helps maintain the uniformity of the spatial distribution of primitives in each layer, avoiding a situation where primitives in one layer are concentrated in a local area of ​​the scene while primitives in other areas are sparse. This results in a more balanced spatial improvement in image quality during progressive rendering.

[0142] Please see Figure 5 , Figure 5 This is a schematic diagram illustrating how the cumulative number of primitives varies with the hierarchy, as provided in the embodiments of this specification.

[0143] like Figure 5 As shown, the horizontal axis represents the quality level l (0 is the basic layer, and 1 to L-1 are the enhancement layers), and the vertical axis represents the cumulative number of primitives. The cumulative number of primitives is calculated according to... The growth follows a geometric progression, with the number of primitives corresponding to each layer ranging as follows: Basic layer Include Each primitive, the first enhancement layer Include One basic unit, second enhancement layer Include Each basic unit, and so on, up to the highest enhancement layer. The cumulative total reaches N primitives. From Figure 5 As can be seen, the interval width of the base layer and the early enhancement layer is relatively narrow, while the interval width of the later enhancement layer gradually increases.

[0144] This geometric progression can be used to calculate the number of primitives in each enhancement layer as follows. Let the total number of layers be L (1 base layer plus L-1 enhancement layers), and the number of primitives in the base layer be... If the total number of primitives in the complete model is N, then the geometric growth factor is... satisfy ,Right now .

[0145] At this point, the l-th enhancement layer The number of primitives is: in, This corresponds to the first enhancement layer (the enhancement layer immediately adjacent to the base layer). This corresponds to the highest enhancement layer (the last enhancement layer to be stripped). The above formula shows that the number of primitives in each enhancement layer increases geometrically with the increase of the layer level, that is... Minimum, maximum.

[0146] In some embodiments, after the creation of any level of enhancement layer is completed, the importance scores of the remaining Gaussian elements in the dynamic 3D scene can be recalculated.

[0147] Specifically, the hierarchical partitioning adopts an iterative stripping approach, starting from the complete model and stripping outwards level by level. In each round of stripping, the importance score is first recalculated on the current set of remaining primitives, and then the primitives with the lowest importance scores are stripped out as the current highest-level enhancement layer.

[0148] The reason for recalculation is that the relative importance relationships among the remaining primitives may change after some primitives are removed. A primitive may have a high importance score in the complete model, but its marginal contribution in the remaining set may have decreased; and vice versa. By recalculating, we can ensure that the primitives removed each time are the ones with the smallest contribution in the current remaining set, thus making the stratification results more reasonable in terms of rate distortion.

[0149] For example, after initially calculating the importance score, the scores with the lowest importance scores are first stripped. Each primitive serves as the highest enhancement layer. Then, the gradient is recalculated using forward-backward propagation on the remaining primitives to obtain new importance scores, and then the primitives with the lowest new importance scores are stripped away. Each element serves as the second-highest enhancement layer. Repeat the above process until the product is peeled off. After that, the remaining Each basic element forms the foundation layer. Through this iterative stripping and recalculation method, the rate-distortion boundaries between layers become more reasonable.

[0150] In some embodiments, the number of Gaussian elements contained in the multiple enhancement layers increases exponentially from the initial number according to a preset coefficient; the preset coefficient is calculated based on the total number of Gaussian elements in the dynamic 3D scene, a specified number preset for the enhancement layers, and the initial number. Specifically, the geometric growth factor β is calculated according to the following formula: This formula is equivalent to In actual calculations, because and The ratio may be large; using a logarithmic-exponential form can avoid numerical overflow and improve computational stability. Once the preset coefficient β is determined, the number of primitives in each enhancement layer is determined according to the aforementioned formula. Sure.

[0151] This design allows the number of primitives in the base layer and each level of enhancement layers to adapt to the total number of primitives N and the preset total number of layers L, without needing to adjust the layer size individually for each scene.

[0152] In practical implementation, the number of basic layer primitives can be pre-set according to the complexity of the scenario and transmission requirements. And the total number of layers L. For complex scenarios with a large number of primitives, the number of primitives in the basic layer or the total number of layers can be increased appropriately; for simple scenarios with a small number of primitives, the number of primitives in the basic layer or the total number of layers can be decreased appropriately. The value typically ranges from several thousand to tens of thousands; for example, in a typical indoor dynamic scene, It can be set to 10000; for simpler scenarios, It can be set to 3000. The total number of layers L can be determined according to the requirements of asymptotic granularity. For example, L=8 is used in experimental verification. The highest layer or more layers can be discarded through pruning later.

[0153] In the above embodiments, since the rate-distortion curves after importance ranking typically exhibit diminishing marginal returns, a small initial number of primitives can bring significant quality improvements, while the quality gain from subsequent primitives of the same number gradually decreases. Therefore, a geometric progression approach is adopted to make the early enhancement layers finer-grained, enabling more significant quality improvements with less data; the later enhancement layers have coarser granularity, suitable for providing fine-grained supplementation when bandwidth is ample. In some embodiments, when each layer is stored as an independent transmittable unit, the following approach can be used: For each layer, all Gaussian parameters of the primitives belonging to that layer are serialized according to a predetermined format, and layer header information (including layer identifier, layer number, number of primitives, data type and length of each parameter, etc.) is added. This information is then written to a separate file or data block. Simultaneously, the server generates a manifest file recording the identifier, byte offset, byte size, number of primitives, and expected rendering quality (such as peak signal-to-noise ratio) for each layer. The manifest file is loaded into memory during subsequent transmission scheduling, allowing the scheduler to quickly query the attribute information of each layer.

[0154] Step 206: In response to a request to transmit a dynamic 3D scene to a client, determine the client's receiving bandwidth, and based on the receiving bandwidth, determine the target dynamic 3D scene to be transmitted from the stored dynamic 3D scenes, and transmit the base layer and multiple enhancement layers contained in the target dynamic 3D scene sequentially to the client according to a progressive transmission strategy that prioritizes the transmission of the base layer.

[0155] After completing offline preprocessing, the server can respond to the client's transmission request and enter the online transmission phase. When requesting transmission, the client can include parameters such as its device information, network type, and initial buffer size in the request message. Upon receiving the request, the server starts the transmission service for that client.

[0156] First, the server determines the client's receiving bandwidth. Receiving bandwidth refers to the available data transmission rate of the downlink from the server to the client, usually measured in megabits per second (Mbps). Because network bandwidth is time-varying, the server needs to continuously estimate the client's receiving bandwidth during transmission and dynamically adjust the transmission strategy based on the estimation results.

[0157] In some embodiments, determining the client's receive bandwidth may include: Obtain the most recent multiple receive rate samples reported by the client; calculate the harmonic mean of the multiple receive rate samples as an estimate of the client's receive bandwidth.

[0158] For example, in practical applications, during data reception, the client can measure the reception rate for each data block or each time window and periodically report these rate samples to the server. The server maintains a sliding window, storing the K most recent reception rate samples (e.g., K=5). The harmonic mean is then calculated using the following formula: in, Indicates the first The harmonic mean is a sample of the received rate at each time point. It is more sensitive to smaller values, and when rate fluctuations exist, the harmonic mean will bias towards lower values, thus providing a conservative bandwidth estimate. Compared to the arithmetic mean, using the harmonic mean can reduce the risk of transmission delays or packet loss caused by sending more data than the actual transmission capacity due to overestimating bandwidth.

[0159] In other implementations, bandwidth estimation may use an exponentially weighted moving average (EWMA) instead of a harmonic mean to more smoothly track bandwidth changes. In still other implementations, the harmonic mean may be combined with EWMA for a rapid response when bandwidth is rapidly decreasing and smooth tracking when bandwidth is increasing.

[0160] Then, the server can determine the target dynamic 3D scene to be transmitted from the stored dynamic 3D scene based on the estimated receiving bandwidth.

[0161] In dynamic 3D scene streaming, the complete video content is divided into multiple consecutive segments (each segment corresponds to a dynamic 3D scene model with a set of images), and the client plays these segments sequentially. The server needs to track the transmission progress of each segment, i.e., which layers each segment has sent to the client. Based on this, the server determines the target segment and target layer for this transmission, according to the current estimated receiving bandwidth and the client's buffer state.

[0162] After determining the target dynamic 3D scene and the target layer, the server can sequentially transmit one base layer and multiple enhancement layers contained in the target dynamic 3D scene to the client according to a progressive transmission strategy that prioritizes the transmission of the base layer.

[0163] It should be noted that the so-called progressive delivery strategy means that when transmitting each segment, the base layer is sent first, so that the client can start rendering as early as possible; and then, when network conditions permit, the enhancement layers are sent in sequence to gradually improve the rendering quality.

[0164] During transmission, once the client receives the base layer of a segment, it can begin visual rendering of that segment to obtain an initial display effect. When the client subsequently receives the enhancement layer of the same segment, it overlays the Gaussian primitives from the enhancement layer onto the existing primitives and updates relevant parameters (such as opacity) to achieve an improved display effect. Since the base layer contains the most important Gaussian primitives in the segment, even if only the base layer arrives, the image rendered by the client already possesses a recognizable scene structure and main visual content.

[0165] In some embodiments, the target dynamic 3D scene to be transmitted is determined from the stored dynamic 3D scene based on the received bandwidth, and a base layer and multiple enhancement layers contained in the target dynamic 3D scene are sequentially transmitted to the client according to a progressive transmission strategy that prioritizes the transmission of the base layer. Specifically, this may include: Based on the received bandwidth, the target dynamic 3D scene to be transmitted is determined from the stored dynamic 3D scene. From the base layer and multiple enhancement layers contained in the target dynamic 3D scene, at least one target layer to be transmitted is adaptively determined for the client. According to the progressive transmission strategy of prioritizing the transmission of the base layer, at least one target layer is transmitted to the client as an independent transmittable unit in sequence.

[0166] In other words, the server-side decision-making unit selects one or more layers from all available layers as the target layer for this transmission based on the current network conditions and client status. The selected target layer can be the base layer of a segment, one or more enhancement layers of a segment, or a combination of different layers from different segments. Then, the server sends these target layers to the client in a progressive order (base layer first, then enhancement layer). Each transmission unit is a complete layer; that is, once the client receives a layer, all the data for that layer has arrived and can be used immediately for rendering.

[0167] In some embodiments, the target dynamic 3D scene to be transmitted can be determined from the stored dynamic 3D scene based on the remaining buffer space and receiving bandwidth reported by the client, and at least one target layer to be transmitted can be adaptively determined for the client from a base layer and multiple enhancement layers contained in the target dynamic 3D scene. Then, the at least one target layer is transmitted to the client as an independent transmittable unit.

[0168] Specifically, the server can obtain the remaining space in the cache reported by the client.

[0169] The remaining buffer space refers to the playback duration corresponding to the video data that the client has downloaded but has not yet played, usually measured in seconds. During playback, the client continuously consumes data in the buffer while replenishing it by receiving new data. The larger the remaining buffer space, the longer the client can withstand network fluctuations; the smaller the remaining buffer space, the higher the risk of playback interruption for the client.

[0170] Then, it can be determined whether the remaining space is lower than a preset first threshold.

[0171] For example, in practical applications, this first threshold can be set to 6 seconds, corresponding to the duration of 3 image groups (2 seconds per image group). When the remaining space in the buffer is less than 6 seconds, the client is at risk of near-lag.

[0172] If the remaining space is below the first threshold, the first dynamic 3D scene that has not yet been transmitted to the client can be found from all the stored dynamic 3D scenes based on the recorded transmission progress for all stored dynamic 3D scenes; this first dynamic 3D scene is determined as the target dynamic 3D scene to be transmitted, and the base layer contained in this target dynamic 3D scene is transmitted to the client as the target layer to be transmitted.

[0173] When the buffer space is low, playback may be about to be interrupted. At this time, the server should provide the client with new data available for playback as quickly as possible. The first dynamic 3D scene that has not yet been transmitted through any layer is the next segment needed in the client's playback progress, and this segment is not currently in the client's buffer at all. Sending the base layer of this segment, which is the smallest in data volume and contains the most important Gaussian elements, can provide the client with playable data with the least amount of transmission time, thereby alleviating the buffer shortage.

[0174] In some embodiments, if the remaining space is not less than a first threshold, it can be further determined whether the received bandwidth is less than a preset second threshold.

[0175] For example, in practical applications, this second threshold can be set to 10 Mbps, which corresponds to the bit rate required to maintain a full-quality dynamic 3D scene stream. When the receiving bandwidth is less than 10 Mbps, it indicates that the network bandwidth is insufficient to support continuous transmission at full quality, but may still support the transmission of the base layer or part of the enhancement layer.

[0176] If the received bandwidth is lower than the second threshold, it can enter the limited bandwidth transmission mode. Based on the recorded transmission progress for all stored dynamic 3D scenes, it searches for the first dynamic 3D scene that has not yet been transmitted to the client from all stored dynamic 3D scenes. Then, the first dynamic 3D scene is determined as the target dynamic 3D scene to be transmitted, and the quality scores of the base layer and each level of enhancement layer contained in the target dynamic 3D scene are calculated. The layer with the highest quality score contained in the target dynamic 3D scene is then determined as the target layer to be transmitted. Specifically, if the target layer to be transmitted is the base layer, it can be transmitted to the client; if the target layer is any level of the multi-level enhancement layer, the base layer of the target dynamic 3D scene, the various enhancement layers before the target enhancement layer, and the target enhancement layer are transmitted to the client in sequence according to the progressive transmission strategy of prioritizing the transmission of the base layer.

[0177] In some embodiments, calculating the quality scores of the base layer and each level of enhancement layers contained in the target dynamic 3D scene may include: Calculate the quality gain score of the base layer and each level of enhancement layers contained in the target dynamic 3D scene. This quality gain score represents the rendering quality gain relative to the base layer. Calculate the bitrate of each layer; The quality score of each layer is further calculated based on the quality gain score, bitrate, and remaining buffer space. The quality score is positively correlated with the quality gain score, negatively correlated with the bitrate, and negatively correlated with the remaining buffer space.

[0178] For example, for the l-th layer, its utility value can be calculated first. (i.e., mass gain score): in, This represents the code rate (bit rate) of the l-th layer. This represents the bitrate of the base layer. This utility value reflects the increment of the layer's bitrate relative to the base layer on a logarithmic scale, and is an approximate measure of the rendering quality improvement that layer may bring. Since bitrate is usually positively correlated with rendering quality, a logarithmic transformation can make the utility differences between layers more uniform.

[0179] Then, calculate the quality score of the layer according to the following formula. : in, These are preset weighting coefficients used to control the importance of quality utility relative to buffer pressure; This represents the remaining space in the current cache. This represents the bitrate of the layer. From this formula, it can be seen that the quality score is related to the utility value. and weighting coefficients The product of these parameters is positively correlated; the higher the utility value (i.e., the greater the quality gain the layer brings), the higher the score. Simultaneously, the quality score is also related to the remaining cache space. Negative correlation; the larger the buffer, the lower the score (but this penalty applies the same penalty to all layers, and its effect is less than the bitrate). (and different); quality score and bitrate There is a negative correlation; the higher the bitrate, the lower the score. Therefore, this scoring mechanism can strike a balance between "pursuing the image quality improvement brought by high-quality layers" and "avoiding high-bitrate layers exhausting the cache".

[0180] When there is ample remaining space in the buffer, selecting a higher bitrate layer results in less buffer pressure, and the scoring naturally tends to favor higher layers. Conversely, when there is limited remaining space in the buffer, selecting a lower bitrate layer is more conducive to quickly filling the buffer and avoiding stuttering, and the scoring naturally tends to favor lower layers. After calculating the scores for each layer, the server selects the layer with the highest score as the target layer for transmission.

[0181] In some embodiments, if the received bandwidth is greater than or equal to the second threshold, a high-bandwidth transmission mode can be entered, and the byte budget for data transmission is calculated based on the remaining space in the buffer.

[0182] For example, in one instance, the byte budget can be calculated as follows: in The time factor, whose value varies with the remaining space in the cache. It increases with the increase of; This is an estimated value for the received bandwidth. A safety factor (e.g., 0.75) is used to reserve some bandwidth to account for estimation errors. The specific value of the time factor can be determined as follows: when... seconds, seconds; when seconds, seconds; when seconds, Linear interpolation between 2 and 6 seconds.

[0183] Then, the server can, based on the recorded transmission progress for all stored dynamic 3D scenes, find the dynamic 3D scenes that have been transmitted to the client with at least one base layer (some layers of these scenes are already cached on the client and can receive enhancement layers to upgrade the visualization rendering quality), as well as the first dynamic 3D scene that has not yet been transmitted to the client with any layers (this scene needs to be forward transmitted).

[0184] Then, a set of dynamic 3D scenes can be created based on the found dynamic 3D scenes for optimization, serving as the solution space for optimization.

[0185] Furthermore, from this set of dynamic 3D scenes, we can further identify the dynamic 3D scenes whose display effect on the client has not reached the highest level of enhancement layer, that is, the scenes whose quality has not yet reached the highest level, and list all candidate transmission actions for these scenes.

[0186] The candidate transmission action indicates that at least one layer of the base layer or enhancement layer of a certain scene should be transmitted to the client. Each candidate transmission action pre-calculates the number of bytes required to execute the action and the corresponding user experience score (such as a QoE score). The user experience score is used to represent the user's perceived experience of the display effect obtained by the client's visualization rendering based on the layer transmitted by the action after the action is executed.

[0187] Finally, the optimization objective can be "the candidate action combination with the highest cumulative value of user experience score under the condition that the total number of bytes consumed does not exceed the byte budget". The optimization solution can be performed on all the listed candidate transmission actions to find the target candidate action combination for the target dynamic 3D scene to be transmitted in the set.

[0188] Then, the target candidate transmission action with the highest user experience score in the target candidate action combination is determined, and the target candidate transmission action is executed. That is, according to the progressive transmission strategy of prioritizing the transmission of the base layer, at least one layer indicated by the target candidate transmission action is transmitted to the client as the target layer.

[0189] Through the above three-layer decision-making logic, the server can adaptively determine the transmission strategy under different network conditions and cache status: when the cache is tight, priority is given to ensuring playback continuity; when bandwidth is limited but the cache is still available, the optimal quality layer is selected on the premise of ensuring continuity; when bandwidth is sufficient, bandwidth resources are allocated among multiple scenarios through global optimization, while taking into account forward transmission and backfill upgrade.

[0190] In practical implementation, the aforementioned dynamic 3D scene set serves as a decision window. This window starts with the segment currently playing on the client and ends with the first dynamic 3D scene that has not yet been transmitted to the client at any layer. Segments within the window have all been transmitted through at least one base layer and can be used for backfilling and upgrading; the first untransmitted segment immediately outside the window is used for forward transmission. User experience scoring is also considered. The calculation method can be: in, The peak signal-to-noise ratio gain resulting from upgrading from the current quality layer to the target quality layer l; This is a quality switching penalty term, used to prevent the quality layer from fluctuating drastically between adjacent decision cycles; The urgency weight decays exponentially with the remaining time before playback; segments with shorter remaining playback time have a higher urgency weight. The transmission cost of candidate actions. It is the sum of the byte sizes of all layers from the next layer of the current layer to the target layer.

[0191] In some embodiments, the above-mentioned optimization problem can be solved using a dynamic programming algorithm with a grouped knapsack problem.

[0192] In this context, the optimization solution includes dynamic programming related to the group knapsack problem, with the goal of calculating the candidate action combination that yields the highest cumulative user experience score while ensuring that the total number of bytes consumed does not exceed the byte budget. Correspondingly, the optimization solution algorithm includes the dynamic programming algorithm corresponding to the group knapsack problem.

[0193] Based on the dynamic programming algorithm corresponding to the grouped knapsack problem, each dynamic 3D scene is considered as a group, containing all candidate transfer actions for that scene (including the no-upgrade action, which has a cost of 0 and a value of 0). Each group can select at most one action. Under the byte budget constraint, the action combination with the highest total value is selected.

[0194] Of course, in addition to transforming it into a grouped knapsack problem for solution, in practical applications, greedy algorithms or branch and bound algorithms can also be used to replace dynamic programming in order to balance the solution speed and optimality.

[0195] Based on the above implementation, since each dynamic 3D scene can only be selected to execute at most one candidate action within the same decision cycle, and each candidate action has different byte consumption and user experience score, the problem naturally presents itself as a "group knapsack" structure where each scene is a group and at most one action can be selected within each group.

[0196] Dynamic programming can provide a globally optimal solution with a finite byte budget. , where m is the number of groups and B is the number of discretization steps for the byte budget. Since the number of segments within the decision window is usually small (e.g., no more than 10), the discretization step size for the byte budget can be set to an appropriate value (e.g., 1KB or 10KB) so that dynamic programming can complete the solution in a short time.

[0197] It's important to note that during transmission, each layer can be transmitted through independent QUIC streams. The QUIC protocol supports establishing multiple independent streams over a single connection, with each stream handling flow control and packet loss recovery independently. By mapping the base layer and each enhancement layer to different QUIC streams, independent transmission and out-of-order arrival of each layer can be achieved. After receiving data from each layer, the client assembles it according to the layer sequence number. If data from a certain layer is lost or delayed during transmission, it does not affect the decoding and rendering of already received layers. This design improves the robustness and flexibility of the transmission.

[0198] After receiving the data from each layer sent by the server, the client can perform rendering operations.

[0199] First, the client can perform preliminary visualization rendering of the target dynamic 3D scene based on the Gaussian meta-meta-elements contained in the received base layer to obtain the initial display effect.

[0200] Specifically, the client loads the Gaussian primitive data contained in the base layer into the rendering buffer, and performs Gaussian splash rasterization rendering on each time frame and each viewpoint of the target dynamic 3D scene based on these Gaussian primitives to obtain the initial display effect.

[0201] Since the base layer only contains the most important parts of all primitives, the rendered image may not be as good as the complete model in terms of geometric details and texture precision, but it already has a recognizable scene structure and main visual content.

[0202] Furthermore, if the client receives the subsequent transmission of the enhancement layer (or may not receive it), it can append the new Gaussian primitives contained in the enhancement layer to the render buffer, and overwrite the opacity values ​​of all Gaussian primitives already stored in the render buffer with the updated opacity vector carried by the enhancement layer. Then, it can perform Gaussian splash rasterization rendering based on all the updated Gaussian primitives to obtain the improved display effect.

[0203] By progressively stacking enhancement layers, the client's rendering quality gradually improves until it reaches the highest usable quality for that segment.

[0204] It should be noted that during the rendering process, the layer stacking method is "incremental merging" rather than "replacement".

[0205] Since the geometric, motion, and appearance parameters of primitives in the base layer remain unchanged when subsequent enhancement layers arrive, with only the opacity parameter being updated, all transmitted data is preserved and continues to function. During client-side rendering, all active Gaussian primitives are sorted in order of depth from farthest to nearest, and then alpha composited sequentially to obtain the final pixel color. New primitives in the enhancement layers are inserted into their corresponding positions in the rendering queue, participating in compositing alongside existing primitives, thereby achieving a gradual improvement in rendering quality.

[0206] Please see Figure 6 , Figure 6 This is a schematic diagram of the client-side progressive rendering process provided in the embodiments of this specification.

[0207] like Figure 6 As shown, the client first receives the base layer. The initial image is rendered (left image); then the first enhancement layer is received. ,Will The primitives in the image are added to the render buffer and the opacity is updated, rendering the improved image (middle image); the second enhancement layer is then received. Repeat the above steps to render a higher quality image (right image). As enhancement layers are gradually stacked, the geometric details, texture precision, and dynamic smoothness of the image are improved step by step.

[0208] In some embodiments, after dividing a dynamic 3D scene into a base layer and multiple enhancement layers, visual artifacts may occur when rendering directly using the layered primitives because the layering operation disrupts the joint optimization relationship between the primitives in the original model. To eliminate these artifacts, this specification also provides an implementation method of layer-by-layer fine-tuning training.

[0209] In some embodiments, the aforementioned multiple sets of Gaussian parameters include spatial opacity parameters.

[0210] After dividing the dynamic 3D scene into a base layer and multiple enhancement layers, the server performs further training on the base layer and each enhancement layer.

[0211] The further training of the base layer and each level of the enhancement layer includes: The base layer is trained by using all the Gaussian parameters of each Gaussian unit contained in the base layer as trainable parameters. After training the base layer is completed, freeze all Gaussian parameters of each Gaussian element in the base layer except for the spatial opacity parameter. Use the spatial opacity parameter of each Gaussian element in the base layer and all Gaussian parameters of each Gaussian element in the first level of the multi-level enhancement layer as trainable parameters to jointly train the base layer and the first level enhancement layer. After the joint training of the base layer and the first-level enhancement layer is completed, the Gaussian parameters of each Gaussian element in the base layer, except for the spatial opacity parameter, are frozen. The spatial opacity parameter of each Gaussian element in the base layer and the full set of Gaussian parameters of each Gaussian element in the first and second-level enhancement layers in the multi-level enhancement layers are used as trainable parameters, and the base layer, the first-level enhancement layer and the second-level enhancement layer are jointly trained. This process continues until all enhancement layers at each level have been trained.

[0212] It should be noted that the core idea of ​​the above fine-tuning training strategy is to "freeze high-dimensional attributes and fine-tune low-dimensional opacity".

[0213] Specifically, after the layering operation is completed, the base layer is first independently fine-tuned with all parameters to adapt the primitives in the base layer to a rendering scenario where "only these primitives exist". Then, the first enhancement layer is introduced. At this point, all parameters of all primitives in the base layer except for spatial opacity (including position, scale, rotation, time parameters, motion coefficients, appearance features, etc.) are frozen, allowing only the spatial opacity parameter of the base layer primitives to be displayed. All parameters of all primitives in the base layer are used as trainable variables. This asymmetric freezing strategy allows the geometry, motion trajectory, and appearance features of the base layer primitives to remain unchanged, and only the opacity is adjusted to accommodate the addition of new primitives, thus avoiding the destruction of the learned effective geometric and motion knowledge during fine-tuning. The new primitives in the model lack prior knowledge, so full-parameter training is used to enable them to quickly adapt to the scene. After training, introduce in the same way , ...until all enhancement layers have been trained.

[0214] For example, see Figure 7 , Figure 7 This is a schematic diagram of the layer-by-layer fine-tuning training process provided in the embodiments of this specification.

[0215] like Figure 7 As shown, the first row represents the training phase, from left to right: independent training of the base layer, training of the base layer + ... Joint training, base layer+ + Joint training. The second row indicates the parameter states at each stage: base layer primitives are represented in dark color, and their parameter states include "fully trainable", "opacity-only trainable", and "opacity-only trainable"; newly added enhancement layer primitives are represented in light color, and their parameter states are all "fully trainable". From Figure 7 As can be seen, as the training phase progresses, the parameters of the existing primitives are gradually frozen, with only the opacity remaining as an adjustable variable.

[0216] Based on the above implementation of layer-by-layer fine-tuning training, by freezing all parameters of existing primitives except for the spatial opacity parameter during the process of introducing enhancement layers, training only the opacity parameter, and simultaneously training all parameters of the primitives added to the enhancement layers, it is possible to make the spatial opacity of each primitive adapt to the composition of the current set of activated primitives without shifting the geometric structure, motion trajectory, and appearance features of the existing primitives. This solves the problem of alpha synthesis weight imbalance caused by the absence of some primitives after layering.

[0217] Specifically, in the complete model, the opacity of each primitive is optimized when all primitives coexist. When only some primitives exist, the original opacity values ​​will cause an imbalance in the occlusion relationship; for example, a primitive that was originally occluded by other primitives and had a high opacity may become the foreground after layering, and its original opacity will cause the rendering result to be too bright or too dark.

[0218] By fine-tuning layer by layer, the opacity of each layer configuration can be re-optimized, ensuring visual consistency of the rendering results across all layers.

[0219] In other implementations, layer-by-layer fine-tuning can employ different freezing strategies. For example, the position and motion parameters of existing primitives can be frozen, while the scale and rotation parameters are allowed to continue training; or only the geometric parameters can be frozen, while the appearance feature parameters are allowed to continue training.

[0220] The choice of freezing strategy depends on which type of parameters are more prone to harmful drift during fine-tuning in a specific scenario. For example, a progressive unfreezing strategy can also be used, which involves gradually unfreezing some of the frozen parameters during training, allowing for more detailed adjustments to the model in the later stages of fine-tuning.

[0221] In some embodiments, after joint training of any level of enhancement layer and base layer is completed, the updated spatial opacity parameter obtained through training can be obtained; then, the obtained updated spatial opacity parameter is saved as data to be transmitted to the enhancement layer, so that the client can perform visualization rendering of the target dynamic 3D scene based on the updated spatial opacity parameter saved in the enhancement layer during the process of further visualization rendering of the target dynamic 3D scene based on the Gaussian meta-meta-encapsulated in the enhancement layer.

[0222] It should be noted that when saving the updated spatial opacity parameter to the enhancement layer, the difference between the updated spatial opacity parameter and the previous spatial opacity parameter can be calculated, and then only the calculated residual can be saved to the enhancement layer as the data to be transmitted.

[0223] In this way, the full opacity parameter can be avoided by adding it to the enhancement layer, thus enabling lightweight transfer of the updated spatial opacity parameter.

[0224] In some implementations, the updated spatial opacity parameter can also be transmitted independently of the enhancement layer.

[0225] Specifically, after completing the joint training of a certain level of enhancement layer and base layer, the server packages the updated opacity parameter vector separately, stores it as an independent transmittable unit, and sends it to the client. This opacity parameter vector is independent of the enhancement layer's own primitive attribute data and can be scheduled independently during transmission.

[0226] After receiving the primitive attribute data of a certain enhancement layer, if the client has not yet received the corresponding updated opacity parameter, it will continue to render using the current opacity value; after receiving the independently transmitted opacity parameter, it will overwrite the rendering buffer to complete the full overlay of the layer.

[0227] Compared to the method of embedding the opacity parameter into the enhancement layer and transmitting it together, this independent transmission method increases the flexibility of transmission scheduling, allowing the server to schedule primitive attribute data and opacity parameters separately according to their urgency.

[0228] For example, when the buffer is tight, primitive attribute data is sent first to ensure that the client can obtain new primitives in a timely manner, and the corresponding opacity parameters are sent later when bandwidth is sufficient. At the same time, when the client has cached the primitive attributes of a certain enhancement layer but fails to receive its opacity parameters due to network fluctuations, it is not necessary to re-request the entire enhancement layer; only the opacity parameters need to be requested to complete the rendering configuration of that layer, thereby reducing unnecessary retransmission overhead.

[0229] Based on the above implementation, by saving the updated spatial opacity parameters to the transportable unit of the enhancement layer, the client can obtain the optimal opacity values ​​of all activated primitives under the current layer configuration when subsequently receiving the enhancement layer. The client only needs to overwrite the opacity of existing primitives with this set of opacity parameters to ensure that the primitives in the base layer and previously received enhancement layers achieve the correct occlusion relationship and compositing effect under the current layer configuration, without retransmitting any geometric or motion parameters. Since the storage and transmission overhead of opacity parameters is much smaller than that of geometric and appearance feature parameters, this design ensures rendering consistency between layers with minimal transmission overhead.

[0230] Of course, in practical applications, to further compress the data volume, the updated opacity parameter can be quantized and compressed before being saved to the enhancement layer. For example, 8-bit or 16-bit integer quantization can be used instead of 32-bit floating-point storage, reducing storage and transmission overhead within an acceptable range of precision loss.

[0231] Since the base layer and enhancement layers play different roles in transmission and decoding, the base layer is the foundation for all quality levels, and its compression errors will propagate to all enhancement layers that depend on it. This specification also provides implementation methods for differential compression.

[0232] In some embodiments, before storing the base layer and the multiple enhancement layers as independent transportable units, the method further includes: Differentiated quantization methods are used to quantize and compress the Gaussian parameters of each Gaussian element contained in the base layer and multiple enhancement layers. The degree of quantization compression of the quantization method used to quantize and compress at least some of the Gaussian parameters of each Gaussian element contained in the enhancement layer can be higher than the degree of quantization compression of the quantization method used to quantize and compress the Gaussian parameters of each Gaussian element contained in the base layer. Furthermore, entropy encoding can be applied to the base layer and multiple enhancement layers after quantization and compression.

[0233] Specifically, for the base layer, since its compression error propagates to all enhancement layers, a high-precision quantization strategy with low fidelity loss is adopted. For example, FP16 (16-bit floating-point) quantization is used to store all Gaussian parameters. FP16 quantization halves the data volume compared to the original FP32 storage, while introducing minimal precision loss, and will not cause perceptible error accumulation in subsequent enhancement layers.

[0234] For enhancement layers, since the errors of each enhancement layer only affect the quality of that layer and the layers above it, a quantization strategy with a higher compression ratio can be used to reduce storage and transmission overhead.

[0235] Specifically, FP16 quantization is used for the geometric attributes (position and rotation) in the enhancement layer to ensure the accuracy of the geometric structure; two-level residual vector quantization (RVQ) is used to compress the high-dimensional feature attributes (basic appearance features, dynamic appearance features, motion coefficients, rotation offsets, and time-varying appearance features).

[0236] After quantization compression is complete, LZMA entropy encoding can be performed on all compressed data (including FP16 arrays, codebooks, indexes, etc.) to further remove statistical redundancy in the data.

[0237] LZMA is a lossless compression algorithm based on the Lempel-Ziv algorithm, which can further compress data without losing any information.

[0238] Based on the above implementation, since compression errors in the base layer propagate to all enhancement layers that depend on it, a quantization strategy with a lower compression ratio and higher fidelity can avoid error accumulation. Furthermore, errors in the enhancement layers only affect themselves and the quality improvement of subsequent layers; therefore, a quantization strategy with a higher compression ratio can significantly reduce storage and transmission overhead while keeping quality loss under control. This differentiated compression strategy effectively reduces the amount of data in each layer while minimizing overall quality loss.

[0239] In other implementations, different quantization compression degree allocation strategies may also be employed.

[0240] For example, for enhancement layers close to the base layer (such as...) , A higher degree of quantization compression can be used to reduce error propagation; for enhancement layers far from the base layer (such as... , (This allows for the use of a lower level of quantization compression to maximize the compression ratio.)

[0241] In some embodiments, before quantizing and compressing the Gaussian parameters of each Gaussian element contained in the base layer and the plurality of enhancement layers using a differentiated quantization method, the method further includes: Discard the last one or more enhancement layers among multiple enhancement layers, and retain the base layer and the remaining enhancement layers at each level to achieve model pruning.

[0242] Based on the above implementation method, since the last high-level enhancement layer contains Gaussian primitives with the lowest importance score, these primitives contribute very little to the rendering quality. Discarding them has a negligible impact on the overall rendering quality, but can reduce the amount of data stored and transmitted.

[0243] In practice, the number of layers to discard can be determined based on the target storage space or target transmission bandwidth. For example, if the original layering contains 8 layers (1 base layer plus 7 enhancement layers), the last 2 layers can be discarded, retaining only the first 6 layers. Experimental data shows that discarding the last two layers reduces the model size by approximately 56%, while the decrease in rendering quality is negligible.

[0244] In other implementations, pruning decisions can be determined based on the rate-distortion performance of each enhancement layer.

[0245] For example, if the marginal quality gain (relative to the increase in its data volume) of a certain enhancement layer is lower than a preset threshold, then that layer and all layers above it are discarded.

[0246] In other implementations, adaptive adjustments can be made according to the needs of different application scenarios. More layers can be retained for scenarios that pursue the ultimate image quality, while more layers can be discarded for scenarios that pursue the minimum storage and transmission overhead.

[0247] In practical applications, the server can also continuously receive closed-loop feedback from the client to adjust subsequent transmission strategies. In the closed-loop feedback control flow, the client can periodically or based on specific triggered events report status information to the server during the playback process. The server can then update the bandwidth estimate and buffer status based on the received status information and adjust subsequent transmission strategies accordingly.

[0248] In some embodiments, during operation, the client can generate status reports at predetermined time intervals (e.g., every 200 milliseconds or every 500 milliseconds) and send them to the server via a communication network.

[0249] The status report shall include at least one or more of the following information: The received rate sample is the rate value obtained by dividing the amount of data actually received by the client in the most recent measurement window by the duration of the measurement window. The remaining space in the buffer is the playback duration corresponding to the video data that the client has currently downloaded but has not yet played, expressed in seconds; Playback position, which is the frame number or timestamp that the client is currently playing.

[0250] In some embodiments, when a client detects that the remaining space in the cache is lower than a preset alarm threshold, it can immediately send a status report containing the cache status to the server without waiting for the next cycle.

[0251] This allows the server to receive more timely feedback when the buffer is low. Furthermore, the client can also trigger an immediate report when its receive rate experiences a jump exceeding a preset range.

[0252] In practical implementation, the triggering conditions that the client can use include: the remaining space in the buffer is less than 6 seconds; the change in the remaining space in the buffer compared to the last report exceeds 2 seconds; and the change in the receiving rate compared to the last report exceeds 30%. The values ​​in the above triggering conditions are only examples and can be adjusted according to the actual network environment and application requirements.

[0253] After receiving the status information reported by the client through the communication network, the server can perform corresponding update and adjustment operations.

[0254] Specifically, the server-side bandwidth estimation unit can extract a received rate sample from the status information, add the sample to a sliding window, and recalculate the harmonic mean to update the estimated received bandwidth.

[0255] The server-side buffer monitoring unit can also extract the remaining buffer space and playback position from the status information, and update the current buffer status and playback progress.

[0256] The server-side decision-making unit can also adjust the subsequent transmission strategy in the next decision cycle based on the updated received bandwidth estimate and buffer status.

[0257] If the remaining buffer space is below the first threshold, the base layer is forced to be sent; if the remaining buffer space is not below the first threshold but the receiving bandwidth is below the second threshold, the target layer is selected by quality scoring; if the receiving bandwidth is not below the second threshold, the byte budget is allocated by global knapsack optimization.

[0258] Through this closed-loop feedback mechanism, the server's transmission strategy can continuously track changes in network conditions and client status, achieving dynamic and adaptive progressive transmission.

[0259] The communication protocol for client-side status reporting can employ either HTTP / 2-based server push or QUIC-based streaming. In the QUIC implementation, the client can send status reports via independent control flow over the same QUIC connection, while the server sends layer data via data stream over the same connection. The control flow and data stream are independent; status reporting does not block data transmission, and vice versa. This design ensures timely feedback while preventing feedback data from interfering with main data transmission.

[0260] In some embodiments, data interaction between the server and the client can be performed using the QUIC protocol.

[0261] Specifically, the server can map the base layer and each enhancement layer to different QUIC streams for transmission. Each QUIC stream performs congestion control and packet loss recovery independently, so the transmission delay or packet loss of one layer does not affect the data transmission of other layers.

[0262] The client can receive different layers transmitted by the server in an out-of-order manner. That is, the client can receive lower enhancement layers before higher enhancement layers, or receive the base layer of a subsequent segment before receiving the enhancement layer of a previous segment. Subsequently, the client can sort and assemble the received data according to the layer sequence number to ensure the correct stacking order of the layers during rendering.

[0263] For example, the server can carry the layer number and fragment identifier in the header of each layer's data packet so that the client can correctly identify and assemble them.

[0264] This QUIC-based multi-stream design improves the robustness and flexibility of transmission, and further optimizes the performance of progressive transmission.

[0265] Figure 8 This is a schematic structural diagram of an electronic device provided in an exemplary embodiment.

[0266] like Figure 8As shown, device 800 mainly consists of a communication interface 802, a user interface 804, a processor 806, and a data storage 808. These components are interconnected and communicate with each other via a system bus, network, or other connection mechanism 810. The communication interface 802 enables device 800 to communicate with other devices, access networks, and transmission networks via analog or digital modulation. For example, the communication interface 802 may include a chipset and antenna for wireless communication with a radio access network or access point. Furthermore, the communication interface 802 can be a wired interface such as Ethernet, Token Ring, or a USB port, or a wireless interface such as Wi-Fi, Bluetooth, Global Positioning System (GPS), or a wide-area wireless interface (e.g., WiMAX or LTE). Of course, the communication interface 802 can also support other forms of physical layer interfaces and standard or proprietary communication protocols. The communication interface 802 may also include multiple physical communication interfaces, such as Wi-Fi, Bluetooth, and wide-area wireless interfaces.

[0267] User interface 804 includes receiving user input and providing output to the user. Therefore, user interface 804 may include input components such as a keypad, keyboard, touch-sensitive or presence-sensitive panel, computer mouse, trackball, joystick, microphone, still camera, and video camera, and output components such as a display screen (which may be combined with a touch-sensitive panel), CRT, LCD, LED, display using DLP technology, printer, and other similar devices known or developed in the future. User interface 804 may also generate auditory output via speakers, speaker jacks, audio output ports, audio output devices, headphones, and other similar devices known or developed in the future. In some embodiments, user interface 804 may include software, circuitry, or other forms of logic capable of transmitting and receiving data from external user input / output devices. Additionally or alternatively, device 800 may support remote access from other devices via communication interface 802 or another physical interface (not shown). User interface 804 may be configured to receive user input, the position and movement of which may be indicated by indicators or cursors described herein. User interface 804 may also be configured as a display device for rendering or displaying text fragments.

[0268] The processor 806 may contain one or more general-purpose processors and / or special-purpose processors.

[0269] Data storage 808 may include one or more volatile and / or non-volatile storage components and may be integrated wholly or partially with processor 806. Data storage 808 may include removable and non-removable components.

[0270] Processor 806 is capable of executing program instructions 818 (e.g., compiled or uncompiled program logic and / or machine code) stored in data storage 808 to perform the various functions described herein. Data storage 808 may contain a non-transitory computer-readable medium on which program instructions are stored, which, when executed by device 800, enable device 800 to perform any methods, processes, or functions disclosed in this specification and / or the accompanying drawings. Execution of program instructions 818 by processor 806 may result in processor 806 using data 812.

[0271] For example, program instructions 818 may include an operating system 822 (e.g., an operating system kernel, device drivers, and / or other modules) installed on device 800 and one or more application programs 820 (e.g., a browser, social application, or game application). Similarly, data 812 may include operating system data 816 and application data 814. Operating system data 816 is primarily accessible to the operating system 822, while application data 814 is primarily accessible to one or more application programs 820. Application data 814 may reside in a file system visible or hidden from the user of device 800.

[0272] Application 820 can communicate with operating system 822 through one or more application programming interfaces (APIs). These APIs help application 820 read and / or write application data 814, transmit or receive information via communication interface 802, receive or display information on user interface 804, etc.

[0273] In some terminology, application 820 may be simply referred to as an app. Furthermore, application 820 can be downloaded to device 800 through one or more online app stores or app markets. However, applications can also be installed on device 800 in other ways, such as through a web browser or a physical interface on device 800 (e.g., a USB port).

[0274] Please refer to Figure 9 This specification also proposes a dynamic three-dimensional scene transmission device, which can be applied to applications such as... Figure 8 The device shown is used to implement the technical solution of this specification. The device may include: The calculation module 901 calculates the importance score of each Gaussian primitive in the pre-trained dynamic 3D scene; wherein the importance score represents the degree of importance of the Gaussian primitive to the rendering quality of the dynamic 3D scene; The layering module 902 divides the dynamic 3D scene into a base layer and multiple enhancement layers based on the importance score, and stores the base layer and the multiple enhancement layers as independent transportable units. The base layer contains the Gaussian elements with the highest importance scores in the dynamic 3D scene; the multiple enhancement layers contain the remaining Gaussian elements in the dynamic 3D scene excluding the aforementioned Gaussian elements; the importance scores of the Gaussian elements in the multiple enhancement layers decrease sequentially. The transmission module 903, in response to a request to transmit a dynamic 3D scene to a client, determines the client's receiving bandwidth, and based on the receiving bandwidth, determines the target dynamic 3D scene to be transmitted from the stored dynamic 3D scenes. It then sequentially transmits a base layer and multiple enhancement layers contained in the target dynamic 3D scene to the client, allowing the client to perform preliminary visualization rendering of the target dynamic 3D scene based on the Gaussian elements contained in the received base layer, obtaining an initial display effect corresponding to the target dynamic 3D scene. Upon receiving subsequent transmissions of enhancement layers, the client further visualizes and renders the target dynamic 3D scene based on the Gaussian elements contained in the enhancement layers, obtaining a display effect that enhances the initial display effect based on the enhancement layers.

[0275] For ease of description, the above devices are described by dividing them into various modules or units based on their functions. Of course, when implementing one or more of these specifications, the functions of each module or unit can be implemented in the same or different software and / or hardware, or a module that performs the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0276] Based on the same concept as the methods described above, this specification also provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor performs the steps of the method as described in any of the above embodiments by executing the executable instructions.

[0277] Based on the same concept as the methods described above, this specification also provides a computer-readable storage medium having computer instructions stored thereon that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0278] Based on the same concept as the methods described above, this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0279] What those skilled in the art will understand is: In this specification, the terms include, encompass, or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, product, or apparatus that includes a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitation, the presence of additional identical or equivalent elements in a process, method, product, or apparatus that includes said elements is not excluded.

[0280] In this specification, "one," "a," and "the" do not specifically refer to the singular, but may also include the plural.

[0281] In this specification, ordinal numbers such as "first," "second," etc., do not necessarily indicate order; they are often used to distinguish between objects. For example, "first server" and "second server" usually refer to two servers. To differentiate between these two servers, they are described as "first server" and "second server." Of course, sometimes these two servers may be the same server.

[0282] In this specification, unless explicitly stated otherwise, the receipt and transmission of data are not necessarily direct; they can be indirect. For example, A receiving data sent by B can be understood as A directly receiving the data sent by B, or it can be understood as A indirectly receiving the data sent by B through other entities such as C. Similarly, B sending data to A can be understood as B sending the data directly to A, or it can be understood as B indirectly sending the data to A through other entities such as C. Here, C can be one entity, or it can be two or more entities.

[0283] In this specification, unless explicitly stated otherwise, the relationships between structures can be direct or indirect. For example, when describing a connection between A and B, unless explicitly stated that A and B are directly connected, it should be understood that A can be directly connected to B or indirectly connected to B. Similarly, when describing A above B, unless explicitly stated that A is directly above B (AB is adjacent and A is above B), it should be understood that A can be directly above B or indirectly above B (AB is separated by other elements, and A is above B). And so on.

[0284] This specification uses specific terms to describe embodiments thereof. For example, "an embodiment," "an embodiment," and / or "a number of embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of this specification. Therefore, it should be emphasized and noted that an embodiment, an embodiment, or an alternative embodiment mentioned two or more times in different locations in this specification does not necessarily refer to the same embodiment. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples, without contradiction.

[0285] Although one or more embodiments of this specification provide method steps as described in the embodiments or flowcharts, it is understood that the order of steps listed in the embodiments or flowcharts is only one of many possible execution orders and does not represent the only execution order. Therefore, when the claims involve method steps, any changes or adjustments to the order of such steps, or the parallelism between steps, are also within the scope of protection of the claims.

Claims

1. A method for transmitting a dynamic three-dimensional scene, wherein the dynamic three-dimensional scene comprises a three-dimensional scene model constructed based on Gaussian elements that dynamically change in the time dimension; the method comprises: Calculate the importance score of each Gaussian primitive in the pre-trained dynamic 3D scene; wherein the importance score represents the degree of importance of the Gaussian primitive to the rendering quality of the dynamic 3D scene; Based on the importance score, the dynamic 3D scene is divided into a base layer and multiple enhancement layers, and the base layer and the multiple enhancement layers are stored as independent transportable units. The base layer contains the Gaussian elements with the highest importance scores in the dynamic 3D scene; the multiple enhancement layers contain the remaining Gaussian elements in the dynamic 3D scene excluding the aforementioned Gaussian elements; the importance scores of the Gaussian elements in the multiple enhancement layers decrease sequentially. In response to a request to transmit a dynamic 3D scene to a client, the receiving bandwidth of the client is determined, and based on the receiving bandwidth, a target dynamic 3D scene to be transmitted is determined from the stored dynamic 3D scenes. A base layer and multiple enhancement layers contained in the target dynamic 3D scene are sequentially transmitted to the client, so that the client performs preliminary visualization rendering of the target dynamic 3D scene based on the Gaussian elements contained in the received base layer, obtaining an initial display effect corresponding to the target dynamic 3D scene. Upon receiving subsequent transmissions of enhancement layers, the client further performs visualization rendering of the target dynamic 3D scene based on the Gaussian elements contained in the enhancement layers, obtaining a display effect that enhances the initial display effect based on the enhancement layers.

2. The method as described in claim 1, wherein the target dynamic 3D scene comprises a base layer and multiple enhancement layers, which are sequentially transmitted to the client, includes: The target dynamic 3D scene, consisting of a base layer and multiple enhancement layers, is transmitted sequentially to the client according to a progressive transmission strategy that prioritizes the transmission of the base layer.

3. The method as described in claim 2, wherein the importance score corresponding to any Gaussian primitive includes a score calculated based on the gradient magnitude of the rendering loss corresponding to each Gaussian primitive generated during the pre-training process of the dynamic 3D scene; Calculate the importance score of each Gaussian unit in the pre-trained dynamic 3D scene, including: For each Gaussian primitive in the pre-trained dynamic 3D scene, calculate the sum of the gradient magnitudes of the rendering loss generated during the pre-training process of the dynamic 3D scene for each of the multiple sets of Gaussian parameters of the Gaussian primitive, and obtain the original gradient magnitude of the Gaussian primitive on each set of Gaussian parameters. The original gradient magnitude of the Gaussian element on each set of Gaussian parameters is normalized to obtain the normalized gradient magnitude of the Gaussian element on each set of Gaussian parameters. The importance score corresponding to the Gaussian element is obtained by summing the normalized gradients of the Gaussian parameters of all groups.

4. The method as described in claim 3, wherein the number of Gaussian elements contained in the base layer is a preset initial number; the number of Gaussian elements contained in the multi-level enhancement layer increases sequentially according to a preset coefficient based on the initial number; Based on the importance score, the dynamic 3D scene is divided into a base layer and multiple enhancement layers, including: Obtain the initial quantity preset for the base layer; The first number of Gaussian elements contained in the first-level enhancement layer is calculated based on the initial number, and the Gaussian elements with the lowest importance scores are stripped from all Gaussian elements in the dynamic 3D scene according to the first number, and the first-level enhancement layer is created based on the stripped Gaussian elements. Based on the first quantity, the second quantity of Gaussian elements contained in the second-level enhancement layer is calculated, and the Gaussian elements with the lowest importance scores are stripped from the remaining Gaussian elements in the dynamic 3D scene according to the second quantity, and the second-level enhancement layer is created based on the stripped Gaussian elements. This process continues until the number of enhancement layers reaches a specified number, at which point the base layer is created based on the remaining Gaussian elements in the dynamic 3D scene.

5. The method of claim 4, further comprising: After creating any level of enhancement layer, the importance scores of the remaining Gaussian elements in the dynamic 3D scene are recalculated.

6. The method as described in claim 4, wherein the number of Gaussian elements contained in the multiple enhancement layers increases exponentially sequentially according to a preset coefficient based on the initial number; in, The preset coefficients are calculated based on the total number of Gaussian elements in the dynamic 3D scene, the specified number preset for the enhancement layer, and the initial number.

7. The method of claim 4, wherein the plurality of Gaussian parameters comprises: Spatial opacity parameter; The method further includes: After dividing the dynamic 3D scene into a base layer and multiple enhancement layers, the base layer and each enhancement layer are further trained: The further training of the base layer and each level of the enhancement layer includes: The base layer is trained by using all the Gaussian parameters of each Gaussian unit contained in the base layer as trainable parameters. After training of the base layer is completed, the Gaussian parameters of each Gaussian element in the base layer, except for the spatial opacity parameter, are frozen. The spatial opacity parameter of each Gaussian element in the base layer and the full set of Gaussian parameters of each Gaussian element in the first level of the multi-level enhancement layer are used as trainable parameters, and the base layer and the first level enhancement layer are jointly trained. After the joint training of the base layer and the first-level enhancement layer is completed, the Gaussian parameters of each Gaussian element in the base layer, except for the spatial opacity parameter, are frozen. The spatial opacity parameter of each Gaussian element in the base layer and the full set of Gaussian parameters of each Gaussian element in the first and second-level enhancement layers in the multi-level enhancement layers are used as trainable parameters, and the base layer, the first-level and second-level enhancement layers are jointly trained. This process continues until all enhancement layers at each level have been trained.

8. The method of claim 7, further comprising: After joint training of any level of enhancement layer and the base layer is completed, the updated spatial opacity parameter obtained through training is acquired. The updated spatial opacity parameter is saved as data to be transmitted to the enhancement layer, so that when the client performs further visualization rendering of the target dynamic 3D scene based on the Gaussian elements contained in the enhancement layer, it can perform visualization rendering of the target dynamic 3D scene based on the updated spatial opacity parameter saved in the enhancement layer.

9. The method of claim 1, further comprising, before storing the base layer and the multi-level enhancement layers as independent transportable units: The Gaussian parameters of each Gaussian element contained in the base layer and the multi-level enhancement layer are quantized and compressed using a differentiated quantization method. The degree of quantization compression of the quantization method used to quantize and compress at least some of the Gaussian parameters of each Gaussian element contained in the enhancement layer is higher than the degree of quantization compression of the quantization method used to quantize and compress the Gaussian parameters of each Gaussian element contained in the base layer. Entropy encoding is performed on the base layer and the multi-level enhancement layer after quantization and compression.

10. The method of claim 9, further comprising, before quantizing and compressing the Gaussian parameters of each Gaussian unit contained in the base layer and the multi-level enhancement layers using a differentiated quantization method: Discard the last or more enhancement layers in the multi-level enhancement layers, and retain the base layer and the remaining enhancement layers at each level.

11. The method of claim 1, wherein determining the receiving bandwidth of the client comprises: Obtain the most recent multiple receive rate samples reported by the client; The harmonic mean of the multiple received rate samples is calculated as an estimate of the received bandwidth of the client.

12. The method of claim 11, wherein the target dynamic 3D scene to be transmitted is determined from the stored dynamic 3D scene based on the receiving bandwidth, and the target dynamic 3D scene, comprising a base layer and multiple enhancement layers, are sequentially transmitted to the client according to a progressive transmission strategy prioritizing the transmission of the base layer, comprising: Obtain the remaining space in the cache reported by the client; Based on the received bandwidth and the remaining space, the target dynamic 3D scene to be transmitted is determined from the stored dynamic 3D scene. From the base layer and multiple enhancement layers contained in the target dynamic 3D scene, at least one target layer to be transmitted is adaptively decided for the client. According to the progressive transmission strategy of prioritizing the transmission of the base layer, the at least one target layer is transmitted to the client as an independent transmittable unit in sequence.

13. The method of claim 12, wherein a target dynamic 3D scene to be transmitted is determined from a stored dynamic 3D scene based on the received bandwidth, at least one target layer to be transmitted is adaptively determined for the client from a base layer and multiple enhancement layers included in the target dynamic 3D scene, and the at least one target layer is sequentially transmitted to the client as an independent transmittable unit according to a progressive transmission strategy that prioritizes the transmission of the base layer, comprising: Determine whether the remaining space is lower than a preset first threshold; If the remaining space is lower than the first threshold, based on the recorded transmission progress for all stored dynamic 3D scenes, find the first dynamic 3D scene that has not yet been transmitted to the client from all stored dynamic 3D scenes; The first dynamic 3D scene is identified as the target dynamic 3D scene to be transmitted, and the base layer contained in the target dynamic 3D scene is transmitted to the client as at least one target layer to be transmitted.

14. The method of claim 13, further comprising: If the remaining space is not lower than the first threshold, it is further determined whether the receiving bandwidth is lower than a preset second threshold; If the received bandwidth is lower than the second threshold, based on the recorded transmission progress for all stored dynamic 3D scenes, the first dynamic 3D scene that has not yet been transmitted to the client from all stored dynamic 3D scenes is searched. The first dynamic 3D scene is identified as the target dynamic 3D scene to be transmitted, and the quality scores of the base layer and each level of enhancement layer contained in the target dynamic 3D scene are calculated; wherein, the quality score is used to represent the rendering quality; The layer with the highest quality score in the target dynamic 3D scene is identified as at least one target layer to be transmitted; If the at least one target layer contains only a base layer, the base layer is transmitted to the client; If the target layer includes any one of the multiple enhancement layers, the base layer, the enhancement layers preceding the target enhancement layer, and the target enhancement layer are transmitted sequentially to the client according to a progressive transmission strategy that prioritizes the transmission of the base layer.

15. The method of claim 14, comprising calculating the quality scores of the base layer and each level of enhancement layers contained in the target dynamic 3D scene, including: Calculate the quality gain score of the base layer and each level of enhancement layer contained in the target dynamic 3D scene; wherein, the quality gain score represents the rendering quality gain relative to the base layer; Calculate the bitrate of the base layer and each level of enhancement layer contained in the target dynamic 3D scene; Based on the rendering quality gain, the bitrate, and the remaining space in the buffer, the quality scores of the base layer and each level of enhancement layers contained in the target dynamic 3D scene are further calculated; wherein, the quality score is positively correlated with the rendering quality gain, negatively correlated with the bitrate, and negatively correlated with the remaining space in the buffer.

16. The method of claim 14, further comprising: If the received bandwidth is greater than or equal to the second threshold, calculate the byte budget for data transmission based on the remaining space in the buffer area; Based on the recorded transmission progress of all stored dynamic 3D scenes, find the dynamic 3D scene that has been transmitted to the client at least once a base layer, and the first dynamic 3D scene that has not been transmitted to the client at any layer. A set of dynamic 3D scenes is created based on the found dynamic 3D models for optimization solutions; From the set of dynamic 3D scenes, at least one dynamic 3D scene is further identified where the display effect on the client does not reach the final display effect enhanced by the highest-level enhancement layer on the initial display effect. All candidate transmission actions for each of the at least one dynamic 3D scene are listed. Each candidate transmission action indicates the transmission of at least one layer of the base layer and the multi-level enhancement of each dynamic 3D scene to the client. The number of bytes required to execute each candidate transmission action and the corresponding user experience score are pre-calculated. The user experience score represents the user's perceived experience of the display effect obtained by the client visually rendering the target dynamic 3D scene based on the Gaussian elements contained in the at least one layer transmitted according to the candidate transmission action after the execution of the candidate transmission action. The optimization objective is to find the candidate action combination with the highest cumulative value of the user experience score, provided that the total number of bytes consumed does not exceed the byte budget. The optimization solution is then performed on all the listed candidate transmission actions to find the target candidate action combination for the target dynamic 3D scene to be transmitted in the set of dynamic 3D scenes. The target candidate transmission action with the highest user experience score in the target candidate action combination is determined, and the target candidate transmission action is executed to transmit the client as at least one target layer to be transmitted, in accordance with a progressive transmission strategy that prioritizes the transmission of the base layer.

17. The method of claim 16, wherein the optimization solution comprises: The dynamic programming solution for the group knapsack problem is based on the calculation of the candidate action combination with the highest cumulative value of user experience score, with the total number of bytes consumed not exceeding the byte budget as the planning objective. Accordingly, the optimization algorithm corresponding to the optimization solution includes the dynamic programming algorithm corresponding to the grouped knapsack problem.

18. The method of claim 1, wherein the three-dimensional scene model is pre-trained in the following manner: Non-overlapping GoPs are obtained by segmenting video data sources used to train dynamic 3D scene models. The modeling method based on spatiotemporal Gaussian splashing is used to pre-train the GoP to obtain a 3D scene model based on Gaussian meta-modules that change dynamically in the time dimension.

19. The method of claim 18, wherein the video data source comprises multi-dimensional volumetric video with multiple perspectives.

20. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method as described in any one of claims 1 to 19.

21. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the method as described in any one of claims 1-19.

Images