Rate-controlled immersive communication system and method
By using the Gaussian splashing algorithm to convert multi-view videos into 3D point clouds and transmitting them in layers according to parameter importance, the problems of large data volume and frequent network bandwidth fluctuations in immersive video transmission are solved. This achieves a flexible balance between rendering quality and transmission bitrate, improving the system's network adaptability and user experience.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANGHAI JIAOTONG UNIV
- Filing Date
- 2026-03-31
- Publication Date
- 2026-07-24
AI Technical Summary
Existing immersive multi-view video transmission technologies cannot achieve unified closed-loop optimization among content complexity, bitrate control, network status, terminal computing power, and decoding recoverability, resulting in bandwidth fluctuations, device heterogeneity, and insufficient reconstruction quality stability.
Employing a layered transmission module, a semantic compression module, and an adaptive quantization and entropy coding module, the system transforms multi-view videos into 3D Gaussian point clouds using a Gaussian splashing algorithm. The data is then packaged and transmitted in layers according to parameter importance, and the bitrate is dynamically adjusted based on network conditions and terminal capabilities to achieve a flexible balance between rendering quality and transmission bitrate.
It significantly improves compression efficiency, alleviates bandwidth pressure, enhances the continuity and stability of user experience, adapts to network fluctuations and heterogeneous terminals, and ensures reliable transmission of critical visual information.
Smart Images
Figure CN122457784A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of three-dimensional video representation and compression technology, specifically relating to an immersive communication system and method with controllable bitrate. Background Technology
[0002] With the evolution of 6G communication technology, immersive communication, as one of the six typical application scenarios defined by IMT-2030, places higher demands on the real-time performance, interactivity, and realism of multimedia transmission. Traditional planar video can no longer meet users' needs for free-viewpoint, six-degrees-of-freedom (6DoF) immersive experiences, and multi-view video has become the core media form supporting immersive services. However, multi-view video has a huge data volume, and directly transmitting the raw video stream will consume extremely high bandwidth resources, placing a heavy burden on the communication network.
[0003] In recent years, breakthroughs in neural rendering technology have provided new technological pathways for immersive communication. Among these, the 3D Gaussian Splatting algorithm represents scenes using explicit 3D point clouds, enabling high-quality real-time rendering while significantly reducing computational complexity while maintaining visual quality. This algorithm models the scene as a time-varying 3D Gaussian point cloud, where each Gaussian point contains parameters such as position, covariance, color, and opacity, allowing for efficient rendering of images from any viewpoint through rasterization. Simultaneously, semantic communication technology mines the semantic information of the data and performs intelligent compression at the source, transmitting only information critical to the task objective, rather than the original bitstream, thus achieving more efficient transmission. Combined with the explicit parameterization characteristics of Gaussian Splatting, semantic-level compression and optimization of point cloud parameters can be performed, dynamically adjusting the transmission strategy according to network conditions to achieve a flexible trade-off between bitrate and rendering quality.
[0004] The patent document "A Method and System for Immersive Video Coding Based on 3DGS" (CN121547592A) discloses a point cloud initialization method using multi-view video frames and 3D sparse reconstruction. It extracts the spatial context information of anchor points through multi-resolution hash coding, combines this with neural network prediction of 3D Gaussian distribution parameters, and performs quantization and entropy coding to optimize anchor point parameters and generate a compressed 3D scene representation. This scheme does not involve transmission processes or network adaptation and is a fundamental coding technique.
[0005] The patent document "A Video Stream Processing Method with Dynamic Gaussian Compression and Adaptive Bitrate Adjustment" (CN120547374A) discloses a video stream processing method that employs dynamic Gaussian compression and adaptive bitrate adjustment. By constructing a multi-resolution binary hash grid and a deformation prediction network, combined with mask pruning mechanisms and entropy modeling, it achieves efficient encoding and adaptive bitrate control for dynamic 3D scenes. The bitrate adjustment in this scheme is primarily manifested as switching between preset bitrate versions, limited to discrete switching between different model versions.
[0006] The paper "GIFStream: 4D Gaussian-Based Immersive Video with FeatureStream" (H. Li, S. Li, X. Gao, A. Batuer, L. Yu and Y. Liao, 2025 IEEE / CVFConference on Computer Vision and Pattern Recognition, Nashville, TN, USA, 2025, pp. 21761-21770) discloses a proposal to enhance deformation field representation capabilities with temporally correlated feature streams, supporting complex dynamic scenes. It achieves motion-adaptive pruning and video parameter reorganization, balancing representational capability and compression efficiency, supporting end-to-end compression and compatibility with traditional video encoders, and enabling high-quality immersive real-time video rendering at high bitrates. This scheme focuses compression on temporal feature streams, making it closer to a temporally driven compression system rather than a content-aware structured compression system.
[0007] Existing technical solutions generally only perform adaptation at a single level, such as the presentation layer, feature layer, or transport layer, and cannot truly achieve unified closed-loop optimization among "content complexity, bitrate control, network status, terminal computing power, and decoding recoverability". They are still insufficient for the coordinated control of bandwidth fluctuations, device heterogeneity, decoding load differences, and reconstruction quality stability in real streaming media environments.
[0008] Based on the above background, an immersive communication system with controllable bitrate is proposed to solve the problems of large data volume, frequent network bandwidth fluctuations, and difficulty in balancing rendering quality and transmission bitrate in immersive multi-view video transmission. Summary of the Invention
[0009] In view of the shortcomings of the prior art, the purpose of this invention is to provide an immersive communication system and method with controllable bit rate.
[0010] According to the present invention, an immersive communication system with controllable bit rate includes a receiver and a transmitter, and further includes a layered transmission module;
[0011] The sending end includes an immersive content representation module, a semantic compression module, and an adaptive quantization and entropy coding module; The immersive content representation module performs content preprocessing and point cloud modeling; The semantic compression module compresses and prunes the point cloud parameters to obtain a compressed point cloud; The adaptive quantization and entropy coding module nonlinearly quantizes and compresses the point cloud according to the target bit rate requirement to generate a bit stream; The layered transmission module packages the bitstream into layers according to parameter importance and transmits the compressed point cloud. The receiving end decodes and compresses the point cloud, and renders the video from the corresponding viewpoint in real time.
[0012] Preferably, the transmitting end acquires multi-view video sequences, and the immersive content representation module uses a content complexity-aware Gaussian point density adaptive control method to convert the multi-view videos into corresponding three-dimensional Gaussian point clouds and real-time field changes through a Gaussian splashing algorithm.
[0013] The semantic compression module constructs a joint optimization objective for rendering loss and bitrate loss, as well as an evaluation algorithm for parameter importance, using a low-rank residual projection compression method with content complexity constraints, to compress and prune point cloud parameters.
[0014] The adaptive quantization and entropy coding module calculates the target quantization step size based on the current channel bandwidth estimate, performs nonlinear quantization on the compressed point cloud parameters, and generates a compressed bitstream through entropy coding.
[0015] Preferably, the joint optimization objective includes the photometric loss between the rendered image and the original viewpoint image and the bitrate loss estimated by the super-prior network.
[0016] The algorithm for evaluating the importance of the parameters is optimized by gradient descent to learn the quantization mask or importance weight of each parameter.
[0017] The entropy encoding employs an arithmetic encoder based on parameter type context.
[0018] The adaptive quantization and entropy coding module monitors network round-trip latency and packet loss rate in real time and estimates available bandwidth. If network congestion is detected, the compression ratio is increased and the quantization step size is increased. If the network condition is good, the compression ratio and quantization compensation are decreased.
[0019] Preferably, the layered transmission module divides the bitstream into a basic layer and an enhancement layer according to the importance of parameters, and prioritizes the transmission of the basic layer.
[0020] The base layer contains parameters that have the greatest impact on rendering quality, while the enhancement layer contains detailed parameters.
[0021] The receiving end decodes to obtain a compressed point cloud, and renders the corresponding viewpoint image in real time using a Gaussian sputtering rasterization pipeline according to instructions.
[0022] Preferably, there are several receivers, each feeding back the channel status and device capabilities to the transmitter.
[0023] The transmitting end classifies users into high, medium, and low computing power users based on the channel status and equipment capabilities fed back by each receiving end. Based on the classification results, it implements hierarchical coding and personalized transmission, and dynamically adapts the code rate.
[0024] The device capabilities include whether it supports real-time Gaussian splash rendering, supported display resolution, target rendering frame rate, number of Gaussian points that can be processed per unit time, GPU / CPU computing power, and at least one of video memory or memory resources, and the channel state is the current channel bandwidth estimate.
[0025] Preferably, the high-computing-power user is a receiving end capable of processing a complete Gaussian point cloud parameter sequence and performing real-time Gaussian splash rendering at a preset resolution and frame rate.
[0026] The medium-computing-power user is a receiver capable of processing Gaussian point cloud parameters after cropping or downsampling and performing restricted real-time rendering.
[0027] The low-computing-power user is a receiver that can only process basic layer parameters, downsampled point clouds, or lightweight rendering tasks.
[0028] The transmitting end sets compression parameters, quantization step size, and hierarchical transmission strategy according to the channel state and equipment capabilities of each receiving end, including: Transmit the complete parameter sequence to users with high computing power; Transmit base layer parameters and some enhancement layer parameters to users with medium computing power; For low-bandwidth users, only the base layer parameters or downsampled point clouds are transmitted.
[0029] The present invention provides an immersive communication method with controllable bit rate, comprising: Step S1: Instruct the immersive content representation module to perform content preprocessing and point cloud modeling; Step S2: Instruct the semantic compression module to compress and prune the point cloud parameters to obtain a compressed point cloud; Step S3: Instruct the adaptive quantization and entropy coding module to nonlinearly quantize and compress the point cloud according to the target bit rate requirement to generate a bit stream; Step S4: Instruct the layered transmission module to package the bitstream into layers according to parameter importance and transmit the compressed point cloud; Step S5: The receiving end decodes the compressed point cloud and renders the video from the corresponding viewpoint in real time.
[0030] Preferably, in step S1, the transmitting end acquires a multi-view video sequence.
[0031] The immersive content representation module uses a content complexity-aware Gaussian point density adaptive control method to convert multi-view videos into corresponding three-dimensional Gaussian point clouds and real-time field changes through a Gaussian splashing algorithm.
[0032] The semantic compression module constructs a joint optimization objective for rendering loss and bitrate loss, as well as an evaluation algorithm for parameter importance, using a low-rank residual projection compression method with content complexity constraints, to compress and prune point cloud parameters.
[0033] The adaptive quantization and entropy coding module calculates the target quantization step size based on the current channel bandwidth estimate, performs nonlinear quantization on the compressed point cloud parameters, and generates a compressed bitstream through entropy coding.
[0034] Preferably, the joint optimization objective includes the photometric loss between the rendered image and the original viewpoint image and the bitrate loss estimated by the super-prior network.
[0035] The algorithm for evaluating the importance of the parameters is optimized by gradient descent to learn the quantization mask or importance weight of each parameter.
[0036] The entropy encoding employs an arithmetic encoder based on parameter type context.
[0037] The adaptive quantization and entropy coding module monitors network round-trip latency and packet loss rate in real time and estimates available bandwidth. If network congestion is detected, the compression ratio is increased and the quantization step size is increased. If the network condition is good, the compression ratio and quantization compensation are decreased.
[0038] Preferably, the layered transmission module divides the bitstream into a basic layer and an enhancement layer according to the importance of parameters, and prioritizes the transmission of the basic layer.
[0039] The base layer contains parameters that have the greatest impact on rendering quality, while the enhancement layer contains detailed parameters.
[0040] There are several receiving ends, which respectively feed back the channel status and device capabilities to the sending end; The compressed point cloud is obtained by decoding, and the corresponding viewpoint image is rendered in real time using the Gaussian splash rasterization pipeline according to the instructions.
[0041] The transmitting end classifies users into high, medium, and low computing power users based on the channel status and equipment capabilities fed back by each receiving end. Based on the classification results, it implements hierarchical coding and personalized transmission, and dynamically adapts the code rate.
[0042] The device capabilities include whether it supports real-time Gaussian splash rendering, supported display resolution, target rendering frame rate, number of Gaussian points that can be processed per unit time, GPU / CPU computing power, and at least one of video memory or memory resources, and the channel state is the current channel bandwidth estimate.
[0043] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention transforms video content into an explicit 3D point cloud representation, achieving a flexible balance between rendering quality and transmission bitrate.
[0044] 2. This invention significantly improves compression efficiency by fully utilizing the statistical characteristics and visual importance of Gaussian parameters through semantic compression and adaptive quantization mechanisms.
[0045] 3. This invention enables the system to have strong network adaptability through layered transmission and dynamic bit rate control, and can prioritize the reliable transmission of key visual information in environments with fluctuating bandwidth.
[0046] 4. This invention effectively alleviates the bandwidth pressure of immersive video transmission and improves the continuity and stability of user experience. Attached Figure Description
[0047] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a schematic diagram of an immersive communication system architecture with controllable bit rate. Figure 2 This is a schematic diagram of the multi-user immersive communication system architecture according to an embodiment of the present invention. Detailed Implementation
[0048] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.
[0049] This invention provides a rate-controlled immersive communication system. This system is an immersive communication system that adopts a rate-controlled immersive communication architecture based on Gaussian splashing. On the basis of dynamic Gaussian representation, it further introduces a content complexity-aware Gaussian point density adaptive control method (CCPS) and a content complexity-constrained low-rank residual projection compression method (CCNP). It is not limited to any special network topology, encoding tool, quantization scheme or transmission protocol. It is an integrated closed-loop mechanism of "joint optimization of representation structure, point set density, residual representation and entropy coding", so that the compression process directly participates in and constrains the dynamic Gaussian scene representation itself.
[0050] A content-complexity-aware Gaussian point density adaptive control method (CCPS) is used to jointly control Gaussian point densification, pruning, and local expansion; a content-complexity-constrained low-rank residual projection compression method (CCNP) is used to achieve low-rank compression of high-dimensional Gaussian attribute residuals; and by combining explicit anchor-residual representation and hash parameter-oriented conditional entropy coding, a multi-level bitrate control framework at the point, residual, and parameter levels is formed to achieve differentiated bitrate allocation for different content regions. Specifically, this includes a transmitter, a receiver, and a layered transmission module.
[0051] The transmitting end performs multi-view video to dynamic Gaussian point cloud and time-varying field generation, bitrate-controllable compression representation, adaptive bitrate selection, and layered transmission control; the receiving end mainly achieves free-view display through decoding, decompression, and Gaussian splash rendering technology.
[0052] The sending end includes an immersive content representation module, a semantic compression module, and an adaptive quantization and entropy coding module. Specifically: Immersive content representation module: The sending end converts multi-view videos into 3D point clouds and changes the field in real time using a Gaussian splashing algorithm.
[0053] Specifically, with Figure 1 For example, in a single-user immersive video communication scenario, the sending end encodes multi-view video into a time-varying Gaussian point cloud, which is then transmitted over the network to the receiving end for free-view rendering.
[0054] The immersive content representation module performs content preprocessing and point cloud modeling. The transmitting end acquires multi-view video sequences and estimates the corresponding 3D Gaussian point clouds and real-time field variations based on the Gaussian splash algorithm.
[0055] In a 3D Gaussian point cloud, each Gaussian point is described by the following parameters: 3D position coordinates, color described by spherical harmonic coefficients, and opacity. A time-varying field describes the change of each Gaussian point's parameters over time.
[0056] By employing a content complexity-aware Gaussian point density adaptive control method (CCPS), which jointly considers the gradient strength, dynamics, and block-level low-rank residual error of Gaussian points, content-aware control is achieved for the densification, retention, and pruning of Gaussian points. This effectively compresses redundancy in low-value regions while ensuring the reconstruction quality of key dynamic regions.
[0057] Semantic compression module: Constructs a joint optimization objective for rendering loss and bitrate loss, and an algorithm for evaluating parameter importance, to compress and prune point cloud parameters.
[0058] In single-user immersive video communication scenarios, a joint optimization objective function is constructed in the semantic compression module, including the photometric loss between the rendered image and the original viewpoint image and the bitrate loss estimated by the super-prior network. Through gradient descent optimization, the quantization mask or importance weight of each parameter is learned, achieving fine preservation of key parameters and compression of minor parameters.
[0059] By employing a content complexity-constrained low-rank residual projection compression method (CCNP), high-dimensional Gaussian attribute residuals are represented and compressed in a low-rank subspace, further reducing coding redundancy and improving code rate controllability.
[0060] Adaptive quantization and entropy coding module: dynamically adjusts the quantization step size according to the target bit rate requirement, and generates the final bit stream through entropy coding.
[0061] In single-user immersive video communication scenarios, the adaptive quantization and entropy coding module calculates the target quantization step size based on the current channel bandwidth estimate and performs nonlinear quantization on the compressed parameters. An arithmetic encoder based on parameter type context is then used for entropy coding to generate the compressed bitstream.
[0062] Real-time monitoring of network round-trip latency and packet loss rate estimates available bandwidth. If network congestion is detected, the compression ratio is increased while the quantization step size is increased; if network conditions are good, the compression ratio and quantization compensation are decreased to improve rendering quality.
[0063] By combining explicit anchor-residual representation with conditional entropy modeling oriented towards hash parameters, integrated compression of the representation layer, residual layer, and parameter layer is achieved.
[0064] Layered transmission module: Divides point cloud parameters into basic layer and enhancement layer transmission according to their importance.
[0065] The layered transmission module performs bitrate control and transmission, and packages the bitstream into layers according to the importance of parameters. The base layer contains parameters that have the greatest impact on rendering quality, while the enhancement layer contains detailed parameters. Priority is given to ensuring the transmission of the base layer.
[0066] The layered transmission module can be integrated inside the sending end or implemented as a network-side or edge-side module. Its core function is to organize, schedule, and transmit data at different layers based on network status and / or terminal computing power.
[0067] For immersive communication scenarios involving dynamic Gaussian point clouds and changing fields, a systematic, interconnected design is implemented, incorporating adaptive quantization and entropy coding, hierarchical transmission based on parameter importance, and receiver-side free-viewpoint rendering, all aimed at achieving a controllable bitrate. Specifically: The compression ratio and quantization step size are dynamically adjusted based on the current network round-trip latency, packet loss rate, and available bandwidth, and further adjusted according to the computing power of the user terminal. Simultaneously, the base layer and enhancement layer are divided based on the degree of parameter impact on rendering quality, prioritizing the transmission of critical parameters, thus balancing bitrate, reconstruction quality, and real-time performance under limited bandwidth and heterogeneous terminal conditions. For immersive communication scenarios, specific selection, combination, and coordinated control of relevant technologies enable dynamic Gaussian immersive communication with controllable bitrate, adjustable quality, and stable transmission.
[0068] Receiver rendering module: After decoding, the receiver uses Gaussian splashing to render the video from a given perspective.
[0069] In single-user immersive video communication scenarios, after the receiving end decodes and obtains the point cloud parameters, it uses the Gaussian sputtering rasterization pipeline to render the corresponding viewpoint image in real time according to the user's head posture or interaction commands, presenting an immersive visual experience.
[0070] by Figure 2 For example, in a multi-user broadcast scenario, different user devices have different computing capabilities and display requirements. The transmitting end adaptively adjusts the encoding parameters and transmission strategies according to the channel status and device capabilities of each user.
[0071] Each receiving end performs user capability reporting, that is, it feeds back to the sending end its own device computing power (whether it supports Gaussian splash real-time rendering), display resolution requirements, and current channel bandwidth estimate. Layered coding and personalized transmission are implemented, and the bitrate is dynamically adapted.
[0072] Specifically, users with high, medium, and low computing power are categorized based on the receiving end's processing capabilities for dynamic Gaussian point cloud decoding and Gaussian splash rendering. In more preferred embodiments, the sending end makes a joint judgment based on the device capability information and network status information (channel status) fed back by the receiving end. The device capability information includes whether it supports real-time Gaussian splash rendering, the supported display resolution, the target rendering frame rate, the number of Gaussian points that can be processed per unit time, GPU / CPU computing power, and at least one of video memory or memory resources.
[0073] Among them, high-computing-power users are preferably those capable of processing complete Gaussian point cloud parameter sequences and performing real-time Gaussian splash rendering at preset resolution and frame rate; medium-computing-power users are preferably those capable of processing cropped or downsampled Gaussian point cloud parameters and performing restricted real-time rendering; and low-computing-power users are preferably those capable of processing only basic layer parameters, downsampled point clouds, or lightweight rendering tasks. Based on the above classification results, the sending end sets compression parameters, quantization step size, and layered transmission strategies for different users.
[0074] Specifically, the sending end configures differentiated compression parameters and quantization step sizes for different users based on feedback information. For users with high computing power and high bandwidth, the complete parameter sequence is transmitted; for users with low bandwidth or mobile devices, only the base layer parameters or downsampled point clouds are transmitted, and the receiving end recovers the visual content through lightweight rendering or upsampling.
[0075] The transmitting end continuously monitors the channel fluctuations of each user and ensures that the transmission rate of each receiving end is adapted to its channel capacity by adjusting the quantization step size and hierarchical transmission strategy, thereby avoiding congestion or underload.
[0076] This invention can dynamically adjust the compression ratio not only based on network bandwidth fluctuations but also based on the computing power of the user terminal, thus balancing transmission stability, decoding real-time performance, and reconstruction quality under different network conditions and on different terminal devices. Simultaneously, it supports compressed bitstream export, decoding recovery, and rendering verification, ensuring that the transmitted model has recoverable, decodeable, and displayable practical application capabilities.
[0077] In more preferred examples, the compression rate is dynamically adjusted according to bandwidth fluctuations and user terminal computing power. This results in more precise bitrate control, more comprehensive compression targets, stronger fidelity in high dynamic areas, and more thorough compression of low-value areas. Even when there are large variations in scenario complexity, significant network bandwidth fluctuations, and large differences in user terminal computing power, it can still better balance compression rate, reconstruction quality, and real-time decoding. Furthermore, it is more suitable for stable transmission and real-time decoding in real network and heterogeneous terminal scenarios.
[0078] CCPS does not rely on a single mask for judgment, but rather performs continuous adaptive adjustment of the Gaussian point structure based on content complexity. This is used to integrate the gradient information, dynamic attributes, and block-level low-rank residual errors of Gaussian points, performing adaptive densification, preservation, and pruning. CCNP does not simply entropy encode attributes directly; instead, it first performs low-rank projection compression on the residuals before entering the entropy encoding stage. This is used to project and compress the Gaussian attribute residuals into a low-rank subspace, thereby reducing high-dimensional redundancy. Therefore, the system emphasizes structured residual compression and content-aware point set control, simultaneously performing adaptive adjustments at multiple levels: point set structure, residual representation, entropy encoding objects, and transmission adaptation strategies.
[0079] This invention provides a bitrate-controllable immersive communication method. By using a Gaussian splashing algorithm to convert multi-view video into time-varying 3D point clouds, it achieves efficient parameterized representation. Through semantic compression and layered transmission mechanisms, the system can dynamically adjust the transmission strategy according to network conditions, optimize bandwidth utilization while ensuring rendering quality, and achieve adaptive optimization of rendering quality and transmission bitrate, thereby improving the network adaptability and user experience of the immersive communication system.
[0080] This includes: a multi-view video point cloud representation and transmission framework based on the Gaussian splashing algorithm; a semantic compression mechanism for joint optimization of rendering quality and bitrate; and a hierarchical transmission and dynamic bitrate control strategy based on parameter importance. The specific steps are as follows: Step S1: Instruct the immersive content representation module to perform content preprocessing and point cloud modeling; Step S2: Instruct the semantic compression module to compress and prune the point cloud parameters to obtain a compressed point cloud; Step S3: Instruct the adaptive quantization and entropy coding module to nonlinearly quantize and compress the point cloud according to the target bit rate requirement to generate a bit stream; Step S4: Instruct the layered transmission module to package the bitstream into layers according to parameter importance and transmit the compressed point cloud; Step S5: The receiving end decodes the compressed point cloud and renders the video from the corresponding viewpoint in real time.
[0081] In more preferred embodiments, in step S1, the transmitting end acquires a multi-view video sequence.
[0082] The immersive content representation module uses a content complexity-aware Gaussian point density adaptive control method to convert multi-view videos into corresponding 3D Gaussian point clouds and real-time field changes through a Gaussian splashing algorithm.
[0083] The semantic compression module constructs a joint optimization objective for rendering loss and bitrate loss using a low-rank residual projection compression method with content complexity constraints, and an evaluation algorithm for parameter importance, to compress and prune point cloud parameters.
[0084] The adaptive quantization and entropy coding module calculates the target quantization step size based on the current channel bandwidth estimate, performs nonlinear quantization on the compressed point cloud parameters, and generates a compressed bitstream through entropy coding.
[0085] In more preferred embodiments, the joint optimization objective includes the photometric loss between the rendered image and the original viewpoint image and the bitrate loss estimated by the super-prior network.
[0086] The algorithm for evaluating the importance of the parameters is optimized by gradient descent to learn the quantization mask or importance weight of each parameter.
[0087] The entropy encoding employs an arithmetic encoder based on parameter type context.
[0088] The adaptive quantization and entropy coding module monitors network round-trip latency and packet loss rate in real time and estimates available bandwidth. If network congestion is detected, the compression ratio is increased and the quantization step size is increased. If the network condition is good, the compression ratio and quantization compensation are decreased.
[0089] In more preferred embodiments, the layered transmission module divides the bitstream into a basic layer and an enhancement layer according to the importance of parameters, giving priority to the transmission of the basic layer.
[0090] The base layer contains parameters that have the greatest impact on rendering quality, while the enhancement layer contains detailed parameters.
[0091] There are several receiving ends, which respectively feed back the channel status and device capabilities to the sending end; The compressed point cloud is obtained by decoding, and the corresponding viewpoint image is rendered in real time using the Gaussian splash rasterization pipeline according to the instructions.
[0092] The transmitting end classifies users into high, medium, and low computing power users based on the channel status and equipment capabilities fed back by each receiving end. Based on the classification results, it implements hierarchical coding and personalized transmission, and dynamically adapts the code rate.
[0093] The device capabilities include whether it supports real-time Gaussian splash rendering, supported display resolution, target rendering frame rate, number of Gaussian points that can be processed per unit time, GPU / CPU computing power, and at least one of video memory or memory resources, and the channel state is the current channel bandwidth estimate.
[0094] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function as logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.
[0095] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.
Claims
1. An immersive communication system with controllable bit rate, comprising a receiver and a transmitter, characterized in that, Also includes: Layered transmission module; The sending end includes an immersive content representation module, a semantic compression module, and an adaptive quantization and entropy coding module; The immersive content representation module performs content preprocessing and point cloud modeling; The semantic compression module compresses and prunes the point cloud parameters to obtain a compressed point cloud; The adaptive quantization and entropy coding module nonlinearly quantizes and compresses the point cloud according to the target bit rate requirement to generate a bit stream; The layered transmission module packages the bitstream into layers according to parameter importance and transmits the compressed point cloud. The receiving end decodes and compresses the point cloud, and renders the video from the corresponding viewpoint in real time.
2. The immersive communication system with controllable bit rate according to claim 1, characterized in that, The transmitting end acquires multi-view video sequences, and the immersive content representation module uses a content complexity-aware Gaussian point density adaptive control method to convert the multi-view videos into corresponding three-dimensional Gaussian point clouds and time-varying fields through a Gaussian splashing algorithm. The semantic compression module constructs a joint optimization objective for rendering loss and bitrate loss, as well as an evaluation algorithm for parameter importance, using a low-rank residual projection compression method with content complexity constraints, to compress and prune point cloud parameters. The adaptive quantization and entropy coding module calculates the target quantization step size based on the current channel bandwidth estimate, performs nonlinear quantization on the compressed point cloud parameters, and generates a compressed bitstream through entropy coding.
3. The immersive communication system with controllable bit rate according to claim 2, characterized in that, The joint optimization objective includes the photometric loss between the rendered image and the original viewpoint image and the bitrate loss estimated by the super-prior network. The algorithm for evaluating the importance of the parameters is to learn the quantization mask or importance weight of each parameter through gradient descent optimization. The entropy encoding employs an arithmetic encoder based on parameter type context; The adaptive quantization and entropy coding module monitors network round-trip latency and packet loss rate in real time and estimates available bandwidth. If network congestion is detected, the compression ratio is increased and the quantization step size is increased. If the network condition is good, the compression ratio and quantization compensation are decreased.
4. The immersive communication system with controllable bit rate according to claim 1, characterized in that, The layered transmission module divides the bitstream into a basic layer and an enhancement layer according to the importance of parameters, and prioritizes the transmission of the basic layer. The base layer contains parameters that have the greatest impact on rendering quality, while the enhancement layer contains detailed parameters. The receiving end decodes to obtain a compressed point cloud, and renders the corresponding viewpoint image in real time using a Gaussian sputtering rasterization pipeline according to instructions.
5. The immersive communication system with controllable bit rate according to claim 1, characterized in that, There are several receiving ends, which respectively feed back the channel status and device capabilities to the sending end; The transmitting end classifies users into high, medium, and low computing power users based on the channel status and equipment capabilities fed back by each receiving end, implements hierarchical coding and personalized transmission based on the classification results, and dynamically adapts the code rate. The device capabilities include whether it supports real-time Gaussian splash rendering, supported display resolution, target rendering frame rate, number of Gaussian points that can be processed per unit time, GPU / CPU computing power, and at least one of video memory or memory resources, and the channel state is the current channel bandwidth estimate.
6. The immersive communication system with controllable bit rate according to claim 5, characterized in that, The high-computing-power user is a receiving end capable of processing a complete Gaussian point cloud parameter sequence and performing real-time Gaussian splash rendering at a preset resolution and frame rate; The medium-computing-power user is a receiving end capable of processing the cropped or downsampled Gaussian point cloud parameters and performing restricted real-time rendering; The low-computing-power user is a receiver that can only process basic layer parameters, downsampled point clouds, or lightweight rendering tasks. The transmitting end sets compression parameters, quantization step size, and hierarchical transmission strategy according to the channel state and equipment capabilities of each receiving end, including: Transmit the complete parameter sequence to users with high computing power; Transmit base layer parameters and some enhancement layer parameters to users with medium computing power; For low-bandwidth users, only the base layer parameters or downsampled point clouds are transmitted.
7. A rate-controllable immersive communication method, characterized in that, include: Step S1: Instruct the immersive content representation module to perform content preprocessing and point cloud modeling; Step S2: Instruct the semantic compression module to compress and prune the point cloud parameters to obtain a compressed point cloud; Step S3: Instruct the adaptive quantization and entropy coding module to nonlinearly quantize and compress the point cloud according to the target bit rate requirement to generate a bit stream; Step S4: Instruct the layered transmission module to package the bitstream into layers according to parameter importance and transmit the compressed point cloud; Step S5: The receiving end decodes the compressed point cloud and renders the video from the corresponding viewpoint in real time.
8. The immersive communication method with controllable bit rate according to claim 7, characterized in that, In step S1, the transmitting end acquires multi-view video sequences; The immersive content representation module uses a content complexity-aware Gaussian point density adaptive control method to convert multi-view videos into corresponding three-dimensional Gaussian point clouds and real-time field changes through a Gaussian splashing algorithm. The semantic compression module constructs a joint optimization objective for rendering loss and bitrate loss, as well as an evaluation algorithm for parameter importance, using a low-rank residual projection compression method with content complexity constraints, to compress and prune point cloud parameters. The adaptive quantization and entropy coding module calculates the target quantization step size based on the current channel bandwidth estimate, performs nonlinear quantization on the compressed point cloud parameters, and generates a compressed bitstream through entropy coding.
9. The immersive communication method with controllable bit rate according to claim 8, characterized in that, The joint optimization objective includes the photometric loss between the rendered image and the original viewpoint image and the bitrate loss estimated by the super-prior network. The algorithm for evaluating the importance of the parameters is to learn the quantization mask or importance weight of each parameter through gradient descent optimization. The entropy encoding employs an arithmetic encoder based on parameter type context; The adaptive quantization and entropy coding module monitors network round-trip latency and packet loss rate in real time and estimates available bandwidth. If network congestion is detected, the compression ratio is increased and the quantization step size is increased. If the network condition is good, the compression ratio and quantization compensation are decreased.
10. The immersive communication method with controllable bit rate according to claim 7, characterized in that, The layered transmission module divides the bitstream into a basic layer and an enhancement layer according to the importance of parameters, and prioritizes the transmission of the basic layer. The base layer contains parameters that have the greatest impact on rendering quality, while the enhancement layer contains detailed parameters. There are several receiving ends, which respectively feed back the channel status and device capabilities to the sending end; Decode the compressed point cloud and render the corresponding viewpoint image in real time using the Gaussian splash rasterization pipeline according to the instructions; The transmitting end classifies users into high, medium, and low computing power users based on the channel status and equipment capabilities fed back by each receiving end, implements hierarchical coding and personalized transmission based on the classification results, and dynamically adapts the code rate. The device capabilities include whether it supports real-time Gaussian splash rendering, supported display resolution, target rendering frame rate, number of Gaussian points that can be processed per unit time, GPU / CPU computing power, and at least one of video memory or memory resources, and the channel state is the current channel bandwidth estimate.